Is Visibility an Art of Possible or Impossible?
summary
Argues full visibility is rarely absolute but good enough is reachable by treating logs as a first-class data pillar and building a secure logging program with governance, architecture, and runbooks. Describes a pipeline (receivers → processors → tiered storage → dashboards/alerts), routing cloud logs via S3 + SQS into SIEM/SOAR, using Kafka for a centralized queue, adopting detection-as-code (Sigma), and facing brutal migration pain (example: ~100 TB/day indexed historically so teams prefer leaving legacy data). Cites 72% cloud adoption and ~70% SOC alert overload, and recommends focused asset scope, layered tuning, analyst training, and measured automation to reduce noise.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:07Defining visibility in IT
- 5:08Monitoring vs observability
- 7:16Data pillars for monitoring
- 9:30Logs' critical role
- 13:56Secure logging maturity
- 16:05Logging architecture
- 20:26Cloud visibility threats
- 23:24Public cloud logs hierarchy
- 27:44Logging and SIEM rationalization
- 31:20Data migration challenges
- 33:33Alert fatigue in SOC
- 34:18Alert management triad
- 39:48Data volume vs visibility
- 40:54Takeaways and summary
related talks
31mVisibility from Code to Cloud - How pipeline analytics changes how we think about SDLC
36mObservability? It's a Data Analytics Problem, You Fool!
31mSeeing Through the Fog - Navigating the Security Landscape of a Cloud-First World
35mIs It Time To Version Observability? (Signs Point To Yes)
29mCrucial Log Management Skills for DevSecOps
29mObservability; a gentle introduction
52mObservability: it's not just an ops thing
13mObservability that matters