Network Flow Data in the Cloud
summary
Builds per-service flow maps from sFlow + hsfd agent sampled on every host, parses to pmacct -> Parquet to S3 and annotates with CI data to attribute traffic to services, AZs, and regions. Uses Kentik (or equivalent AWS describe calls) to reconstruct topology and run graph-feasibility checks for migrations and AZ failures. Deployed over one quarter, running since Jan 2024, costs reported about three orders of magnitude lower than vendor logging, with ~1 PR/month and a single pager event.
AI-written summary · about
chapters (14)
- 0:00Intro
- 0:29Motivation
- 2:31Flow Network
- 4:44AWS Infrastructure Diagram
- 5:37AWS Infrastructure Graph
- 6:02Questions to Answer
- 6:29SFlow
- 7:34Slack
- 9:55Implementation
- 10:26Why not just use VPC logs
- 11:58Use cases
- 12:30Tradeoffs
- 13:04Migrations
- 14:50Summary
related talks
23mNetwork Flow Aggregation: Pay for the Logs You Care About!
21mTransparent and Flexible Network Management for Big Data Processing in the Cloud
29mFinding the Needle in a Haystack: Tenant-Level Network...
35mNetwork Flow Monitoring in Kubernetes with Contiv/VPP CNI and Elastic Stack
41mNetwork Observability
35mNetwork Flow Monitoring in Kubernetes with Contiv/VPP CNI and the Elastic Stack
14mSkyline: A Cloud Centric Internet Monitoring Engine
29mPractical ISP CSIRT Incident Handling w/ Network Flows, ELK Stack & Cybersecurity Intelligence...