Week 3 Live Q&A
summary
Live Q&A covers concrete system behavior and runbooks: Kafka achieves 100 MB/s on six cheap machines in tests, scales roughly linearly by adding brokers, and requires careful partition key design plus rate limiters for multi‑tenant clusters; Confluent/Databricks use dedicated vs logical clusters and rely on container/VM isolation; deployments use Conveyor with conveyor-in-a-box plus Twine CLI fallbacks, configurable rollbacks, retries for resizes; MySQL topologies are primary + two secondaries (no erasure coding); RDMA gets cautious R&D and Unity Catalog adds table‑level access control.
AI-written summary · about
chapters (24)
- 0:00Welcome to live Q&A
- 0:45Host introduction
- 1:05Kafka performance beyond config
- 2:33MySQL Shard Manager disaster recovery
- 3:37Trade-offs in system design
- 5:02Monolith decomposition strategies
- 7:07Multi-tenant Kafka isolation challenges
- 10:41Fox disaster recovery guarantees
- 12:07Conveyor resilience bypasses
- 13:34Risks of deployment bypasses
- 15:21Hardest abstractions in multi-cloud
- 17:48Next steps for data platform
- 19:32Kafka benchmarks and tuning
- 23:50Fox durability in regional failures
- 25:54Elastic deployments at Facebook
- 29:07Kubernetes service frameworks
- 32:17Open source at Facebook
- 33:22Duplicate message patterns
- 36:55Push failure handling
- 39:46Self-managed vs cloud Kubernetes
- 42:12Facebook cloud and serverless vision
- 44:00RDMA in Kafka deployments
- 47:13Rust challenges at Facebook
- 50:05Closing remarks
related talks
13mSystems @Scale Summer Live Q&A Session #3
38mQuest to Connect Teams with Kafka
20mSystems @Scale Summer Live Q&A Session #2
24mDatabases beyond the tutorials
27mHow to be data-driven when you aren’t Netflix (or even if you are)
16mStraight from the Field
34mLightning Talks
35mWe're Marketers. If we can Learn Distributed Systems + K8s, so can you!