Mixing data streams at scale
summary
Shows a production design that combines Kafka firehose, Python workers, Ably and Aerospike durability layer to deliver personalized streams for millions of mobile users. Claims Ably provides edge publish guarantees with global edge points (~200) and ~65 ms median edge latency, and Aerospike holds transactional subscription state so updates reach users within ~100 ms. Describes tradeoffs: per-user topics vs per-topic fanout, consumer-group rebalances, a Taylor Swift spike three orders of magnitude above normal, and replacing volatile Redis caches with durable records.
AI-written summary · about
chapters (9)
- 0:00Start
- 0:26Quick Agenda
- 2:14Mass Personalization
- 3:11Personalized Websites and Dashboards
- 7:01What Makes these Apps Feel Good
- 7:38Predictability
- 8:43Microservices
- 12:25Appsync
- 21:18Fraud Alerts
related talks
23mAdobe + Aerospike: Real-time Streaming Segmentation
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
05mHighlights From Aerospike APJ Summit 2023
08mReal-Time Data Systems: Speed, Scale, and Resilience
14mThe Streaming Plane
43mBuilding a community fountain around your data stream
48mRebuilding a RT Datastore: the design, deployment & performance story
33mExponential Growth Without Sacrificing Platform Speed