Lessons in Building a System that Processes More than 70 Billion Events Daily

on YouTube ↗57m

" summary

Company processes 70B events/day through an event-driven Kafka pipeline that validates HTTP SDK messages on web tier, transforms them in small Clojure microservices, and persists streams to S3 as parquet for batch analysis while feeding ClickHouse for real-time queries and Aerospike as the in‑memory key/value lookup for attribution. Migration from Python to Clojure solved concurrency limits but created serialization headaches—pickle to JSON then shimmed—leaving a persistent JSON vs Protobuf schema tension that complicates column selection, backward compatibility, and team coordination. Deploys in Docker on AWS with spot instances and mesos/k8s staging.

AI-written summary · about

" chapters (9)

1. 0:00Start
2. 12:41Toku Database
3. 22:09Python Pickled Messages
4. 24:09Architecture
5. 27:22Json Deserialization
6. 35:24Micro Services Architecture
7. 36:10Mono Repo with Multiple Sub Modules
8. 53:11How Do We Test Things
9. 56:04Reason That We'Re Using Spark

" related talks

• 38mHow to handle 100k events per secondNiels Basjes · Berlin Buzzwords 2019
• 34mChasing the Stream Processing UtopiaKartik Paramasivam · Strange Loop 2018
• 47mSpeeding Event Processing to 1.6M/sec: Badoo’s ExperienceAleksandr Krasheninnikov · HighLoad++ Moscow 2018
• 50mMoving from legacy to event-driven with KafkaJeroen van der Gulik · Dutch PHP Conference 2018
• 32mBuilding a Real-time Ingestion Platform using Kafka and AerospikeMatteo Pelati · Aerospike Virtual Summit ’20
• 48mHyper Speed! When Big Data BloomsScott Helme · NDC London 2023
• 20mLessons learned scaling our SaaS on Postgres to 8+ billion eventsJonathan Denney · Citus Con: An Event for Postgres 2022
• 33mLightning talksKM tman · Clojure/conj 2014