Building a Data Pipeline with Clojure and Kafka
summary
Builds a resilient pipeline that streams DB changes into Redshift and Elasticsearch by sending trigger-produced events from Postgres into Kafka and processing them in Clojure actors (Quasar/Pulsar). Shows pgq loses queued events after ~1.5 hours, Kafka holds data for configurable windows, and observed throughput of 250,000–300,000 messages/sec. Implements per-shard partitions, per-merchant actors, rewindable replay for outages, and production-tuned GC/heaps — system works and recovers by rewinding Kafka rather than redoing DB work.
AI-written summary · about
chapters (8)
- 0:00Start
- 5:02Search and Reporting Infrastructure
- 22:53Abstractions
- 25:42Concurrency Mechanisms
- 27:16Using Agents
- 38:12Future Uses
- 38:48Real-Time Fraud Monitoring
- 39:12Report Generation
related talks
31mOne Data Pipeline to Rule Them All
01h 11mChange data capture pipelines with Debezium and Kafka Streams
43mBuilding Scalable and Extendable Data Pipeline
46mClojure Scaling the Event Stream
31mBridge to the Cloud
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...
46mKafka Streams at the Big Data Crossroads at Lightning Speed
30mReconciling Kafka, SQL and Parquet