Literate Stream Processing

on YouTube ↗36m

" summary

Shows how to write production Kafka Streams services in Clojure using literate programming in org-mode to keep docs and runnable code together, building a microservice that updates loan state and calls SBA via Jackdaw + Kafka Streams. Uses specs to generate Avro, a spec->avro compiler, structured JSON logging, Prometheus metrics, and a Test Machine wrapper around the topology test driver so unit, multi-topology integration and full-system tests run without a live Kafka cluster and the demo topology passes end to end.

AI-written summary · about

" chapters (9)

1. 0:00Introduction and speaker background
2. 5:51Literate programming with Org mode
3. 11:31Kafka streams introduction
4. 14:20Deploying Kafka streams apps
5. 15:01Testing Kafka streams applications
6. 23:02Structured logging and metrics
7. 26:15Production app demo overview
8. 33:41Integration testing with Test Machine
9. 35:47Conclusion and references

" related talks

• 46mTame Kafka streams with ClojureAndrea Crotti · Lambda Days 2018
• 58mGame of StreamsMark Heckler · Devnexus 2020
• 52mStream Processing with the Spring Framework Like You’ve Never Seen It BeforeJosh Long, Viktor Gamov · Spring I/O 2019
• 33mKafka Microservices with Kafka plus Akka and Kafka StreamsBoris Lublinsky · Scala Days Berlin 2018
• 01h 11mBootiful Stream Processing!Viktor Gamov · Java Z-Day 2021
• 44mKafka based Microservices with Akka Streams and Kafka StreamsDean Wampler · Scala Days New York 2018
• 57mLessons in Building a System that Processes More than 70 Billion Events DailyMorri Feldman · Clojure/north 2019
• 44mFrom REST to CQRS with Clojure, Kafka, & DatomicBobby Calderwood · Clojure/conj 2015