FUNCTIONAL PROGRAMMING AND KAFKA: DO THEY MIX?

on YouTube ↗26m

" summary

Functional Scala integrates with Kafka using concrete libraries and patterns: examples use FS2 Kafka with cats-effect resources, reuse a single producer, and convert consumer.records into an FS2 stream that mapAsync processes with concurrency 10 to produce and then commit offsets. The demo builds an enrich-and-publish pipeline that composes producer+consumer resources, runs an HTTP server and the Kafka pipeline concurrently with racePair/fibers, parallelizes partitions via partitionStream + powerJoinUnbounded, and recommends commitBatchWithin (500 msgs or 15s) to avoid frequent commits.

AI-written summary · about

" chapters (12)

1. 0:00Start
2. 1:22Kafka Overview
3. 2:11Kafka Architecture
4. 3:05Kafka Workflow
5. 8:16Kafka Summary
6. 9:06Why Scala & Streaming?
7. 11:49Scala & Kafka - Creating Producer
8. 12:25Scala & Kafka - Creating Consumer
9. 12:44Scala & Kafka - Combining Resources
10. 20:13Performance tips
11. 20:41Parallel partition processing
12. 22:12Batching offset commits

" related talks

• 29mKafka on Kubernetes: Does it really have to be the hard way?Victor Gamov · Montréal Cloud Native Day 2019
• 30mCloudflow: Spark, Flink, and Akka Working together on KubernetesNolan Grace · Scale By the Bay 2020
• 40mKafka, DevOps...And Resilience for AllJames Bowkett · GOTOpia Chicago 2021
• 26mProduction Grade Kafka on K8sAnand Iyer · Kubernetes Community Days Chennai 2022
• 24mMake Your Kafka Cluster Production-ReadyJakub Scholz · Data on Kubernetes Day North America 2023
• 03mjoins Bert Jan Schrijver during Devoxx BE 2019 to talk Kubernetes & KaftaKatherine Stanley, Bert Jan Schrijver · Devoxx Belgium 2019
• 46mKafka Streams at the Big Data Crossroads at Lightning SpeedFrançois Sarradin, Jérémy Sebayhi · Devoxx France 2022
• 28mTackling Kafka, with a Small TeamJaren Glover · SREcon19 Americas