Turning Python into a Hydra, or How We Combined Multiprocessing with Kafka

on YouTube ↗30m

" summary

Describes turning a Python application into a high-throughput worker system by combining multiprocessing with Kafka. Covers using Python multiprocessing features together with Kafka messaging to parallelize work and coordinate processes for scalable processing.

AI-written summary · about

" chapters (7)

1. 0:00Introduction
2. 4:46Overloaded task manager
3. 9:46Kafka mechanisms
4. 15:32Rebalance and reprocessing
5. 17:07At-most-once solution
6. 24:26Five advantages
7. 26:35Conclusion & Q&A

" related talks

• 40mUsing Kafka in Your Python ApplicationsAleksandr Tarasenko · Python fwdays'20 online conference
• 45mEvent-Driven applications: Apache Kafka and PythonFrancesco Tisiot · Berlin Buzzwords 2021
• 38mEvent-driven applications: Apache Kafka and PythonFrancesco Tisiot · Conf42 Python 2021
• 29mUnified batch and stream processing in pythonArthur Andres · PyData Global 2023
• 27mReal-time Event Streaming with PythonDave Klein · PyCon Italia 2023
• 31mPython Multithreading and Multiprocessing: Concurrency and ParallelismHitul Mistry · EuroPython 2015
• 26mEvent-driven applications: Apache Kafka and PythonFrancesco Tisiot · PyCon US 2021
• 36mConcurrency and Parallelism with Python for HumansSantiago Basulto · PyCon Latam 2019