Hybrid batch and stream processing

on YouTube ↗25m

" summary

XComponent Koordinator is a central catalog and REST API that lets business users compose light orchestration scenarios mixing batch jobs and streaming pipelines, claiming no-code integration rather than developer-only code. Implementation uses Kafka for stream links, a polling REST worker API, and a typed catalog; demo runs with a single-node Kafka and four workers downloading URLs into zipped results. Coordinator enforces input/output schemas, starts consumers before producers, supports retries and timeouts, and the live demo partly fails due to Kafka timing.

AI-written summary · about

" chapters (10)

1. 0:00Start
2. 3:15BUT THE TRANSITION IS NOT IMMEDIATE...
3. 5:21ORCHESTRATION X CHOREOGRAPHY
4. 7:27I OUR TECHNICAL SOLUTION: ILIGHT ORCHESTRATION
5. 11:27HIGH LEVEL ARCHITECTURE
6. 12:45DEMO SCENARIO
7. 16:12INSIDE THE KAFKA WORKER: CATALOG
8. 17:09INSIDE THE KAFKA WORKER: EXECUTION
9. 18:33DEMO: ON THE BUSINESS SIDE
10. 23:47CONCLUSION

" related talks

• 03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...F. Ramière, D Petisme · Devoxx France 2021
• 45mApplications in the Emerging World of Stream ProcessingNeha Narkhede · GOTO Chicago 2016
• 50mApplication development and data in the emerging world ...Neha Narkhede · Berlin Buzzwords 2016
• 33mKafka Microservices with Kafka plus Akka and Kafka StreamsBoris Lublinsky · Scala Days Berlin 2018
• 31mBridge to the CloudSteve Howard · Indy Cloud Conf 2020
• 55mDemystifying Stream Processing with Apache KafkaEwen Cheslack-Postava · Philly ETE 2016
• 40mEvent Streaming Applications with Kafka Streams, Spring Kafka and ActuatorKo Turk · Voxxed Days Bucharest 2022
• 47mFlowable business processing from Kafka eventsJoram Barrez, Tijs Rademakers · Devoxx Belgium 2019