DataEngBytes 2023 - MEL-T2-05 - Paul Brebner
summary
A simulated drone-delivery system uses Cadence workflows for stateful drone and order logic, connects those workflows to Apache Kafka topics for event coordination, and streams hourly aggregates into TensorFlow for online learning. The demo runs 2,000 simulated drones (4,000 concurrent workflows), computes waypoints every 10s, and reliably recovers from activity failures. Batch training hits ~0.88–0.9 accuracy, Kafka-based incremental learning drops to ~0.72, concept drift causes sharp accuracy falls unless models are reset or multiple models are employed.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 2:07What is Cadence architecture
- 5:18Use cases workflow orchestration
- 7:27Drone delivery workflows
- 11:22Performance testing at scale
- 13:29Real-time ML with Kafka TensorFlow
- 16:21Incremental learning models
- 23:45Key findings on incremental learning
- 25:11Q&A
related talks
25mSpinning Your Drones with Cadence Workflows and Apache Kafka
53mData Engineering and AI
21mDataEngBytes 2023 - SYD-T2-04 - Suneeta Mall
21mDataEngBytes 2023 - SYD-T1-06 - Sana Sanai
52mMachine Learning Data Pipelines
33mFeeding the Cloud: Designing Data-Driven Robotics Systems
30mArchitecting Applications With Multiple Open-Source Big Data Technologies
01h 17mData Engineering in the AI Era: Building a Robust Foundation for AI Applications