Dataflow Engines for Executing Cloud Applications: a Maslow's Hammer?
summary
A compiler turns imperative Python into a dataflow graph and runs it on a streaming engine, removing all failure-handling code from application logic and delivering exactly-once processing. The prototype is under 4k lines (<4000 lines), targets Apache Flink, and demonstrates local recovery in ~3s (local recovery ≈3s). Classroom attempts to assemble the same system by hand never reach 10,000 TPS (none do), and current work focuses on transactions, auto‑scaling, and making the model safe for general apps.
AI-written summary · about
chapters (11)
- 0:00Cloud apps and dataflow engines
- 1:24Scalable cloud development today
- 9:07State management and fault tolerance
- 17:07Master students' scalable app failures
- 22:22Two generals problem limits
- 26:57Hiding failures in streaming
- 33:46Dataflow engine limitations
- 37:35Long recovery times solution
- 39:01Compiling imperative to dataflow
- 47:18Future of dataflow engines
- 48:44Q&A
related talks
20mDataflow Engines for Executing Cloud Applications: a Maslow's Hammer or Natural Outcome?
42mApache Flink deep-dive #bbuzz
38mDeploying Stateful FaaS on Streaming Dataflows
24mUnder Pressure: Applying ML in Real Time
34mA journey of scaling high-volume low-latency stateful processing
38mDistributed Processing for Machine Learning Production Pipelines
01h 02mHydroflow: A Compiler Target for Fast, Correct Distributed Programs
35mApache Flink 2.0: Unified Enterprise Data Processing System and Beyond