Continuous Intelligence: A Sea-Change in Streaming Data Processing
summary
Claim: continuous intelligence processes streaming data in-memory to analyze, learn and predict on the fly rather than store-then-analyze. Implementation: an actor-style system (called Swim) runs web agents as containers on Kubernetes, communicating over websockets, using Java/Python, small DNNs trained online with backprop. Scale examples: a mobile carrier case processes 4 petabytes per day for >100 million devices and tens of thousands of base stations, cutting infrastructure from 400 nodes to 40 nodes by keeping state in memory.
AI-written summary · about
chapters (7)
- 0:00Introduction to Continuous Intelligence
- 2:28Why Store-and-Analyze Fails
- 7:33Stateful Computing Model Explained
- 11:24Swim Architecture and Web Agents
- 16:27Building Active Graph Applications
- 17:52Where to Run Swim Applications
- 20:20Summary and Key Takeaways
related talks
32mBig Data Processing : Where do we go next ?
09mInterview with Jack Norris, MapR SVP Data & Applications
50mData Science in the Enterprise
53mIntroduction to data streaming
29mLakeside AI: Bringing Models to Data with Open Lakehouses & Agents)
29mData Mesh Streaming Patterns: A Real-time Perspective
06mJay Kreps interview
41mStreaming: an Alternative to CRUD & Batch Processing