The Essence of Online Data Processing
summary
Claims a first rigorous foundation called DON Calculus for online data processing that models backend and frontend behavior and centers on IOP and TLO. The key engineering insight models operations as operation streams flowing through node-stations, with three primitives (add, map, fold) that encode graph-style read/write patterns. The semantics is mechanized (~6,000 lines) and proves confluence and an observable-equivalence theorem: despite nondeterministic incremental propagation and temporal-local rewrites, results match conventional, nonincremental processing.
AI-written summary · about
chapters (12)
- 0:00Whole stack verification
- 1:04Online data processing systems
- 3:32Defining online processing
- 5:20Essential features of online systems
- 7:50Incremental operational processing
- 11:45Temporal locality optimization
- 13:54Unifying IOP and TLO
- 16:11Operation streams and semantics
- 18:19Formalizing IOP
- 20:06Deterministic results
- 22:34Mechanizing proof
- 23:37Conclusion and future
related talks
32mBig Data Processing : Where do we go next ?
47mAll the DataOps, all the paradigms
30mDevOps is to Infrastructure as Code, as DataOps is to...?
01h 26mFrom On-Ground to On-Cloud Data Engineering
03mDBTB INT Monal Daxini
36mDataEngBytes 2023 - PER-T1-02 - Josh Armitage
21mData-Oriented programming
30mDevOps to DataOps