A Short Summary of the Last Decades of Data Management
summary
Claim: relational systems keep winning and will "eat" key value, time series, vector and many other niches. Evidence: DuckDB 1.0 released, TPC‑H SF100 benchmark on a 32‑core 64GB machine with ~40GB compressed data shows DuckDB is ~7x faster than Spark on EMR and remains faster even with 32‑node scaleout. History notes: IMS 1968, System R 1973, Oracle 1979, Postgres 1996, MapReduce → Hive → Spark → Databricks, MongoDB, Cassandra. Core triad: tables SQL ACID.
AI-written summary · about
chapters (17)
- 0:00Intro
- 1:36DuckDB
- 2:38Ancient history
- 11:25One size fits all?
- 19:05No SQL!
- 22:39Hadoop
- 24:02Hive
- 25:45MongoDB
- 27:09Cassandra
- 28:55Apache Spark
- 30:54Tables, SQL, ACID
- 33:28New SQL!
- 35:21Relational eats everything
- 42:55What refuses to be eaten?
- 43:57Big data is dead?
- 48:24Summary
- 49:20Outro
related talks
51mMotherduck: Data Analytics in the Post Big-Data Era
27mThe Duck and the DataFrame: A Data Engineer’s Journey with DuckDB
30mRetooling for a Smaller Data Era
34mData Wrangling with DuckDB
40mDuckDB: Harnessing in-process analytics for data science and beyond
28mAnalyze data easily with duckdb - and the implications on data architectures
27mIn Process Analytical Data Management with DuckDB
24mData infrastructure to build bigger with less