Rethinking Relational Database CRUD Design Patterns Since We Live in a Big Data World
summary
Argument: columnar analytics work best if production systems stream every change as immutable events and analytics store keeps full history while a snapshot is derived. Practical stack: read MySQL/Postgres binlogs into Kafka binlog daemon, push directly into Vertica (no staging) to avoid update/delete churn; run a 40-node Vertica cluster in production, Vertica free tier is three nodes/1TB. BigQuery was append-only until recently and charges extra for updates. Snapshot techniques: SQL window functions or Vertica top-K projection for fast latest-row views.
AI-written summary · about
chapters (25)
- 0:00Intro
- 2:45Relational Database Index
- 4:06Relational Database Columnar
- 7:08Columnar Store
- 7:52Updates and Deletes
- 9:38Change History
- 13:53RM Logging
- 14:39Maintaining Change History
- 15:34Sending Data to Analytics
- 16:25ETL Pattern
- 17:47Better Workflow
- 20:33Data Analysts Use Cases
- 23:11General Solution
- 24:50Vertical Solution
- 28:01Quiz
- 30:26Burgi
- 33:03Postgres
- 34:27Binary Logs
- 35:53Code is Craft
- 36:50Duplicate Data
- 39:39Address Table
- 40:42Snapshots
- 41:22Daily Snapshot
- 42:27Path to Data
- 44:42Merge Join
related talks
20mVertica & Column Stores
28mRelational Databases Don't call it a comeback!
58mUnderstanding Big Data for Software Engineers
46mBig Data. Or Just Too Much Data?
35mQuestioning Database Claims: Design Patterns of Storage Engine
52mData storage for modern applications - Mike Lehan
29mWide Event Analytics
26mColumnar Storage: Redefining Data Management for the Modern Era