Taming Billions of Rows: Data Lifecycle Management Lessons From a Cloud Native Database…
summary
Cloud-native migration to TiDB was blocked by dormant rows until a lifecycle strategy moved 40% of a multi‑billion‑row table into BigQuery via CDC→Kafka→Flink, Argo workflows and GCS snapshots, using a database reliability “data_updator” to batch-delete with rollback. Initial TiDB cost estimate was nearly 2× too low; after archiving the on‑prem MySQL footprint shrinks 2.3× (MySQL) and 36% for the largest TiDB table, unblocking the migration and cutting storage and backup costs.
AI-written summary · about
chapters (14)
- 0:00Introduction
- 1:05Why we needed migration
- 3:36Database selection criteria
- 5:43Migration challenges
- 7:33Historical data impact
- 10:00Data access patterns
- 12:09Technical and organizational challenges
- 15:16Offloading pipeline design
- 18:09Archive data tier
- 20:23Validating archived data
- 21:48Rollback safeguards
- 23:37Results and savings
- 24:23Lessons learned
- 26:48Q&A
related talks
29mDistributed SQL How TiDB Modernized Data Management
29mDistributed SQL: How TiDB Modernized Data Management
47m10 Reasons Why the TiDB Database Should Power Your Next Development Project
22mMigrating from Monolithic to Cloud Native Distributed SQL Database with Zero Downtime
35mA Customer’s Journey: Migrating from NoSQL to a Cloud Native Distributed SQL Database
55mThe Future of MySQL
25mData Demons: Unleashing Chaos or Driving Performance?
08mScaling and innovating with databases