Reordering 59 Million NYT Publishing Assets Using Go and BadgerDB
summary
Reorders 59 million NYT publishing assets stored on a single Kafka partition into correct chronological and topological order using Go and BadgerDB, shrinking the log to 35 million assets and enabling date-based scanning. Implements reordered 59 million events in-memory, compact encoding (167 bytes vs 777), a string→ID store for 22M URIs, a timestamp uniquing cache for pre‑1970 dates, and BadgerDB write batches for 9x faster writes; runs end-to-end and speeds iterations dramatically.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:37My Background
- 2:45T Publishing Assets
- 11:12Encode Assets and Insert Into Database
- 11:42Find and Resolve Missing References
- 12:54Find and Remove Duplicates
- 15:56T Development Philosophy
- 17:50Fast Performance (v 1.6.1)
- 22:17I'm New to Go
- 24:04T Badger Wrapper
- 26:04T Batch DB Wrapper
- 29:20TEncoding Strings via Enumeration
- 32:59T Marshal - Read
- 33:51T Timestamps as Keys
- 34:43Refresher: Endianness
- 35:18Endianness - Sorting
- 36:53T Unique Timestamp
- 38:27TUnique Timestamp Cache
- 39:46T Panic Error Handling - Goroutines
related talks
01h 27mFlash Talks
35mFast on paper, slow in reality: what we got wrong about performance
07mBadgers, Gophers and Graphs
15mTime Series and FoundationDB: Millions of Writes/s and 10x Compression in 2000 Lines of Go
48mBuilding Petabyte-Scale PostgreSQL Deployments
37mKEYNOTE: Data Architecture Turned Upside Down
20mWrites, 3 ways: Postgres, Apache Kafka® and Apache Iceberg™
57mLessons in Building a System that Processes More than 70 Billion Events Daily