Using small data in the client instead of big data in the cloud
summary
Claim: ship complete Transit App schedules to the device and query them client-side so UI is instant and offline, avoiding round trips and server load. Implements compact formats: binary files instead of ᴊsᴏɴ, experiments with MessagePack, capn proto and FlatBuffers, and a custom named-struct encoder. Berlin feed shrinks from ~26 MB raw to ~2 MB with compression and to ~200 kilobytes with bit-level packing. Result: fast local queries, much lower server traffic, production success.
AI-written summary · about
chapters (10)
- 0:00Small data approach
- 1:50Transit App offline schedules
- 3:35Large transit data challenges
- 6:24Optimizing transit schedule data
- 9:31Compressing structured transit data
- 12:19MessagePack vs binary formats
- 18:02Cap'n Proto for binary data
- 24:48FlatBuffers bit-level control
- 31:55Final binary format results
- 34:26Q&A
related talks
33mConversational Big Data
30mCaching, Payloads, and Other Dark Arts
33mOptimization - Making your app faster
32mResponsible Javascript
43mFine-grained everything
36mBreaking The Bloat: Achieving The Fastest Navigations on The Web
45mReducing App Size: Moving Images and Localization Strings to the Server
51mFront End Data