Lightning Talk: A Fast, Concurrent Data Loader for Time-Series Data

on YouTube ↗05m

" summary

A concrete design that reads ~20,000 files (10 GB) by timestamp and aggregates heterogeneous streams uses three levels: data loader, node loader, and data reader with a loading thread per reader feeding a lockless q sized to bound memory; conditional variables coordinate consumption so frames are freed when used. Measured results drop load time from 87% of runtime to ~1.5%, delivering a 60x speedup and a 10x program speedup that enables CI runs and concurrent simulations.

AI-written summary · about

" chapters (4)

1. 0:00Introduction
2. 1:11What is a data loader
3. 2:22The design
4. 3:52Results

" related talks

• 07mLightning Talk: Scaling Time Series Databases on KubernetesKen Ahrens · Data on Kubernetes Day North America 2023
• 33mLightning talksKM tman · Clojure/conj 2014
• 06mLightning TalkRaghu Reddy · Kubernetes Community Days Bengaluru 2023
• 56mLightning TalksMark Wong, Chris Ellis, Jelte Fennema-Nio, Claire Giordano, Neeta Goel, Hannu Krosing, Christoph Berg, Di Qi, Enes Cakir, Raoul De Guchteneere, Marat Bogatyrev, Eren Basak · PostgreSQL Conference Europe 2024
• 13mLightning Talk: Highlights of the Snowflake Elastic Data WarehouseTorsten Grabs · FoundationDB Summit 2018
• 32mLoading at Hyperscale: Advanced Data Import Techniques for CitusColton Shepard · Citus Con: An Event for Postgres 2022
• 48mHyper Speed! When Big Data BloomsScott Helme · NDC London 2023
• 30mScaling Time-Series Data to Infinity: A Kubernetes-Powered Solution with EnvoyHiroki Sakamoto · KubeDay Japan 2024