An Exploration in Storing Telemetry in Cloud Object Storage
summary
Shows building a telemetry data lake using NDJSON, Parquet, DuckDB and Apache Iceberg with S3 object storage and a small Go CLI to ingest and benchmark. Converts ~10 million log lines from NDJSON (4.4 GB) to Parquet 390MB (≈91% reduction), demonstrates naive JSON scans read ~42 GB to extract ~150 MB of fields, and contrasts ~60s JSON scans with 0.02s query on Parquet via DuckDB. Implements Iceberg for table-level snapshots/partitioning and highlights costs, metadata churn, and encoding/CPU vs IO tradeoffs.
AI-written summary · about
chapters (18)
- 0:00Telemetry storage challenges
- 3:26Cloud object storage for telemetry
- 6:56Building telemetry data lake
- 8:43JSON inefficiency for queries
- 9:04Parquet for telemetry storage
- 12:37Parquet encoding techniques
- 14:06Converting to Parquet format
- 15:11Querying Parquet with DuckDB
- 17:40Managing many Parquet files
- 18:47Table formats for data lakes
- 21:15Iceberg for telemetry data
- 23:41Creating Iceberg table demo
- 28:46Future telemetry challenges
- 29:48Hardware compression limits
- 31:37GPU processing constraints
- 33:01Simpler metadata formats
- 34:27Cloud storage cost optimization
- 35:30Recommended papers and closing
related talks
22mData Exploration Made Easy with Open Source
28mDataEngBytes 2023 - PER-T1-04 - Nathan Glover
28mAnalyze data easily with duckdb - and the implications on data architectures
13mBuild your open data lakehouse with Dremio and AWS (PRT085)
30mOLAP on Your Cassandra Data with Arrow, Flight SQL, ADBC, and DuckDB
26mBuilding Data Lake Using DuckDB
25mRemove Slow, Expensive ETL & Bring Data To The Data Lake For Lower Cost, More Performant Analytics
29mServing a multi-platform Data Lake via Delta, Iceberg, Unity and Glue