Continuous Optimization for distributed BigData analysis
summary
Platform ingests about 1 million records per second and processes trillions per day using Hadoop and Presto. It replaces Parquet with a messagepack-based columnar format to cut network I/O and embed types, adds a PostgreSQL time-range index with GiST to push down file selection and provide transactional metadata, and implements a Presto-driven "stereo" connector that rewrites partitions continuously (default targets: ~1M records or 256 MB) and supports user-defined partition keys to enable co-located joins and reduce shuffle.
AI-written summary · about
chapters (16)
- 0:00Start
- 9:05Enterprise Data Analysis
- 9:46Arm x Treasure Data
- 12:46Distributed Processing Engines
- 15:52Distributed Plan
- 18:24Our Approach
- 19:18Columnar Storage
- 21:56MessagePack x Columnar File
- 25:11Time Index Pushdown
- 27:46Time-Range Partitioning
- 30:05Partition Size?
- 30:41Auto Optimization
- 31:12Repartitioning
- 33:54User Defined Partitioning
- 36:29Colocated Join
- 38:09Stella Connector
related talks
22mIntroduction to Presto: The SQL Engine for Data Platform Teams
51mPresto Open Source SQL Engine for Massive Scale Data Lakehouses
22mPresto for Your Fast, Open SQL Lakehouse
50mPresto: SQL-on-Anything using Kubernetes
49mPresto in the Cloud
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
32mSome Like It Hot
49mData Platform Architecture, Evolution, and Philosophy