MilliSort and MilliQuery: Large-Scale Data-Intensive Computing in Milliseconds
summary
Benchmarking shows flash-burst analytics are feasible: MilliSort sorts 0.84M records in 1 ms using 120 servers by using recursive hierarchical partitioning to avoid a central sampler, and MilliQuery processes 0.03–48M items in 1 ms across 60–280 servers for three representative SQL queries. Prototype runs on 40-core, 100Gbps nodes with a RAMCloud transport (kernel bypass, 5 µs RTT) but is limited by per-message costs; small-message throughput, inefficient shuffles and coordination overhead are the primary scalability bottlenecks, and total throughput grows roughly quadratically with the time budget.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:38Contributions
- 3:02MilliSort Algorithm
- 3:54Challenge of Partitioning
- 4:51MilliQuery
- 5:22Experiment Setup
- 6:07Overall Performance
- 7:10Quadratic Scaling w/ Time Budget
- 8:29Why not more servers?
- 9:56Discussion
related talks
25mSpeed Wins: The Basics Of How To Push More Queries Through Each CPU Core
35mScaling Parallel Algorithms to Massive Datasets using Multi-SSD Machines
26mPetabyte Scale Datalake Table Management with Ray, Arrow, Parquet, and S3
24mData infrastructure to build bigger with less
01h 06mBest practices when accessing Big Data or any other data!
39mReal Time Datamining and Aggregation at Scale
16mBurst Computing: Quick, Sudden, Massively Parallel Processing on Serverless...
43mArchitecting for Speed & Scale