INSTalytics: Cluster Filesystem Co-design for Big-data Analytics

on YouTube ↗28m

" summary

Describes Instalytics, a co-designed cluster filesystem and compute stack that partitions data simultaneously on four dimensions to increase partition pruning and reduce network shuffles for analytics. It customizes the standard 3-way replication into a heterogeneous replication layout (keeping the same recovery cost and availability) and adds a sliced-read API with coordinated request scheduling and selective caching so multiple compute nodes can read slices of a block efficiently. Implementation results in a production analytics stack show comparable recovery/availability to physical replication while improving query performance.

AI-written summary · about

" chapters (22)

1. 0:00Intro
2. 0:32Big-data Analytics: Motivation
3. 1:20Problem statement
4. 1:57Approach at a glance
5. 2:52Outline
6. 4:18But, utility is limited
7. 5:18Logical Replication
8. 6:13Are 3 partition orders enough?
9. 7:06Challenge: Recovery cost
10. 9:10Super Extents
11. 11:19Chained Intra-extent bucketing
12. 13:19Recovery Cost Evaluation
13. 14:08Recovery cost: Disk I/O
14. 14:40Recovery cost: Network I/O
15. 15:02Other storage challenges
16. 15:29Efficient Filter Queries
17. 16:35Join Queries: Heterogeneous co-location
18. 17:20Efficient Join Queries: Sliced Reads
19. 19:39AMPLab Big Data Benchmark
20. 21:10Production Queries
21. 21:42Implementation
22. 23:02Summary