Hadoop Vectored IO: your data just got faster!
summary
Hadoop adds a new Filesystem API called vectored read (targeted for Hadoop 3.4) that accepts lists of byte ranges and memory allocators so object-store clients can fetch multiple ranges in parallel. The S3A connector issues parallel GETs and coalesces ranges, and a modified ORC library demonstrates tangible wins—about 2x speedup on Hive/ORC in S3 benchmarks and up to 7x on local SSD microbenchmarks—shipping in Hadoop and working with existing apps after library updates.
AI-written summary · about
chapters (10)
- 0:00Introduction to Vector.io
- 1:11Challenges in data reading
- 2:38Columnar formats and IO bottlenecks
- 8:11The Vector.io API solution
- 11:14API implementation details
- 13:52Usage and integration
- 16:16Performance validation
- 22:19Cloud platform considerations
- 24:11Challenges and roadmap
- 28:11Q&A and concluding remarks
related talks
01h 03mThe Future of Apache Hive and Hadoop 2.0: Bigger, Faster, Stronger
33mThe advent of the open data lake
35mAccelerating Spark Workloads in a Mesos Environment with Alluxio
40mFast Access To Your Complex Data #bbuzz
02h 05mBig Data and the Data Lake from the Analyst Perspective
40mOpen Source Memory Speed Virtual Distributed Storage System
25mOvercoming I/O Bottlenecks in LLM Training with Open-Source Distributed...
31mHow Apache Iceberg Enables Multi-Engine Data Platforms