H2O internals
summary
H2O is a pure-Java cluster runtime that stores compressed columnar data in-heap, runs most analytics at memory-bandwidth speeds and uses a MapReduce-style MRTask API for single-threaded chunk loops that auto-parallelize and auto-distribute. It delivers ~200ns K/V ops and chunk compression typically 2–4x smaller than gzip on disk, with a custom serializer faster than protobuf/kryo. Benchmarks claim 7 TB in 90s logistic regression and about 10x faster than Spark for similar workloads; the system supports 500GB+ heaps and exact-consistency distributed K/V.
AI-written summary · about
chapters (10)
- 0:00Introduction to H2O internals
- 2:24Memory bandwidth bound performance
- 4:50Data layout and vectors
- 9:02MapReduce paradigm in H2O
- 13:38Coding examples and speed
- 19:39Cluster setup and resource management
- 21:01Serializer and network protocols
- 24:33Key-value store and caching
- 27:03Front-end and API integration
- 32:46Summary and Q&A
related talks
50mScalable Data Science & Deep Learning with H2O
45mMassive distributed processing with H2O
01h 10mGetting started with H2O on Python
33mProductionizing H2O Models with Apache Spark
32mDifferent Strategies of Scaling H2O Machine Learning on Apache Spark
30mScalable Automatic Machine Learning with H2O
27mDeploying distributed AI and machine learning in financial services
44mScalable Automatic Machine Learning with H2O