Flare: Optimizing Apache Spark for Scale-Up Architectures and Medium-Size Data
summary
Flare is an accelerator module for Apache Spark that compiles DataFrame/SQL plans into low-level C code generation binaries that bypass JVM/RDD iterator overhead, yielding order-of-magnitude speedups on single big servers and medium-size datasets. Evaluation uses TPC-H and real workloads on 10–12 GB in-memory datasets, shows dramatic drops (example demo: ~90 s Spark run to sub‑second execution after compile), includes NUMA-aware optimizations and direct TF XLA/TF compile integration so TensorFlow models run without costly copies.
AI-written summary · about
chapters (9)
- 0:00Introduction to Flair
- 0:49Spark and data processing
- 3:54Demonstrating system performance
- 8:00Flair acceleration techniques
- 10:07Technical design and optimization
- 14:30Performance benchmarks
- 18:35NUMA-aware optimization
- 24:00TensorFlow integration
- 30:25Conclusion and Q&A
related talks
48mFlare: Scale Up Spark with Native Compilation and Set Your Data on Fire
28mAn Apache Spark query's journey through the stack
40mBest Practices and Optimization with the Spark Engine
40mScaling with Apache Spark (or a lesson in unintended consequences)
41mOptimising Apache Spark and SQL for Improved Performance
28mAI-driven query plan optimization in big data processing tools
36mScale your data, not your process: Welcome to the Blaze ecosystem
13mLightning Talk: Highlights of the Snowflake Elastic Data Warehouse