Data-centric Metaprogramming
summary
A Scala compiler plugin lets code keep shape like Vector[Employee] while swapping in custom memory layouts via annotated scopes that the compiler rewrites and inserts conversions automatically, using the code in github.com/miniboxing/ildl-plugin and documented at scala-ildl.org. Concrete results include columnar layouts giving up to 5x in microbenchmarks, making tuple-like allocations behave as stack values for ~14x, a prototype Spark port that reports ~45% gains and a claimed 40% speedup on Spark, while handling open‑world subclasses, overrides and fallbacks with warnings.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 0:52Motivation
- 2:15The problem
- 5:03The transformation
- 7:52Open world
- 11:16Representations
- 13:01Transformations
- 16:44Applications
- 24:06Conclusion
- 25:06Inefficient representation
- 27:50Bypass methods
- 28:36Performance
- 30:21Architectures
- 32:36DollyLinker
- 35:16Limitations of metaprogramming
related talks
21mA Typed Multi-Level Datalog IR and its Compiler Framework
41mHow types can turn a SQL interpreter into a SQL compiler
36mData pipelines engineering made simple with Scala
34mLightweight Relations
41mThese 10000 classes I never wrote
21mScala(ble) High-Performance Computing
47mApache Spark Lakehouse Pushdowns with Transpilation
30mPrograms as Values Pure Composable Database Access in Scala