Gallia: Practical Data Transformation in Scala
summary
Gallia is a Scala library (v0.4.0 for 2.12/2.13) that performs schema-aware meta-validation so type errors are caught before any data is processed, infers or accepts schemas (case classes), and runs transformations both in-memory and at scale via a Spark RDD fallback. Live demo shows TSV/CSV/JSONL I/O, splitting semicolon/colon fields into nested records, grouping donors, and reading/writing Avro/Parquet support. It also offers union types, co-transformations, and an iterator/merge-sort mode for non-Spark scaling (claims ~10% of datasets need true scaling).
AI-written summary · about
chapters (18)
- 0:00Introduction
- 1:40What is Gallia
- 2:40Getting started
- 3:14Goals
- 4:35When to use it
- 5:46Team behind Gallia
- 6:02Case classes
- 8:18IO support
- 9:45Target selection
- 11:06Live coding
- 15:31Adding additional data
- 18:37Transformation
- 23:39Basic processing
- 24:14Spark Context
- 25:56Writing RDDs
- 29:38Scaling
- 35:11Summary
- 37:30Feedback
related talks
23mSpark Datasets: why they aren't great and what could be done
47mScala: The Unpredicted Lingua Franca for Data Science
47mScala: The Unpredicted Lingua Franca for Data Science
46mTame Your Data with Akka Streams
39mComplex Data Transformations Made Easy
28mAutomating API Busywork with Scala
47mGrackle GraphQL server
28mTHOSE 10000 CLASSES I NEVER WROTE