Magnolify all the Things
summary
Magnolify is a Magnolia-based library that auto-derives type class instances used in large Scala data pipelines: Magnolia-derived Arbitrary for ScalaCheck (tested with Discipline), Cats instances like Eq and Semigroup, and converters to Avro/BigQuery/Datastore/Protobuf. Concrete fixes include shrinking the generator size per recursion to avoid stack overflows in recursive ADTs, injecting field index to disambiguate booleans, and replacing a Magnolia-derived Seed with one built from a Long to stop constant-seed failures. Avro schemas are stored as schema as string for Java 1.8 compatibility.
AI-written summary · about
chapters (27)
- 0:00Start
- 1:14Agenda
- 1:47Type classes
- 2:40Type dass derivation
- 3:54Magnolify Show derivation
- 4:50Combine - parameters of a case class
- 6:09Dispatch - subtypes of a sealed trait
- 9:30Magnolify Arbitrary derivation
- 10:22Arbitrary properties
- 11:35ERROR
- 12:01Generation size
- 12:39FAILED
- 13:37Recursive fallback
- 14:05Injecting fallback
- 14:36Cats Eq
- 15:28Cats Semigroup
- 16:36Testing with Discipline
- 17:28Arbitrary function
- 20:04Cogen properties
- 22:15One more bug
- 23:17Pseudo ramdom seed
- 24:13Fixing Seed
- 32:53Other Converters
- 33:55Magnolia vs shapeless
- 35:20Benchmark
- 36:19Other Fun Bits
- 38:19Related work
related talks
40mGallia: Practical Data Transformation in Scala
28mTHOSE 10000 CLASSES I NEVER WROTE
34mBuilding functional programs with bananas, catalysts, shacl's and shapes
38mEnabling Big Data and Machine Learning for the Masses...
15mMetamodelization and Automatic Generation of ML Pipelines
31mSummingbird: A Streaming Map-Reduce API for Storm, Hadoop, & More
41mIntroducing Cascalog: Functional Data Processing for Hadoop
18mDiscovering Types (from Strings) with Cats and Shapeless