Scalable data pipelines with shapeless and cats

on YouTube ↗35m

" summary

Shows how to build compile-time assembled data pipelines using Cats and Shapeless: pipelines become sealed ADTs with HLists for static lists, Coproducts for branching results, and phantom-type guards so selection is a type, not runtime strings. Uses Cats Functor (and a bounded-functor + ID wrapper to support Spark Datasets with Encoder bounds), encodes induction with implicits and a simple implicit-anyval ops operator to let client code write pipeline1 :: pipeline2 :: pipeline3.

AI-written summary · about

" chapters (15)

1. 0:00Intro
2. 0:50Deck
3. 1:10Simple pipeline
4. 4:12Convert function
5. 7:17Abstracting
6. 10:35Pipeline Switching Mechanism
7. 11:05Pipelines with constants
8. 17:08Heterogeneous lists
9. 17:57Categorical duals
10. 20:31Type annotations
11. 21:27Implicit induction
12. 22:43Pipeline ops
13. 24:27Why did we make it a death
14. 26:23BoundedFunctor
15. 27:31Recap

" related talks

• 18mDiscovering Types (from Strings) with Cats and ShapelessJonathan Merritt · Typelevel Summit Oslo 2016
• 30mSMART DATA PIPELINES FOR AIŁukasz Żuchowski · Scalar Conference 2023
• 36mKittens - datatype-generic functional programming with ScalaKailuo Wang · Strange Loop 2016
• 31mCool toolz in the Scalaz and Cats toolboxesJan Pustelnik · Scalar 2016
• 32mReproducible Data Pipelines Using Controlled EffectsYves Parès · Scale By the Bay 2020
• 43mData Pipelines à la ModeTom Hall · Code Mesh LDN 2019
• 28mAdopting CatsRon Aharoni · Scalapeño 2018
• 57mKittens: Shapeless Typeclass Derivation for CatsMiles Sabin · Scala World 2015