Introduction to Apache Spark using Frameless

on YouTube ↗55m

" summary

Compares Spark DataFrame, Dataset and Frameless with concrete examples on 38,000 tweets pulled via Kafka and run in Databricks notebooks; shows DataFrame queries use Catalyst and can reorder filters, Dataset restores JVM objects via tungsten byte buffer codegen causing runtime casts and overhead, and some Dataset filters block optimizer rewrites; demonstrates needing a custom encoder for java.sql.Timestamp and claims Frameless gives compile-time type safety without runtime penalties, plus reusable lazy jobs composable as actions.

AI-written summary · about

" chapters (17)

1. 0:00Start
2. 1:10Data Frames Api
3. 4:33The Data Frames Api
4. 5:16Creating a Data Frame
5. 5:28Display Function
6. 8:18Query Language
7. 10:24Stack Trace
8. 13:41The Data Set Api
9. 14:37Create a Data Set
10. 25:22Data Frame Api
11. 29:07Predicate Push Down
12. 32:13Typed Data Set
13. 35:22Actions
14. 40:00And I'Ve Said Okay Instead of Pulling this into a Tuple Let's Um Let's Map the Tuple into an Instance of this Case Class Where the First Field Is Along in a Second Field Is You Know Is the Option String That I Want if You Use this Approach Then They Have To Line Up so It You Know that It's Almost like an Unapplied Right It Has To Line Up so that the You'Re Coming Up What's Coming out of Here Is a Tuple of Long and Option String so the Case Classes Order of Fields Are GonNa Have To Be Long and Option String Right So Has To Line Up in that in that Fashion but if You Do that with that Cast Then You Get Your Your Common Names Back
15. 41:47So Here's a Case Where I'M Going To Select this in the Opposite Order Okay So Now What I'M Doing out of this Select Is Producing a Tuple of Option String Long Instead of the Other Way Around and if I Try To Cast that into My Id Double Their Name I'M GonNa Get a Compile Time Error because the Field Order Is Not Consistent with the Field Order in the Case Class but At Least It's a Compile Time Error Even if It's a Little Hard To Read and the Same with the Projection Here the Names Don't Match Okay So I Put User Name Instead of User Screening
16. 42:57And Again Compare that to Here Where I'M Doing the Cast from the Data Frame to the Data Set Okay so this Is the Cast We Did Earlier Where I'M Trying To Make My Data Frame into a Data Set So I'M Selecting the Username You Use a Screen Name and the Id and I'M GonNa Cast It into this Here but It's Bad Again because the Username in the Case Class Doesn't Match the Column Name Which Is Used as Screen Name Sparc Is Also Using Names Here To Line these Things Up and the Obvious Question Is Well I Get a Compile Time Error or a Runtime Error Who Thinks Compile Time Error
17. 45:49This Is What I Talked about Earlier I Don't Want To Deal with All that Data for this Example I Only Care about the Hashtags the Text and the Screen Name so What I'Ve Done Is I'Ve Defined Us a Case Class Here That Has a Subset of the Fields That I Want and I'M Projecting Down to that Right So this Will Give Us a Little Bit Less to Little Fewer Columns To Work with that Actually Fits on the Screen Which Is Nice and Then Here I'M GonNa Say I'D Like To Add a Column Here

" related talks

• 12mLessons Learned: Ultra-Fast Feedback with sparkJonathan winandy · Scala.IO 2017
• 40mBest Practices and Optimization with the Spark EngineJohan Hille Bringsdal · Øredev 2024
• 45mApache Spark's New Declarative Streaming Pipelines in ActionFrank Munz · Big Data Conference Europe 2025
• 30mThe Future of Scala Performance on Big DataCarlos Abad · Scala Days Berlin 2016
• 38mCharacterizing the performance of Big Data processing platformsManish Singh · Data By the Bay 2016
• 43mBuilding a community fountain around your data streamMaria Patterson · PyData Seattle 2017
• 21mAnalyzing Massive Time Series Data with SparkSandy Ryza · Data By the Bay 2016
• 43mImproving PySpark Performance: Spark performance beyond the JVMHolden Karau · PyData Amsterdam 2016