Spark for Scala Developers

on YouTube ↗01h 36m

" summary

Loads public UK CSV crime dumps into Databricks notebooks, infers schema, cleans column names, writes to parquet files and runs analyses with Spark DataFrame APIs and Datasets; ETL handles ~6 million rows, finds bicycle thefts peak in September, samples 400 points for a Google Maps iframe, and trains a linear regression pipeline (one‑hot months → features) to predict 2017 from 2015–16 — predictions track trends but are unreliable. Also includes working demos of structured streaming and notebook visualizations.

AI-written summary · about

" chapters (31)

1. 0:00Start
2. 2:27Github Repository
3. 7:14Hadoop File System
4. 10:44Data Frames
5. 11:16Minimum Definition of a Data Frame
6. 12:20Data Frame
7. 19:09Schema Inference
8. 19:36Infer the Schema
9. 24:58Cassandra Adapter
10. 25:15Data Frame Writer
11. 31:57Schema Inference on a Json
12. 46:43Runtime Error
13. 47:39The Datasets Type
14. 50:48In Memory Format
15. 52:48First Year Computer Scientist Approach
16. 1:01:49Statistical Sampling of the Data
17. 1:05:32Display Html
18. 1:08:50Machine Learning
19. 1:12:42And Then You Run the Test Set through the Model That You Built and You Checked the Predicted Values against the Real Values Right How Close Are They We'Re Not Actually Going To Do that Here because I'Ve Already Got a Test Set Right that's into 2017 Data I'M Trying To See How How Valid Is a Prediction Made from 2015 into 2016 against 2017 Data but that's Essentially We'Re Doing the Same Thing except We Already Have Datasets That Are Taken Apart and We'Re GonNa Look at the 2017 Predicted and Real Values When We'Re Done with this
20. 1:13:33So We'Re GonNa Do that Same Winnowing Down We Did before I'M Only Interested in this Particular Crime Type and by the Way this Is Where Sparc Does Something Interesting because if this Were Rdd's It Would Do the Join before this Filter Kind Of Dumb To Do the Join before the Filter Right because I'M Joining a Whole Bunch of Data Together Incurring Potential Network Overhead Only To Throw Away All the Records on One Side of the Join That Don't Match this Crime Type this Is Where the Query Analyzer Comes in It Will Actually Look at this and Say You'Re You'Re Ordering Doesn't Make any Sense and It Will Literally Move the Filter for Me
21. 1:16:02I Could Do Is I Could Just Do a Where Clause a Filter on all of the Training Data and Just Filter It Down to the First Six Months I Don't Want To Do that because I'M Actually Losing Data that Way Right I Want As Much Data To Train On as I Possibly Can Have So Rather than Do that I'M Actually GonNa Pad Out the the Test Data so that It Has the Right Number So Here All I'M Doing Here Is I'M Basically Creating this Case Class with a Month and a Count of Zero So I Know that I Don't Have Seven through Twelve so I'M Making an in-Memory Array from 7 to 12 with a Single Row That Has a Zero in It and Then I'M Creating a Data Frame out of that in Memory Array so You Can Actually Create a Data Frame Not out of a File You Can Create It out of Date a Memory That's in the Driver
22. 1:17:43I'M Basically Just Doing a Concatenation Here so that I End Up with a Test Set That Has all Months Represented Even though the Final Ones Have Nothing in Them All Right Having Done this I Now Can Start Doing the Work the First Problem I Have Is that a Linear Regression once Continuous Data and this Is Not Continuous It's Discrete Right so We'Re GonNa Use a Technique Called a One Hot Encoding Which Allows You To Basically Take a Discrete Feature and Encode It in Such a Way that It Can Be Interpreted Linearly so that I Can Pass It into this Linear Regression Now this Is Where Sparks Newest Approach and this Is Not Necessary Anymore
23. 1:18:04So We'Re GonNa Use a Technique Called a One Hot Encoding Which Allows You To Basically Take a Discrete Feature and Encode It in Such a Way that It Can Be Interpreted Linearly so that I Can Pass It into this Linear Regression Now this Is Where Sparks Newest Approach and this Is Not Necessary Anymore the the Old Version of the Spark Machine Learning Libraries That Worked Off Our Dds every Single Algorithm Worked Completely Differently so You Might Learn How To Use the Linear Regression and Then You Go Look at the Decision Trees and Nothing Makes any Sense because the Api Is Completely Different for that One
24. 1:18:30So You Might Learn How To Use the Linear Regression and Then You Go Look at the Decision Trees and Nothing Makes any Sense because the Api Is Completely Different for that One So When They Move to Data Frames They Started To Move toward an Approach That Is At Least Roughly Similar for all of Them It Follows More of a Builder Pattern and Then There's a Pipeline Approach and They Basically Break Things Down into Transformers and Estimators a Transformer Takes a Data Frame in Does some Sort of Transformation
25. 1:19:23That's another Transformer It's GonNa Add Term Frequency in There Then You Could Add another Transformer That Does Inverse Document Frequency Right because the Word Thus Shouldn't Be Counted High Even if It Appears a Lot Right and You'Re GonNa Chain All those Together that's Transformers They'Re Basically Just Taking Data Frames and Adding New Feature Columns to Them as You Go Along and Then You Get to the Estimator and the Single Thing about an Estimator Is It Has a Fit Call so You'Re Basically Saying Okay Now I Want To Take the Data and Fit It into this Model Use the Data To Fit the Model To Build the Equation
26. 1:19:36They'Re Basically Just Taking Data Frames and Adding New Feature Columns to Them as You Go Along and Then You Get to the Estimator and the Single Thing about an Estimator Is It Has a Fit Call so You'Re Basically Saying Okay Now I Want To Take the Data and Fit It into this Model Use the Data To Fit the Model To Build the Equation so that's What We'Re Doing this Is Actually Really Simple
27. 1:21:12And I'M Not Enough of a Data Scientist To Tell You What the Right Value Is Here the Best I Can Ever Get Talking to Data Sciences Scientists Is Play with this until It's Right but the Regularization Parameter Is the for the Linear Regression It Controls over Fitting or under Fitting Right and I'M Leaving It at the Default but I Have It in There because I Was Actually Playing with the Value Earlier and I Left It Back at the Default because It Didn't Have any Effect and Then if I'M Not Sure all of the Things Basically the Builder Pattern Is if You Know if You Want To Know all of the Things That You Can Set on a Particular Algorithm and They'Re All Different
28. 1:22:34This Is Just an Explanation of the Regularization Parameter Now We'Re Going To Put It Together in a Pipeline So this Is a Really Simple Pipeline We'Re Basically Chaining Together We'Re Creating this Pipeline Object and We'Re Chaining Together the Encoder and the Linear Regression and Things Are Passed through You Know in that Order so It's Almost Unix Pipe like and Then When We'Re Done with that We Tell the Pipeline To Do the Fit for Us this Then Creates the Model Right Based on the Data So Fit this Create the Equation for the Linear Regression Based on the Data That I Told You To Look at in the Training Data Set
29. 1:23:07So Fit this Create the Equation for the Linear Regression Based on the Data That I Told You To Look at in the Training Data Set and It's Actually Analyzing the Data Right Now this Actually Takes About Well 12 Seconds because It's Not Big Data Set and Now We Can Do the Transformation We Can Actually Say Predict What 2017 Is Going To Look like Based on 2015 and 2016 That Didn't Take Long So Let's See What It Looks like and Rather than Run a Statistical Check on this this One I Find It's Just Fine to Eyeball It
30. 1:27:00And It Does some Analysis on that and Then It Explains Why You Don't Want To Use a Tcp Socket in Production because It's Not Rewind Able Right So Therefore There's no Fault Tolerance so Then There's a Kafka Server That It Queries As Well Later On To Show You How To Do that and Obviously that Is Rewind Able to the Degree that There's Enough Disk Space To Store You Know All the Data and the Offsets and that Particular Kafka Server Is Fed by a Scala Server That I Wrote that It's Wikipedia Data Whenever You Edit a Wikipedia
31. 1:28:05They Don't Know Who You Are They Can't Plug Your User Name in So They Plug an Ip Address in either an Ipv4 and Ipv6 Ip Address so What I Had the Server Do Is Oh those Things It's a Gross Level Geocoding Really Probably Only Valid to the Country but Then You Can Begin To Plot in Real Time on a Real-Time Map Hey Who's Editing Wikipedia and You Run this Thing and You See the Map Updating in Real Time Oh the English Wikipedia Is Being Edited and Why Is Somebody in Crimea Editing the Winglet English Wikipedia so It's Kind of an Interesting Little Demo so that Notebook Is in There and There's a Lot of Words in It More than I Like but It Explains Tries To Explain You Know in Depth All the Things That You Can Do with the Streaming Library

" related talks

• 01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live codingOleg Klimenko · Java ON 2022
• 05mBig Data, No Big Deal: PySparkCatherine Devlin · PyOhio 2020
• 01h 35mDeep Dive into the Big Data LandscapeDean Wampler, Eva Andreasson · GOTO Copenhagen 2014
• 40mProductionizing Big Data - Stories from the TrenchesRoksolana Diachuk · Scala Days 2023 Madrid
• 11mScaling R with SparkJavier Luraschi · rstudio::conf 2019
• 39mAnalysing Public Kafka Data from NASA SatellitesFrank Munz · Berlin Buzzwords 2025
• 23mSpark Datasets: why they aren't great and what could be doneFlavio Brasil · Scale By the Bay 2017
• 52mSolving Analytical Problems using Apache SparkRohit Bhardwaj · Developer Summit 2020