Spark 2.0

on YouTube ↗59m

" summary

Overview of what changed and what actually works: Spark 2.0 tightens schemas with DataFrames and Datasets so the engine can optimize, uses a binary columnar format plus runtime code generation under Project Tungsten to cut CPU and memory overhead, and adds structured streaming that treats streams as infinite tables and can push filters into data sources. Claims include over 1,000 deployments and public clusters up to 8,000 nodes; benchmarks show DataFrame execution outperforms RDDs and unifies SQL, MLlib and streaming.

AI-written summary · about

" chapters (43)

1. 0:00Intro
2. 0:45What is Apache Spark?
3. 1:54A Large Community
4. 2:26Apache Spark Users
5. 3:23Original Spark Vision
6. 4:17Motivation: Unification
7. 6:17Motivation: Concise API
8. 7:47How Did the Vision Hold Up?
9. 8:18Libraries Built on Spark
10. 8:45Which Libraries Do People Use?
11. 9:21Top Applications
12. 10:09Main Challenge: Functional API
13. 11:17Which API Call Causes Most Tickets?
14. 12:18Example Problem
15. 14:33Challenge: Data Representation
16. 17:02Why Structure?
17. 17:50DataFrames and Datasets
18. 19:03Execution Steps
19. 20:20DataFrame API
20. 22:43Why DataFrames?
21. 24:30What Structured APIs Enable
22. 25:49Performance
23. 29:29Dataset API Details
24. 33:07Data Sources
25. 34:29Data Source API
26. 35:14Examples
27. 36:34Hardware Trends
28. 38:33Project Tungsten
29. 38:52Tungsten's Compact Encoding
30. 39:51Space Efficiency
31. 40:16Runtime Code Generation
32. 42:14Long-Term Vision
33. 43:36Versioning in Spark
34. 45:40Major Features in 2.0
35. 46:13Background
36. 47:28Structured Streaming High-level streaming API built on DataFrames/Datasets
37. 50:01Structured Streaming API
38. 51:11Example: Batch Aggregation
39. 51:37Example: Continuous Aggregation
40. 52:51Incrementalized By Spark
41. 53:50Release Timeline
42. 54:13Conclusion
43. 54:58Want to Learn Apache Spark?

" related talks

• 10mNew Directions for Spark in 2015Matei Zaharia · Strata + Hadoop World San Jose 2015
• 54mWhat Does the Future Spark Have in Store for Us?Aleksey Zinovev · Joker 2016
• 19mUpdates on Spark, MLflow, and the broader ML ecosystemJavier Luraschi · rstudio::conf 2019
• 25mCONTINOUS APPLICATIONS WITH SPARK 2.0Kamil Owczarek · Scalar Conference 2017
• 43mIntroduction to Spark DatasetsHolden Karau · Scala Days Copenhagen 2017
• 44mProcessing large data with Apache SparkVenkata Naga Ravi · The Fifth Elephant 2015
• 44mData Science at Scale with Apache SparkDean Wampler · GOTO Chicago 2015
• 11mHow Companies are Using Spark, and Where the Edge in Big Data Will BeMatei Zaharia · Strata Conference 2014