Spark: One Tool for Everything?
summary
The talk examines Apache Spark's role as a general-purpose data processing engine after more than a decade of changes in hardware, storage formats, and workload patterns. It reviews how improvements in hardware and new storage standards affect when and how Spark is appropriate for different data-processing tasks.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:51Spark landscape changes
- 6:06Spark performance evolution
- 9:56Accelerating Spark approaches
- 13:13How Spark accelerators work
- 18:31Accelerator limits and capabilities
- 23:32Spark for smaller workloads
- 26:26Spark 4.0 new features
- 27:33Spark UI optimization tools
- 28:58Spark future and conclusions
- 30:45Q&A
related talks
45mFrom Big Data to Fast Data: Apache Spark
03mFred Reiss Interview
55mSpark Architecture
53mLord of the Spark
52mSolving Analytical Problems using Apache Spark
50mIntroduction to Real-Time Big Data with Apache Spark
01h 07mMyths about Spark, or Can an Ordinary Java Developer Use Spark?
11mThe Evolution of Apache Spark™