DBTB INT Andrew Or r
summary
Overview of how Apache Spark hides its internal complexity: which runtime components (scheduler, shuffle, memory manager) enforce the abstractions, which APIs surface them, and what tradeoffs make common workflows fast or fragile. Demonstrates concrete use cases — machine learning, SQL, real-time analysis — and an IoT example that aims to prevent downtime, with notes on what was built on Databricks, which parts required engineering hacks, and which worked reliably versus which needed more rework.
AI-written summary · about
related talks
44mProcessing large data with Apache Spark
59mSpark 2.0
43mUse dbt and DuckDB Instead of Databricks for Data Processing
53mTips and tricks for robust big data applications
43mIntroduction to Spark Datasets
43mBeyond Shuffling: Scaling Apache Spark
38mPySpark in Practice
01h 18mScalable Data Science with Spark and R