Tips and tricks for robust big data applications
summary
Demos use Spark (shell and containers), ORC/Parquet and Hive metadata to show concrete wins: a 10 million rows dataframe starts at ~200 KB, jumps to 71 MB with a free-form string, drops to ~5.2 MB with low cardinality, explodes to 974 MB with arrays and then shrinks to ~30 MB by storing arrays as dictionary-backed strings — about 90% storage savings. Shows partition tuning, coalesce vs repartition (37s vs 20s), AWS DQ for assertions, and lineage via Apache Atlas + Spark Atlas connector, plus Lyra/Jupyter and Kubeflow for reproducible notebook pipelines.
AI-written summary · about
chapters (9)
- 0:00Intro
- 1:19Brief History
- 2:57Agenda
- 12:44Know your data and storage format
- 13:25Metadata matters with slow file system
- 20:35Watch out for skewness
- 26:58Guard against bad data
- 33:50Governance
- 43:04Invest in tooling
related talks
44mProcessing large data with Apache Spark
03h 07mData of Unusual Size: Interactive Visualization
02h 05mBig Data and the Data Lake from the Analyst Perspective
35mData Processing Pipelines for Small Big Data
31mStrategies for Working with Data as It Grows
34mTesting Big Data Applications (Spark, Dask, and Ray)
40mPerformance Testing and Big Data
58mUnderstanding Big Data for Software Engineers