High Performance Data Loss
summary
Concrete failures with Apache Spark parsing CSVs produce silently wrong results: invalid dates overflow to other calendar days, type mismatches nullify entire rows, and unquoted multiline fields split records so an aggregate becomes 127453 reported vs 9 true. Fixes land in newer releases but many clusters run old versions. Built small reproductions that demonstrate each bug, show which parser settings avoid the problems, and recommend row‑level validation with Great Expectations or json schema and explicit CSV options.
AI-written summary · about
chapters (12)
- 0:00Silent data loss intro
- 1:30Data loss without errors
- 3:36Invalid dates overflow
- 8:09Schema enforcement nullifies rows
- 9:54CSV parsing quirks
- 13:25Aggregation inflation
- 16:13CSV multi-line ignored
- 19:02Parsing pitfalls overview
- 20:29Remedies for data loss
- 23:59Data reconciliation tools
- 26:25Data loss stories
- 27:50ML for data quality
related talks
35mBad data - anecdotes and examples from the real world
42mDeciphering the way data is tested
26mLarge Scale Data Validation (with Spark and Dask)
46mSparking Success: Unveiling the Journey of Apache Spark Application Development
36mData pipelines engineering made simple with Scala
29mSchemas: The Key To Data Happiness
53mTips and tricks for robust big data applications
30mFailing Fast with Data: Bringing CI to the Land of Data