Untangling Healthcare with Spark and Dataflow
summary
Cerner processes 55 million patients across roughly 3 petabytes of messy, heterogeneous records, links fragments by computing pairwise similarity (Cartesian joins then filtering), applies rule-based semantic mappings and a sepsis prediction model that an early deployment estimates saved ~500 lives. Engineering moves from Spark and Spark Streaming to a Beam-style API (not yet production) to unify batch/stream via Apache Beam / Google Cloud Dataflow concepts: windows, triggers, and retractions so latency, correctness, and cost become configuration rather than bespoke code.
AI-written summary · about
chapters (7)
- 0:00Introduction to healthcare data
- 2:07Act one: Reassembling Humpty Dumpty
- 16:38Act two: Putting healthcare fast
- 26:04Streaming challenges and solutions
- 33:05Act three: Reframing the problem
- 46:42Summary and future directions
- 50:33Q&A session
related talks
44mTransforming FHIR Data for horizontally scalable analytics
33mHealthcare "big" Data from Data Management to Web Apps
35mScalable Data Science with FHIR
36mBig Data Modeling with Spark SQL: Make data valuable
19mApache Beam pipelines at 100TB+ scale using Apache Spark
39mUsing Streaming Data Pipelines in the Cloud to Achieve Success
45mModern Data Architecture
05mHadoop and Healthcare