Scalable Data Science with FHIR
summary
Demonstrates encoding FHIR resources into a columnar Parquet layout and running interactive analysis at scale using Bunsen, Apache Spark and Jupyter Notebooks. Claims petabyte-scale datasets become queryable (billions of rows) with single-digit second Spark SQL responses, supports one-to-one FHIR field mapping, value set broadcast lookups, cohort-building and time-series feature engineering, immutable versioned datasets for reproducibility, and simple export to ML pipelines. The open source pipeline converts FHIR bundles to Spark dataframes and writes Parquet so the approach is repeatable and practical.
AI-written summary · about
chapters (13)
- 0:00Intro
- 0:40FHIR data with consent
- 4:02Data complexity
- 4:50Projections
- 6:31Data like style
- 7:08What does data like
- 8:25How do we make this happen
- 11:47Open Source
- 13:25Getting Started
- 13:58Interactive Notebook
- 15:42Value Sets
- 25:23Time Series
- 30:24Machine Learning
related talks
44mTransforming FHIR Data for horizontally scalable analytics
31mMapping FHIR data to parquet for analytics and machine learning
39mHow to integrate FHIR with your analytics pipeline
36mBig Data Modeling with Spark SQL: Make data valuable
48mAnalytics and AI on FHIR
42mScalable Data Science in Python and R on Apache Spark
27mStorage and analytics
42mBuilding Robust Streaming Data Pipelines with Apache Spark