Transforming FHIR Data for horizontally scalable analytics
summary
FHIR Data Pipes extracts FHIR resources, converts them to Avro then to parquet columnar format, and runs horizontally scalable analytics via Apache Beam pipelines and Spark or other engines. It supports FHIR search, direct JDBC reads (OpenMRS/HAPI), and Debezium-based continuous updates; BigQuery and Postgres are planned. Transformation on one machine is CPU‑bound at the database with about 2,200 resources/sec (file reads reach 60,000+), and a Spark query over 800k patients ~160M observations in ~70s on a 20‑core constrained setup.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 2:52Open Health Stack overview
- 6:41Columnar formats for analytics
- 8:47Big picture of FHIR data transformation
- 12:19Horizontal scalability explained
- 14:25SQL on FHIR schema
- 16:33Transformation pipeline details
- 21:05Synthetic data and performance setup
- 22:09Transformation performance on single machine
- 24:22Querying with Spark
- 26:10Example query TX_NEW
- 27:35Flattening nested FHIR data
- 30:43Why not JSON for analytics
- 31:03Flattened FHIR views
- 33:09Q&A session
related talks
35mScalable Data Science with FHIR
31mMapping FHIR data to parquet for analytics and machine learning
54mUntangling Healthcare with Spark and Dataflow
45mFHIR server scale and performance challenges
19mApache Beam pipelines at 100TB+ scale using Apache Spark
48mUsing Open Source Tech to Swap Out Components of Your Data Pipeline
41mAccelerating privacy-enhancing data processing
27mStorage and analytics