Big Data Modeling with Spark SQL: Make data valuable
summary
Shows how to build a single wide provider analytics fact for 80,000 providers (100M patients) using Spark SQL and Kudu to avoid joining large fact tables, cut query I/O, and support near‑real‑time updates every few minutes. Details schema choices: one grain, conformed dimensions, denormalize into big blocks, divide‑and‑conquer batch or micro‑batches, and track model lineage. Demonstration processes a public 2.1M‑row dataset on Databricks with Spark 1.6 and reproduces aggregations and pivots to validate results.
AI-written summary · about
chapters (17)
- 0:00Intro
- 1:05About Jayesh
- 1:48Agenda
- 2:34Data Modelling
- 4:48Challenges
- 6:30Characteristics of Big Data
- 8:23Traditional Data Modelling vs Big Data
- 10:36Big Data Modelling Challenges
- 11:49What to Expect
- 14:34Features of Spark
- 17:29Spark SQL Sequel
- 19:28Big Data Modeling Principles
- 21:52Big Data Modeling Techniques
- 25:06Case Study
- 27:36Source Integration
- 29:15Results
- 30:42Demo
related talks
35mScalable Data Science with FHIR
40mProductionizing Big Data - Stories from the Trenches
01h 48mBig Data made easy with a Spark (3.0)
01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live coding
08mRevolutionizing Healthcare with Big Data and ML
44mData Science at Scale with Apache Spark
28mAI-driven query plan optimization in big data processing tools
52mSolving Analytical Problems using Apache Spark