Building a data infrastructure for large municipal government
summary
County builds InfoHub that already combines about 1.7 billion rows across ~40 tables from 10 departments, producing ~2.25 million households served monthly and ~20 million MDM rows; ingestion uses IBM DataStage, Azure Data Factory and SQL, identity linking via IBM MDM and SAS, and storage sits in Azure Gov with three logical layers RDS/EDS/ADS (raw with PII, enterprise, anonymized analytics). Current limits are manual ETLs, laptop-bound analytics, poor code/data reuse and weak partner sandboxes; InfoHub 2.0 targets cloud-native ETL, self-service visuals, shared libraries and secured external workspaces.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 2:18About me
- 3:18Topics
- 3:53Background
- 4:45Vision
- 7:07Challenges
- 11:06Current solution
- 17:06Data stores
- 21:41Questions
- 22:03Azure ETLs
- 25:53Graphs
- 29:58Restrictions
- 31:58Limitations
- 35:26Requirements
- 37:25Wrapup
related talks
01h 01mData Con LA 2020 - Friday Keynotes
06mEnterprise Big Data Hub & the Federal Government
26mGrappling with massive data in the Public sector
45mHow Data Hub flies above the clouds!
44mData Mesh: Lessons from the trenches
12mHow external data and intelligence gathering is transforming the public sector
12mModern Data Warehouse for Small and Medium Business
28mHerding Data to the Cloud