Data with a mission: a COVID-19 API case study
summary
A volunteer team builds a COVID-19 data platform in seven weeks that normalizes public health, mobility and resource datasets into a canonical JSON model, ingests CSV/JSON/PDF sources, stores them in Snowflake and exposes them via REST and OData APIs implemented on MuleSoft with DataWeave transformations. They produce 15 RAML specs and ~40 helper functions, use queueing and a CDN, survive a minor DDoS, and scale from ~600k early requests to 2.7M API calls, with Tableau dashboards powering visualizations.
AI-written summary · about
chapters (18)
- 0:00Start
- 2:28The digital data explosion
- 3:12The API-enabled data opportunity
- 4:56APIs and data value
- 14:10Choosing the API protocol and architecture
- 15:26How to consider the data flow
- 16:51API-led approach: how data enters the pipeline
- 18:06Designing the ingestion pipeline
- 19:28Implementing the ingestion pipeline
- 19:50Developing the core API implementations
- 21:11Testing the core implementations
- 21:40Deploying the ingestion pipeline
- 21:58Comparing the outputs
- 22:15Sharing the APIs so users could use the API
- 22:35Expectations of API usability
- 22:56Results since the go-live
- 23:39COVID-19 Data Platform: Pipeline today
- 23:53API-led approach: where we are today
related talks
42mBuilding data driven solutions for COVID 19 Contact Tracing & Vaccine Management
32mData saves lives: The power of crowdsourcing in a pandemic
24mBuilding Open Data Platforms, an Insider's Perspective
01h 01mData Con LA 2020 - Friday Keynotes
24mAPIs and Data products: How do they impact your business model?
35mBuilding a data analytics platform on strong opinions held loosely
34mModern Data Platform, IoT & the Cloud -how...
41mRapid Prototyping Data Platforms: A D4G Case Study with the Clean Air Coalition and partners