Lessons Learned Building Our Own Dashboard Solution
summary
An interactive public dashboard now monitors data quality for 2.2 million LEIs and detects daily outliers that monthly reports missed, built over three years using FastAPI for the API, SQLAlchemy with materialized views and triggers to pre-aggregate heavy queries, and background processing with Celery + Redis plus Flower for visibility. Monitoring uses Sentry Prometheus Grafana, jobs run nightly, migrations and release cycles keep the service stable, and the dashboard is publicly available at dq-dashboard.live.org.
AI-written summary · about
chapters (15)
- 0:00Project introduction
- 1:10Legal Entity Identifier Foundation
- 3:21Distributed legal entity data challenges
- 5:27Interactive data quality dashboard need
- 6:32FastAPI for prototyping
- 8:17FastAPI features and benefits
- 10:03SQLAlchemy and relational databases
- 12:36ORM libraries comparison
- 13:41Database migrations lessons
- 15:48Celery for data processing
- 17:55Celery for task management
- 19:20Celery best practices
- 21:09Monitoring tools importance
- 21:53Project learnings and stakeholders
- 24:41Q&A
related talks
22mGovernment dashboards and open data.
29mImproving Developer Experience with LiveDashboard
45mA Practical Guide to Dashboarding
16mOops! I accidentally made a production dashboard
21mJourney from Broken Dashboards to O11y Maturity
38mData Secrets from a Platform Engineer
32mUsing Dashboards to Help Manage All Community Channels
19mOptimizing Data Retrieval with Python Celery