Global School Connectivity DataOps Platform
summary
An open-source end-to-end DataOps stack now manages location and connectivity records for 2.1 million mapped schools with realtime telemetry from 93k schools and a plan to scale to 6 million. The platform includes an ingestion UI called Giga Sync, Spark-based quality checks and orchestration on Daxter, storage on ADLS with a Delta Lake layer, DataHub for cataloging, Delta Sharing/Trino APIs for consumption, and GearMeter and Giga Maps as products; ingestion hit production last week and three-country pilots run pay-for-performance pilots.
AI-written summary · about
chapters (7)
- 0:00Giga UNICEF mission
- 3:10Open source tools
- 9:33Scalable DataOps platform
- 14:51DataOps architecture
- 22:19Giga Sync ingestion
- 25:31Data Hub discovery
- 28:20Future decentralization
related talks
41mModern Data Stack: Optimising and Scaling Data in a Tech Company
42mDriving Data Democratization
23m‘Samen Slimmer met Data’ – A company wide Data Transformation
37mBuilding a Data Platform from scratch
28mUsing Dataops To Build And Manage Data Products On Snowflake
32mPromoting a data driven culture in a world of microservices
17mDemocratising data analytics
30mGraph Data Science