Breaking out of the habit of bulk loading everything
summary
A utilities data platform captures MQTT from on‑site RabbitMQ (shovel to Event Hubs), uses Event Hubs, Stream Analytics, Azure Functions to transform and route streams into parquet in a data lake and into parquet and Azure SQL for users, and feeds Grafana dashboards that update every five seconds; about 1.1 million measurements per day flow through, implementation uses a page of SQL plus ~20 lines of Azure Function code, works fast and cheaply though Azure SQL is the largest cost.
AI-written summary · about
chapters (14)
- 0:00Introduction and background
- 1:28Data warehouse overview
- 2:10Challenge of new real-time system
- 3:56Cloud solution setup
- 5:01Data ingestion without code
- 6:26Stream analytics introduction
- 7:54Data transformation and storage
- 9:17Grafana dashboard solution
- 10:23Cost optimization strategies
- 12:16Skills and learning path
- 14:02Cost considerations
- 15:46Merging streaming and warehouse data
- 16:53New tools and approaches
- 17:37Key takeaways and conclusion
related talks
41mPredictive Maintenance with Azure Synapse
47mBuilding highly scalable IoT streaming solutions using Azure Event Hubs and MongoDB
27mAzure Cosmos DB Changefeed tales from the trenches
30mProcess large-scale data and computational function in serverless manner
44mCombatting illegal fishing with Azure for less than £10/mth
36mHow to Build a Solution Processing Terabytes of Data (...)?
39mMigrating a data stack from AWS to Azure #bbuzz
28mBest of Stream + Batch: When Streaming Analytics Isn’t