How to Build a Solution Processing Terabytes of Data (...)?
summary
A practical case of processing terabytes of time-series meter data from 500,000 devices (11 channels, 10–15 minute intervals) and scaling the same pipeline to support 4 million meters. The solution uses Microsoft Azure services—Azure Data Lake Storage, Azure Data Factory, Apache Spark and SQL Server—to ingest, process and query the data; the talk also covers operational problems encountered, their fixes, and lessons learned for cost-effective scaling.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 3:11Design assumptions and data
- 10:46Technical requirements and challenges
- 15:48Why cloud and Azure
- 19:40Cloud solution architecture
- 28:56Project problems and lessons
- 32:36Conclusions and recommendations
- 35:26Q&A
related talks
44mApache Spark on Azure - solution (...)
41mPredictive Maintenance with Azure Synapse
26mHow we built from scratch a product analyzing terabytes of data
44mSmart application on Azure at Vattenfall
26mBig Data on Azure with Structured, Unstructured and Streaming Data
55mDesigning performant and scalable data lakes
42mWeb Development To Big Data. A Journey
40mData Lake House in the Cloud: The Azure Solutions