Processing one million data points daily with Dapr
summary
Processes about one million data points daily by replacing ad‑hoc checkpointing with Dapr Workflows that run parent/child workflows and use GetState/SaveState event sourcing to resume at the failed VM, cutting the prior 20–30% duplicate calls from restarts. Runs on AKS with Dapr sidecars, stores blobs in Azure Storage, uses Cosmos DB as state (2–3% of workflows hit the Cosmos DB 5 MB limit), exports metrics to Prometheus/Grafana, and saves months of custom state work while requiring a managed‑identity timeout increase.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 0:43Business problem
- 4:15Legacy architecture limits
- 8:56Checkpointing challenges
- 10:47Dapr workflows overview
- 14:05Integrating Dapr workflow
- 17:40Workflow recovery benefits
- 20:51Dapr infrastructure changes
- 21:58Migration learnings
- 24:11Future Dapr plans
- 25:15Conclusion & Q&A
related talks
43mDatabase capabilities you need for your next awesome app Q&A
51mSuper large, ultra fast, mega scaling cloud computing
20mBuilding realtime efficient queries in an IOT platform
21mSoftware Best Practices for Data Pipelines: Form Energy's Multi-Day Storage R&D Pipeline 🌳
23mAdobe + Aerospike: Real-time Streaming Segmentation
08mReal-Time Data Systems: Speed, Scale, and Resilience
51mAgency Workflows
37mReactive Architecture on AWS: from 0 to 1.6M events/minute