Cloud Foundry Stream Processing - Monitoring Comcast’s...
summary
Comcast builds a Java Spring app using Kafka Streams on Cloud Foundry to monitor 2 million miles of outside plant and 40 million customer devices; it runs ~60 instances against topics with ~60 partitions, uses local state stores backed by changelog topics (compaction to limit size), exposes interactive-query REST endpoints, and ships logs via Kafka Connect to Elasticsearch. Six physical Kafka brokers handle full-load runs (whopper every 4 hours); the system is deployed in production-parallel and reduces developer friction and latency.
AI-written summary · about
chapters (8)
- 0:00Introduction and overview
- 5:47Stream processing fundamentals
- 14:06Kafka client and producer API
- 21:38Processor API patterns
- 34:39Technical notes and tips
- 44:26Logging and monitoring at scale
- 50:45Stateful sets and container runtime
- 1:01:42Q&A session
related talks
50mWalking Through Spring Cloud Data Flow
31mBridge to the Cloud
34mBeaming Flink to the Cloud @ Netflix
50mApplication development and data in the emerging world ...
46mKafka Streams at the Big Data Crossroads at Lightning Speed
39mData Streaming? I don't even know her
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...
28mTransforming the Financial Services Industry with Event Stream Data