Composable microservices for streaming analytics
summary
Shows a concrete pipeline: Kafka topics feeding containerized Python microservices on OpenShift with Spark for distributed compute; demos include a mean/variance accumulator that emits a derived stream for cumulative‑chart anomaly detection and an even/odd filter plus a sentiment transformer using spaCy and VADER. Images build from Git, pods auto‑scale Spark executors, Kafka stream schemas act as the API, old topics replay for testing, and the demo runs live with logs and autoscaling visible in the console.
AI-written summary · about
chapters (23)
- 0:00Intro
- 0:26Agenda
- 1:51Pipelines
- 2:39Microservices
- 3:17Microservices vs UNIX
- 5:02Why use microservices
- 5:28Code simplification
- 6:11Separation of concerns
- 7:21Simplified refactoring
- 8:46Problems with microservices
- 10:04Tools
- 10:35Scalability
- 11:16Spark
- 12:11Capital Architecture
- 13:34Stream Architecture
- 17:11Simple Example
- 18:34Advantages
- 20:58Kafka
- 21:55Technology
- 23:04Data Stream
- 23:36Filter Function
- 30:32Data Transformation
- 38:19Practical Concerns
related talks
47mBuilding High Performance Real-time Stream Processing Microservices
30mEvent Streaming and the Future of Applications
58mGame of Streams
01h 11mBootiful Stream Processing!
44mKafka based Microservices with Akka Streams and Kafka Streams
47mStream All the Things!
36mStream All The Things!
41mMicroservices and Event Sourcing with Kafka and Kubernetes