Architecting Applications With Multiple Open-Source Big Data Technologies
summary
A portfolio of demos uses Apache Cassandra, Kafka (with KRaft coming), Kafka Connect, Kafka Streams, Debezium, OpenSearch, Cadence workflows and Kubernetes to solve real problems: an anomaly detector that reaches 19 billion checks per day with Cassandra and k8s, a low‑latency fintech store using Cassandra for global multi‑DC writes, and a drone delivery app with Cadence plus streaming ML. Things that bite: Spark needs lots of RAM, Connectors fail without error handling, Redis licensing and caching tradeoffs, and streaming ML struggles with concept drift and ordering semantics.
AI-written summary · about
chapters (26)
- 0:00Introduction and background
- 3:28Apache Cassandra overview
- 5:16Apache Spark explained
- 7:02Notebook tools like Zeppelin
- 7:43Apache Lucene for search
- 8:47Apache Kafka fundamentals
- 10:54Kafka Streams API
- 12:00Kafka Connect connectors
- 13:28MirrorMaker 2 replication
- 14:54Apache Camel integration
- 16:38Kafka Parallel Consumer
- 17:42Zookeeper and Curator
- 19:07Kubernetes for orchestration
- 20:12Monitoring with Prometheus and Grafana
- 20:34OpenTelemetry for tracing
- 21:38PostgreSQL database features
- 22:22Superset for visualization
- 22:44OpenSearch dashboard
- 23:25Redis in-memory data store
- 24:09Cadence workflow engine
- 25:34Debezium for CDC
- 25:55Carapace schema registry
- 26:37FedDB database proxy
- 26:57RisingWave stream processing
- 28:02Keras for machine learning
- 28:46Architecture examples
related talks
50mApplication development and data in the emerging world ...
59mStreaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & Akka
47mPatterns for Persistence and Streaming in Cloud Architectures
47mBuilding Modern Data Streaming Apps with Open Source
06mJay Kreps interview
46mKafka Streams at the Big Data Crossroads at Lightning Speed
03h 00mStream Processing, Going Beyond Pub/Sub with Kafka Streams and...
26mLeveraging Data in Motion in a Cloud-first World