Data in Motion: Streaming Static Data Efficiently in Akka Persistence
summary
Implements the Akka Persistence Query stream production spec inside akka-persistence-cassandra and replaces blocking replay iterators with non‑blocking events by persistence id streams (uses eventsByPersistenceId for replay, tuning dispatcher, resultSize and buffer for best throughput). Shows eventsByTag and allPersistenceIds implementations, exposes a correctness problem: Cassandra’s materialized‑view replication can reorder partitions so a default 10s eventual‑consistency delay is a brittle workaround. Proposes write‑ahead logs per node merged into a single causal stream to get read‑your‑own‑writes and exactly‑once semantics; new replay is resource‑efficient and generally faster in practice.
AI-written summary · about
chapters (24)
- 0:00Intro
- 0:26Data at scale
- 1:10Streams
- 4:56Log data structure
- 7:51Akka Persistence Query
- 12:03Streaming static data
- 12:40Pulling data from a log
- 13:12Actor publisher
- 15:18Events by persistence id
- 16:24All persistence ids
- 18:24Events by tag
- 22:50Akka Persistence Cassandra Replay
- 24:06Non blocking asynchronous replay
- 24:41Benchmarks
- 25:49Alternative architecture
- 29:52Event time processing
- 31:40Ordering
- 32:25Distributed causal stream merging
- 34:29Exactly once delivery
- 37:31Akka Analytics
- 39:53Distributed systems
- 42:45Challenges
- 43:36Conclusion
- 44:08Questions
related talks
29mEVENT SOURCING IN AKKA PERSISTENCE
46mTame Your Data with Akka Streams
53mDistributed Eventsourcing with Akka
17mAkka Persistence
48mStreaming all the things with Akka Streams
31mSTREAM DRIVEN DEVELOPMENT - DESIGN YOUR DATA PIPELINE WITH AKKA STREAMS
30mCONSCIOUS CONSISTENCY WITH AKKA CLUSTER, CRDTS AND DISTRIBUTED DATA
41mThe Promise of Event Driven Programming with Fast Data