Distributed Commit Log: Application Techniques for Transaction Processing

on YouTube ↗40m

" summary

Produces concrete practices for using Amazon Kinesis (Kafka-like) for transaction processing: uses per-key ordering and shard-aware state, shows idempotent processing and per-key sequence-number checkpoints to avoid duplicate effects, documents limits — five reads per second and 2 MB/s per shard (implying ~100 ms baseline latency), and describes built tools: an in-memory local stream for testing, shard split/merge handling, and a composite worker pattern to multiplex reads. Costs and trade-offs (shard fees, storage, retry/poison-event strategies) are quantified and evaluated.

AI-written summary · about

" chapters (5)

1. 0:00Start
2. 1:21Goals
3. 20:12Sub-sequence processing
4. 22:43Classify exceptions
5. 31:30Track shard state by partition key

" related talks

• 29mApache Kafka and the Next 700 Stream Processing SystemsJay Kreps · Strange Loop 2015
• 42mDistributed Real Time Stream Processing: Why and HowPetr Zapletal · Scala Days Berlin 2016
• 53mDemystifying KafkaNakul Mishra · JDD 2019
• 42mContemporary Approaches to Data at ScaleBen Stopford · Code Mesh 2015
• 41mStream Processor as a Database: Building Online ApplicationsStephan Ewen · Berlin Buzzwords 2016
• 24mDistributed Transactions at Scale in Amazon DynamoDBJoseph Idziorek, Alex Keyes, Colin Lazier, Somu Perianayagam, Prithvi Ramanathan, James Christopher Sorenson III, Doug Terry, Akshat Vig · USENIX ATC '23
• 01h 00mDistributed transactions implementation trade-offsArtem Aliev · Hydra 2022
• 20mWrites, 3 ways: Postgres, Apache Kafka® and Apache Iceberg™Celeste Horgan · Berlin Buzzwords 2026