Siphon : Modern Data Stack with SF-CH & Iceberg
summary
Siphon reads Postgres logical replication from a read replica, decodes CDC in Go, and pushes events via NATS JetStream to consumers that write to Snowflake, ClickHouse and Apache Iceberg; it achieves sub-minute latency with adaptive batching, ring-buffered WAL buffering and rule-based multi-destination routing. The system targets zero impact on primaries, handles billions of changes daily, offers at-least-once delivery with consumer-side deduplication, and is beta-tested for automatic resume and no-data-loss on restart.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:46GitLab mission and values
- 3:36Production challenges at GitLab
- 6:07Why Siphon was built
- 7:31Siphon architecture overview
- 8:55Key features of Siphon
- 11:03Data consistency and delivery
- 12:28Performance optimizations
- 13:51Multi-destination routing
- 14:55Monitoring and observability
- 15:59Production readiness and lessons
- 17:25Open source and future plans
- 18:28Q&A session
related talks
50mDesigning a scalable data platform based on Apache Spark/Iceberg
26mArchitecting real-time transactional data lake
20mWrites, 3 ways: Postgres, Apache Kafka® and Apache Iceberg™
05mMaking Iceberg walk and talk like Postgres
39mApache Iceberg Merge-On-Read: Streaming CDC
36mApache Kafka Meets Apache Iceberg: Real-Time Data Streaming
29mServing a multi-platform Data Lake via Delta, Iceberg, Unity and Glue
32m7 steps to build-your-own data pipeline - for day 1 of your startup