Data Engineering Architecture
summary
Simple runs a Redshift-centered stack with S3, Kafka (moving off RabbitMQ), PostgreSQL sources and bespoke services called Horizon and Saltooth to batch and load events; PG logical replication to Kafka streams every DB change. Practical surprises: a node fills because of many null user_id values forcing a re-distribution (disk spike from null user_id), hash-join spills bring clusters to 100% disk, and creating Redshift temp tables to force merge joins fixes performance. Operational lessons: set query timeouts, watch varchar(max) joins, use retention-capable queues, and instrument everything.
AI-written summary · about
chapters (20)
- 0:00Data engineering at Simple
- 1:47Why Simple built warehouse
- 3:52Why Redshift for warehouse
- 5:16Redshift architecture and SQL
- 7:00Redshift distribution and sort keys
- 9:26Redshift performance tuning trade-offs
- 11:55Columnar storage and query optimization
- 13:58Redshift compression and storage
- 14:41Redshift operational challenges
- 16:06Redshift stability and black box
- 17:10Python libraries for Redshift
- 18:38Simple’s data pipeline overview
- 19:41PostgreSQL to Kafka for real-time
- 21:45ETL with Horizon into Redshift
- 23:50Other data services at Simple
- 24:53Moving from RabbitMQ to Kafka
- 26:18Metrics health checks and observability
- 27:19Data modeling and schema management
- 28:22Python in data engineering stack
- 30:26Q&A
related talks
38mData Secrets from a Platform Engineer
25mPlatform Engineering for Modern Data Infrastructure
33mAccidental DataOps: How we matured our organization with data, and how you can too!
29mData Acquisition: Overcoming Challenges
25mBuild open format data lakehouse
29mYou Have A Data Lake, Now What?
01h 35mThe Modern Data Stack
50mData-Driven Engineering