Integrating Kafka with a Real-Time Database
summary
Live demo shows MemSQL pulling Kafka topics into a real-time database on AWS, ingesting around 300k–1M rows/sec on a tiny two-node cluster while storing Kafka offsets in tables co-located with data to provide exactly-once semantics. Pipelines call stored procedures or external Python/R transforms before transactional commit; hot data lives in memory-optimized row tables and historical data in a disk-backed columnstore for analytics, letting per-meter queries return in milliseconds. Compression typically 5x, sometimes up to 70x.
AI-written summary · about
chapters (7)
- 0:00Introduction and speaker background
- 4:55Why Kafka matters for real-time data
- 8:51MemSQL architecture overview
- 14:53Demo setup and smart meter data
- 21:13Pricing and deployment options
- 27:44Columnstore vs rowstore performance
- 35:58Q&A session
related talks
05mClose Encounters with the Third Kind of Database
41mApache Kafka and ksqlDB in Action: Let’s Build a Streaming Data Pipeline!
29mApache Kafka and ksqlDB in Action : Let’s Build a Streaming Data Pipeline
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
39mKafka as a Platform: The Ecosystem from the Ground Up
42mStreamlining Real-time Data Analytics: Integrating ClickHouse with Apache Kafka
47mReal-time Fraud Detection Challenges and Solutions
58mRealtime Analytics Geofencing Kafka