Conquering All Stores with Gimel – A Unified Data Processing Platform
summary
Gimel is a Spark-and-Scala based platform that exposes a single dataset.read/dataset.write and SQL surface over Hive, HBase, Cassandra, Druid, Teradata, Elasticsearch, Kafka and cloud stores, backed by a unified data catalog for metadata, classification and redaction. PayPal runs it across >20 on‑prem Hadoop clusters migrating to Google Cloud with ~200k YARN jobs/day, 160+ PB and peaks of ~6M Kafka messages/sec. Gimel adds a Spark Thrift Server for low‑latency SQL, centralizes auditing/instrumentation, and claims to cut infrastructure toil that previously consumed most developer time.
AI-written summary · about
chapters (9)
- 0:00Introduction and agenda
- 1:01PayPal data landscape
- 4:35Gimmel system overview
- 7:21The data lifecycle challenge
- 12:46Unifying access and security
- 17:40Gimmel API implementation
- 22:52Technical deep dive
- 29:21Thrift server integration
- 31:19Future roadmap and wrap-up
related talks
49mData Platform Architecture, Evolution, and Philosophy
45mFrom Big Data to Big Information
14mBig Data Processing with GPUs
15mThe Future of Data
12mthe Rapidly Changing Hadoop Ecosystem
38mUnleashing the serverless data pipelines
41mBig data as we know it @bol.com
36mIn-memory data pipeline and warehouse at scale