Enhancing Flexibility and Productivity with Access Patterns and Storage-Agnostic Abstractions
summary
A deployed platform that maps a single entity-and-attribute data model onto three storage classes—streaming (Kafka, Pub/Sub), batch (HDFS, S3, GCS) and random‑access (Cassandra, HBase, BigTable)—and automatically replicates from commit‑log primaries to replicas so applications run unchanged across regions and tech stacks. It exposes Apache Beam and Flink sources/sinks, supports local in‑process testing, and layers eventual consistency with ACID checks via a transaction coordinator; production IoT use at O2 Czech Republic runs at about 10,000 TPS and is licensed ASLv2.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:18Problem and motivation
- 3:09Data model and entities
- 5:00Attribute families and storage
- 8:20Data flow and commit logs
- 10:04Stream elements and configuration
- 11:48Abstract data operators
- 13:14Reading and writing data
- 15:00Apache Beam integration
- 16:08Ingestion service
- 17:13Cloud migration benefits
- 19:01Multi-region replication
- 21:10Testing and validation
- 22:14Asset transactions
- 26:49Transaction consistency
- 27:52Future plans
related talks
47mPatterns for Persistence and Streaming in Cloud Architectures
30mPlatform Oriented Data Architecture for Acceler...
50mApplication development and data in the emerging world ...
33mThe advent of the open data lake
45mModern Data Architecture
21mBuilding Distributed Platforms at Extreme Scale on Fast Changing Data
16mLife is But a (Data) Stream: Building Quality Data Pipelines
50mDesigning a scalable data platform based on Apache Spark/Iceberg