DataEngBytes Perth 2024
who it's for
data engineers, platform engineers, analytics engineers, data architects, privacy and compliance engineers, engineering managers, site reliability engineers
topics
ClickHouse with cloud storage; real-time OLAP architectures; materialized views and projections; Kafka and Flink streaming; separate storage and compute; on-read aggregation for analytics; high-concurrency query serving; Spark for multi-petabyte jobs; Ibis and Iceberg interoperability; consumer data right APIs; mutual TLS and certificate-bound tokens; data platform provisioning and CI
| title | author | length |
|---|---|---|
| Rethinking Status Quo: Open Source and the Cloud Data Warehouse | Tyler Hannan | 28m |
Argues that combining open-source ClickHouse with cloud storage and streaming unbundles the old cloud data warehouse model, enabling low-latency, user-facing… | ||
| The future of Data Engineering panel | Ash Smith | 50m |
Panel argues the future mixes mastering fundamentals with AI augmentation, not magic fixes; claims 80–90% burnout among data engineers (used as a caution)… | ||
| Moving Towards Real-Time OLAP - With First Principle Thinking and Guided by Access Patterns | Lee McDonald | 28m |
Built a live sports analytics pipeline for pro cycling that shifts heavy work to read time using ClickHouse, Apache Kafka/Flink for streaming, and… | ||
| Securely Democratising Data | Aiden Ziegelaar | 30m |
Consumer Data Right is a law-backed platform that standardizes data via OpenAPI data contracts, replaces PDF uploads and screen‑scraping, and already runs in… | ||
| Blueprints for Scaling: Practical Steps for Scaling Your Data Platform to Handle Exponential Growth | Jonathan Neo | 29m |
Canva’s data platform grows to ~25 PB with ~400–450 contributors and ~7,000 PRs/year; platform teams build an internal provisioning tool called Tundra for… | ||
| nothing matches | ||