DataEngBytes Auckland 2024
who it's for
data engineers, data platform engineers, analytics engineers, machine learning engineers, data architects, engineering managers, privacy engineers
topics
vision-based large language models; retrieval-augmented generation systems; observability with dbt and Elementary; unified dataframe API (Ibis); Kafka streaming and Flink processing; cloud data warehouse vs ClickHouse; data privacy vaults and tokenization; OCR plus CLIP-style embeddings; prompt engineering for extraction; Iceberg and Delta Lake hybrid queries; reverse ETL and operational analytics; Airflow/Astronomer + Starburst/Trino
| title | author | length |
|---|---|---|
| Data Ethics Panel | Jessica Chunyao Zhang | 32m |
Panelists give concrete examples and operational guidance: a supermarket’s facial recognition wrongly accuses a customer and research like Gender Shades shows… | ||
| From Start-Up to Scale-Up: Building a Data Team at a High-Growth Global Ed-Tech Company | Chuan-Zheng Lee | 23m |
Company scales from ~8M to ~45M users and data grows from 4 TB to 185 TB warehouse while a first data group becomes a seven‑person team. Engineering uses… | ||
| Supercharge Information Extraction with Vision-Based LLMs | Elaine Gao, Sahil Bahl | 28m |
Claims automation for an insurer that replaces heavy OCR + regex and custom training with Vision LLMs plus an OCR pre-step and prompt engineering, implemented… | ||
| Measuring Success - Observability and More! | Anna Choi | 26m |
Tracksuit sets up concrete observability: DBT + Elementary on Snowflake to quantify data health with defined formulas for accuracy (accurate records/total)… | ||
| Data Platforms Transformation - Why and How | Gaurang Deshpande | 28m |
Platform transformation is required when patchwork stops working and scale, latency or change management break; expect a minimum three‑year program with a hero… | ||
| Unified dataframe API for different data backend | Chip Huyen | 26m |
A unified dataframe API called Ibis compiles Python dataframe code into relational expressions that target Snowflake, DuckDB, Spark, RAPIDS/GPUs and others… | ||
| Stop Making Your Data Team the 'Data Police' | Sean Falconer | 24m |
Argues that moving PII into a centralized data privacy Vault and replacing values with tokens or proxies (using tokenization and polymorphic encryption) cuts… | ||
| Navigating Streaming Infrastructure | Adi Polak | 27m |
Compares concrete tradeoffs: Kafka as immutable, partitioned log with replicate three times for durability; producers serialize, consumers control… | ||
| How open source is re-shaping the cloud data warehouse landscape | Tanya Bragin | 32m |
Argues the cloud warehouse era is fragmenting: ClickHouse (open source) hits millisecond interactive queries vs seconds on cloud warehouses, and a reproduced… | ||
| Mixed Model Arts | Joe Reis | 45m |
Argues that application, analytics and ML modeling must converge because siloed models and messy upstream data break AI projects — industry failure rates hover… | ||
| Advanced Enterprise RAG Systems | Amr Awadallah | 31m |
Argues that enterprise RAG is solved by three concrete moves: retrieval-augmented generation (embed company docs into a vector store, run a ranker, feed top… | ||
| How I built an entire data platform by myself that thousands of data engineers use in one year | Zach Wilson | 29m |
Claim: one person builds a full data platform in a year that supports thousands of users, with stepwise architectures and clear tradeoffs. Concrete stack… | ||
| nothing matches | ||