DataEngBytes Brisbane 2023
2023-08-29
Brisbane, Australia
who it's for
data engineers, analytics engineers, data platform engineers, data architects, machine learning engineers, business intelligence developers, analytics product managers
topics
API-style data contracts; CI/CD data schema checks; Amundsen catalog enrichment; Spark performance tuning; Delta Lake direct lake mode; Power BI integration with lake; DuckDB local with MotherDuck; Snowflake pushdown with Coalesce; PostGIS routing with osm2pgsql; data modeling for ML and LLMs; OCR and visualization accessibility; vector embeddings and retrieval
| title | author | length |
|---|---|---|
| DataEngBytes 2023 - BRI-T1-01 - Chad Sanderson | Chad Sanderson | 41m |
Argues that API-style data contracts — a YAML contract spec plus CI/CD PR checks tied to schema, semantics and SLAs — prevent churn and costly outages at… | ||
| DataEngBytes 2023 - BRI-T1-02 - Mikiko Bazeley | Mikiko Bazeley | 39m |
Argues that the full-stack data scientist remains essential because MLOps is fragmented and one-size-fits-all stacks fail in practice, with concrete patterns… | ||
| DataEngBytes 2023 - BRI-T1-03 - Yousry Mohamed | Yousry Mohamed | 29m |
Explains concrete Spark fixes: don't collect to the driver because pulling a billion-row DataFrame crashes memory; cache or persist intermediate results to… | ||
| DataEngBytes 2023 - BRI-T1-04 - Mimoune Djouallah | Mimoune Djouallah | 30m |
Early-access report on Microsoft Fabric claims a unified data stack that actually reads and writes Delta Lake files, links SQL warehouse, Spark and Power BI… | ||
| DataEngBytes 2023 - BRI-T1-05 - Larene Le Gassick | Larene Le Gassick | 26m |
Shows that default Google Sheets charts are often unusable without tweaks and that scanned PDFs need OCR that only reaches about 90% accuracy, so human review… | ||
| DataEngBytes 2023 - BRI-T1-06 - Yanir Seroussi | Yanir Seroussi | 22m |
Argues that data engineering often outweighs data science in market value — job ads show about 3× as many data engineering roles at $100–150k and 4× at $200k —… | ||
| DataEngBytes 2023 - BRI-T1-07 - Joe Reis | Joe Reis | 27m |
Argues that data modeling is both abandoned and indispensable for AI, citing a real-world symptom of 39,000 DBT models in one company and widespread ad hoc… | ||
| DataEngBytes 2023 - BRI-T1-08 - Ryan Boyd | Ryan Boyd | 30m |
Argues cloud warehouses act like modern mainframes and demonstrates a hybrid pattern using DuckDB locally (WASM, Python) with MotherDuck coordinating cloud… | ||
| Panel Discussion | Joe Reis, Mikiko Bazeley, Ryan Boyd, Chad Sanderson | 33m |
Argues data engineering expands beyond pipelines into data modeling and ongoing quality work, with concrete ML pitfalls like target leakage and distribution… | ||
| DataEngBytes 2023 - BRI-T1-10 Rishu Saxena & John Cosgrove | Rishu Saxena, John Cosgrove | 32m |
Shows how Coalesce runs only on Snowflake with a pushdown model (no data stored in the tool), auto-generates pattern-based SQL (example: SCD2 code that would… | ||
| DataEngBytes 2023 - BRI-T1-11 - Lizzie MacNeill | Lizzie Macneill | 23m |
Builds a self‑hosted EV charging map by importing OpenStreetMap with osm2pgsql into PostGIS, converting lat/long to geometry, and serving GeoJSON to a Leaflet… | ||
| nothing matches | ||