DataEngBytes Sydney 2023
who it's for
data engineers, MLOps engineers, data platform engineers, analytics engineers, data architects, machine learning engineers, database administrators
topics
retrieval augmented generation and vector databases; feature stores and lineage; Kafka streams for realtime pipelines; Snowflake Data Vault 2.0 patterns; DuckDB for serverless ETL; Delta Lake Iceberg compatibility; OneLake and Microsoft Fabric lakehouse; Neo4j Fabric for graph sharding; MLflow serving with Hugging Face models; LLM production patterns and governance; PII detection and masking in SQL; Postgres cost and DuckDB Lambda usage
| title | author | length |
|---|---|---|
| DataEngBytes 2023 - SYD-T1-02 - Mikiko Bazeley | Mikiko Bazeley | 42m |
Argues companies should stop chasing Google scale and extend existing ML platforms to host LLM use cases using RAG and vector DBs rather than ripping… | ||
| Panel Discussion | Mikiko Bazeley, Kanishka Mohaia, Chad Sanderson | 30m |
Panelists argue MLOps differs from DevOps because models are probabilistic, tightly coupled to data quality and versioning, and can produce different outputs… | ||
| Panel Discussion | Joe Reis, Lizzie Macneill, Ryan Boyd | 41m |
Panelists argue concrete tradeoffs: EDB claims it contributes over 30% of Postgres code annually, Postgres remains a solid default but not ideal for >10TB… | ||
| DataEngBytes 2023 - SYD-T1-05 - Patrick Cuba | Patrick Cuba | 17m |
Claims that Snowflake native features let you implement Data Vault 2.0 patterns end to end: model with three table types (hub, link, satellite), load using… | ||
| DataEngBytes 2023 - SYD-T2-01 - Rene Essomba | Rene Essomba | 29m |
Vector databases store high dimensional embeddings (images, text, audio) to give LLMs memory and reduce hallucination by retrieving source fragments rather… | ||
| DataEngBytes 2023 - SYD-T1-08 - Rishu Saxena & John Cosgrove | Rishu Saxena, John Cosgrove | 29m |
Coalesce plugs into Snowflake and runs metadata-only transformations inside the customer account, generating SQL/Snowpark (Roughly 200 lines per SCD2 pattern)… | ||
| DataEngBytes 2023 - SYD-T1-01 - Ryan Boyd | Ryan Boyd | 33m |
Argues that true “big data” is rare: 95% of BigQuery customers have <1TB and 90% of queries touch <100MB, so distributed clusters are often overkill… | ||
| DataEngBytes 2023 - SYD-T1-06 - Sana Sanai | Sana Sanai | 21m |
Claims a unified lakehouse called OneLake with serverless unified compute and Microsoft Fabric features: Direct Lake reads PB-scale Delta data for Power BI… | ||
| DataEngBytes 2023 - SYD-T3-01 - Scott Eade | Scott Eade | 26m |
Delta Lake 3.0 writes Iceberg-format metadata alongside Delta metadata so tools that speak Iceberg (BigQuery external tables, Snowflake private preview) can… | ||
| DataEngBytes 2023 - SYD-T2-04 - Suneeta Mall | Suneeta Mall | 21m |
Argues for treating data as the development driver, not an afterthought, and shows concrete techniques: automated label-error detection using cleanlab and… | ||
| DataEngBytes 2023 - SYD-T3-03 - Suzanne Nieuwenhuizen | Suzanne Nieuwenhuizen | 27m |
Shows a production pattern that separates stable entities and high-velocity events with Neo4j Fabric, using shards for person/route… | ||
| DataEngBytes 2023 - SYD-T2-02 - Vinny Vijekumaar & Zak Sheikh | Vinny Vijekumaar, Zak Sheikh | 21m |
Claims LLM work is a data engineering problem not just data science and shows it end to end on Databricks Data+AI: use AI_GENERATE_TEXT (OpenAI GPT-3.5) to… | ||
| DataEngBytes 2023 - SYD-T1-04 - Yash Sharma | Yash Sharma | 19m |
Atlassian describes building a multi-region transactional storage platform that centralizes compliance and storage with a control plane, data plane, migration… | ||
| DataEngBytes 2023 - SYD-T2-03 - Akanksha Malik | Akanksha Malik | 26m |
Practical checklist for turning prototypes into production: define team roles (data engineer, data scientist, ML engineer), choose Dev/Test/Prod or… | ||
| DataEngBytes 2023 - SYD-T3-04 - Abhinav Goyal | Abhinav Goyal | 21m |
Argues that a compiler-style semantic layer turns brittle pipelines into verifiable programs: ban base types and replace ints/strings with semantic types with… | ||
| WHAT DOES MLOps LOOK LIKE FOR LARGE LANGUAGE MODELS (LLMs) IN PRODUCTION? | Arezou Soltani | 27m |
Demonstrates an Atlassian Virtual Assistant for JSM implemented as a two-tier system combining an in-house intent classifier plus a ChatGPT integration that… | ||
| PROCESSING 40 TB OF CODE FROM ~10 MILLION PROJECTS WITH A DEDICATED SERVER AND GO FOR $100 | Benjamin Boyter | 24m |
Cloned and analyzed roughly 10 million projects (about 15k short) totaling 40 TB of code, using shallow git clones, SCC for language/count/complexity, and a… | ||
| DataEngBytes 2023 - SYD-T1-03 - Chad Sanderson | Chad Sanderson | 28m |
Defines data contracts as API-like agreements that declare schema semantics and SLAs and enforces them programmatically via YAML contract per resource with a… | ||
| DataEngBytes 2023 - SYD-T3-06 - James Weakley | James Weakley | 19m |
Snowflake native applications let vendors deploy code into customer accounts and expose functionality via Snowflake primitives like tables, UDFs and stored… | ||
| DATA MODELLING IS DEAD! LONG LIVE DATA MODELLING! | Joe Reis | 30m |
Claims that data modelling is both neglected and essential for AI success, arguing that ignoring conceptual and logical modeling leads to brittle systems… | ||
| DataEngBytes 2023 - SYD-T1-10 - Kanishka Mohaia | Kanishka Mohaia | 27m |
Rokt runs a web‑scale real‑time ML stack that hits 450 ms P99 latency while serving about 17M requests/day across roughly 60 concurrent models and terabytes of… | ||
| SUPERCHARGING SQL QUERY PERFORMANCE: UNLEASH THE POWER WITHIN! | Lisa Li | 23m |
Merchant, MerchantProduct and AdvertiserProduct tables on SQL Server grow from ~2M to 24M rows and cause timeouts; adding a composite index (merchant_id… | ||
| BUILD YOUR OWN ELECTRIC VEHICLE CHARGING MAP WITH PostGIS | Lizzie Macneill | 26m |
A live map server is built with PostGIS and pgRouting running on PostgreSQL, importing OpenStreetMap via osm2pgsql and renderd and charging points from the… | ||
| nothing matches | ||