DataEngBytes Melbourne 2024
as the organisers put it
Join the DataEngBytes data engineering community in Melbourne. Connect with data engineers, attend meetups, and participate in conferences.
| title | author | length |
|---|---|---|
| Serving a multi-platform Data Lake via Delta, Iceberg, Unity and Glue | Ankita Saha, Amey Khedekar | 29m |
A rearchitected Orion platform puts S3 as single copy storage serving Delta Lake files with Glue as engineering catalog, Databricks for compute, Airflow… | ||
| GraphRAG: Powering Up LLMs with Knowledge Graphs | Siddhant Agarwal | 32m |
GraphRAG shows how Neo4j combines knowledge graphs with RAG: it ingests SEC EDGAR Form 10‑K (300–400 page PDFs) and Form 13 filings, extracts entities and… | ||
| Dear Podcast, Can You Help Me? Leveraging OpenSearch and LangChain to ask a podcast for advice | Jenn Junod | 20m |
Builds a Retrieval-Augmented Generation pipeline that turns podcast episodes into searchable knowledge: use FFmpeg to extract audio, Whisper → transcribe to… | ||
| Building the Ultimate Databricks Environment | Stu Johnstone | 26m |
Practical checklist for a Databricks deployment: Terraform repo with examples, backend private link and frontend private link setups to eliminate public IPs… | ||
| Garbage In, Garbage Out: Elevating AI with High-Quality Data | Prerna Tiwari | 21m |
Three real failures ground the claim: Google Flu Trends doubles case estimates for many weeks and collapses by 2015, Air Canada’s chatbot gives wrongful… | ||
| Rethinking Data Catalogs: The Promise and Pitfalls | Harmeet Kaur Sokhi, Vishal Srivastava | 27m |
Modern catalogs often sit unused despite integrations with Snowflake, Databricks, Tableau, dbt and Monte Carlo; survey numbers cited are 80% think a catalog is… | ||
| A quack of all trades: DuckDB, a versatile analytical database to keep in your toolkit | Ned Letcher, Simon Aubury | 25m |
DuckDB is a fast in‑process analytical SQL engine that reads nested CSV/Parquet from S3 or local disk, does projection pushdown and zone‑map skipping, and… | ||
| How Atlassian is migrating 40PB data with 99.99% reliability | Will Falconer, Yash Sharma | 23m |
Atlassian migrates 40+ PB of transactional media into a new transactional data platform using a three-phase migration: dual-write validation, cutover of new… | ||
| Data, It’s a People Problem: Understanding the Sociotechnical Elements of a Data-Driven Organization | Gunu Diamond | 26m |
VistaPrint runs ~400 DAGs and supports ~800 engineers, moves from a central team to a hub‑and‑spoke model and embeds responsibilities in product teams… | ||
| Who Really Uses Dataform? Spoiler: We Do, and Here’s Why! | May Hu | 22m |
Pet Circle migrates SQL transforms onto Dataform on BigQuery, tied to GitHub and GitHub Runners, with Cloud Scheduler, Firestore, Cloud Functions and Google… | ||
| nothing matches | ||