DataEngBytes Sydney 2024
as the organisers put it
Join the DataEngBytes data engineering community in Sydney. Connect with data engineers, attend meetups, and participate in conferences.
who it's for
data engineers, analytics engineers, platform engineers, machine learning engineers, data architects, engineering managers, site reliability engineers
topics
knowledge graphs for vector search; retrieval augmented generation (RAG); Protobuf data contracts; Data Vault 2.0 on Snowflake; Flink streaming pipelines; Neo4j vector index; dbt templated transformations; Snowpark Python classification; Fivetran and Confluent Kafka ingestion; Delta + Flink CDC pipelines; LangChain integrations
| title | author | length |
|---|---|---|
| Future of Machine Learning Panel | Suneeta Mall, Chip Huyen | 48m |
Panel argues that scaling hits concrete limits: internet text for training is nearly exhausted, datacenters already use ~1–4% of world electricity, and gains… | ||
| Data Mesh & Data Vault | Patrick Cuba | 31m |
Five-year case study of a cloud migration that centers Snowflake and combines Data Vault 2.0 with domain-driven teams, templates and automation. Ingest uses… | ||
| Knowledge Graphs : Adding a New Dimension to Vector Search and Generative AI | Darren Wood | 38m |
Claims that grounding LLMs with knowledge graphs reduces hallucination and adds explainability while improving relevance by re-ranking vector results. Uses… | ||
| Beyond LLMs and Maturing RAG use cases | David Levy | 31m |
Claims that vanilla LLMs solve reasoning fail on simple tests — a model spending 19 seconds still answers “strawberry” R‑count wrong (says three Rs), and a… | ||
| Breaking into Data Engineering | Eevamaija Virtanen | 24m |
Brings 10+ years consulting and 3+ years in data engineering, lands first data-engineer role in 2022 and rises to senior within two years; learns via Udacity… | ||
| How to NOT build a data integration | Emily Melhuish | 26m |
Integrations with GitHub, PagerDuty and Jira use event webhooks, GraphQL/REST and Forge but real choices mattered: GitHub worked best via webhooks + Lambda →… | ||
| Data Integration at SafetyCulture: Implementing Protobuf for Efficient Data Contracts | Thiago Baldim, Richard Gutierrez | 18m |
SafetyCulture replaces many bespoke CDC feeds with a single central CC Kafka topic and Protobuf contracts that enforce an envelope with metadata (who, when… | ||
| The journey of Data Mesh and Compliance in a Multi-Regional Data Lake | Mike Dias, Paul Ashley | 27m |
Built a customer-facing Atlassian Data Lake that actually serves product analytics, data exports and in-product dashboards, using logical CDC with Protobuf… | ||
| Where are we going with observability? | Adrian Cole | 28m |
Argues that generative AI already augments observability by turning plain-English prompts into actionable queries and runbooks, demonstrates RAG with private… | ||
| Tag! You're I.T. | Victor Korabelnikoff, Sundeep Kaur | 26m |
NIB scans and inventories over 300,000 column-level objects using Snowpark Python, then runs Snowflake SYSTEM$CLASSIFY (sampling up to 10,000 rows) to produce… | ||
| Software Engineering Skills for Data Engineers | Yousry Mohamed | 28m |
Covers concrete fixes: pick numeric types to avoid floating point surprises and cut storage (example dataset stored as 22 GB vs ~11 GB when switching doubles… | ||
| Agile Data Platforms : Innovate at the speed of thought | Andrew Montgomery, Scott Shellien-Walker | 27m |
Argues that true agility means delivering more sophisticated use cases and that AI needs a solved data strategy first, then shows a working Coalesce pipeline… | ||
| Increasing the LLM Accuracy for Question Answering on Structured Data: Knowledge Graphs to the Rescu | Juan Sequeda | 43m |
Benchmark uses an enterprise insurance SQL schema, a set of enterprise questions, an ontology and mappings that virtualize the SQL as an RDF knowledge graph… | ||
| Beyond the Build: Unleashing the Hidden Value in Your Data Platforms | Piotr Katolik | 30m |
REA Group builds a streaming data platform on Kafka that enforces self-service onboarding with one- or two-step flows, eliminates approval bottlenecks, and… | ||
| nothing matches | ||