DataEngBytes 2022
who it's for
data engineers, data platform engineers, machine learning engineers, analytics engineers, data architects, data engineering managers, privacy engineers, site reliability engineers
topics
Snowpark for Python on Snowflake; Delta Lake ACID parquet; Protege and Neo4j ontologies; Snowflake clean rooms and homomorphic encryption; FinOps and managed stacks; event log CDC and replay; feature store scaling and sampling; Airflow and dbt pipelines; metadata driven data discovery; privacy budgets and differential privacy
| title | author | length |
|---|---|---|
| Data Engineering in a Rapidly Changing World | Joe Reis | 40m |
Defines data engineering as managing the data engineering lifecycle: ingesting from systems, transforming, storing and serving for analytics or ML, not tool… | ||
| Data Privacy Engineering to power Responsible Enterprise AI | Lalit Pant, Ashika Hettitantirige | 39m |
Describes a prototype cloud data exchange and data clean rooms (Snowflake secure UDFs and distributed clean rooms) used for secure joins and multi‑party… | ||
| Building a Scalable, Reproducible Ontology System to Improve Deep Learning Data | Zeph Yap, Ann Clark | 31m |
Nearmap builds a reproducible ontology system that links pixel labels to formal class definitions, stores example images in a git examples folder instead of… | ||
| Engineering Trust in Data | Zachary Zeus | 33m |
Argues that trust in data is solved by an engineering-led, continuous improvement methodology rather than vague strategy, citing 15 years of delivery and three… | ||
| Data replication patterns 101 | Karen Davis | 34m |
RealCo needs millisecond freshness for a hybrid feature with ~two months of data available from an API and limited platform infrastructure, so bootstrap +… | ||
| What is the MVP for Data Engineering projects? | Duana Stanley | 15m |
A cautionary playbook: an attempted end-to-end MVP that wires a rules engine into the mobile front end fails because the first use case (an engineer-made… | ||
| Data Engineering simplified with Snowpark for Python on Snowflake | Rishu Saxena | 33m |
Snowpark for Python runs native DataFrame operations in Snowflake, converts the DAG to optimized SQL and executes it 100% pushdown and parallel execution… | ||
| Data Engineering for Social Good | Lizzie Reid | 19m |
Kasna volunteer network The Good Data Institute has grown to about 150 volunteers and delivered 50+ projects and here presents a concrete case: One Tilt… | ||
| Delta Lake: Turn your Data Lake into a Lakehouse | Vinoaj Vijeyakumaar | 20m |
Delta Lake is presented as an open‑source protocol that adds ACID guarantees and warehouse features onto parquet files stored in a data lake, with co‑located… | ||
| AI as an API | Hilton Rosenfeld | 21m |
Combines two UCI spam datasets into a cleaned corpus of roughly 200,000 samples, tokenizes and labels messages, trains an LSTM via Keras/TensorFlow with batch… | ||
| Data Incidents - When things go wrong | Thiago Baldim | 20m |
Timeline from cron chaos to a managed pipeline using Airflow and DBT, Redshift, FiveTran, Segment and Tableau and concrete incident rules: detect by alarms and… | ||
| Change the way you write. Change the way you think. | Xuanyi Chew | 35m |
Argues that switching notation changes how problems are seen and solved, showing concrete results: using Einstein summation reveals algebraic neural-network… | ||
| DataEngBytes Panel discussion - Sydney | Simon Aubury, Wendy Wong, Ingrid Anzola | 28m |
Panel boils down to practice not platitudes: real claims about running Kafka and building Australia's first Big Data streaming pipeline on AWS, the use of… | ||
| Bringing Software Engineering Rigour to Data, DataEngBytes 2022 | Zainab Maleki | 42m |
Argues that applying DORA metrics to pipelines reveals how slow data teams are (elite teams deploy > once/day) and cites startling gaps like 973x more frequent… | ||
| Data Poker not Data Chess - thinking in Bets for Data Teams | Ned Letcher, Nigel Dalton | 37m |
Argues that strategic data choices are bets, not chess moves, and proposes Spotify’s DIBBs process (Data → Insight → Belief → Bet) as a practical method. Gives… | ||
| Metadata driven framework for data discovery and data quality | Xinwei Jiang, Akira Takihara Wang | 24m |
A metadata-driven system harvests Redshift, Airflow and job stats with Python/Spark into a Postgres central store, surfaces results in Amundsen (Elasticsearch… | ||
| Designing Data Governance Framework | Arpita Adhicary | 35m |
Argues that organisations must treat data as a strategic asset and shows a concrete governance blueprint: set up a governance office with legislative /… | ||
| Data Security for Data Engineers | Jenna Olovcic | 21m |
Data engineers must treat security as shared responsibility and adopt concrete controls: integrate GitHub secret scanner into CI/CD to catch leaked… | ||
| Well-Architecture Framework for Machine Learning | Vikas Rajput | 20m |
Maps the Well-Architected Framework to ML by applying its five pillars—security, cost optimization, operational excellence, reliability, performance… | ||
| Automated Data Analytics | Timothy Downs, Jane Ooi | 16m |
Open source AWS solution that deploys as a CloudFormation template into a customer account and wires S3, Glue, Lambdas and managed services into a ready data… | ||
| Customer Journey Analytics using Neo4j Graph Platform | Andrew Conacher | 19m |
Shows that Neo4j stores native property graphs and can turn event logs into actionable links: three supermarket purchases by one customer lead to an inferred… | ||
| Set your data streams in motion in the ocean! | Warren Vella | 20m |
A 3D‑printed ship with a Raspberry Pi runs Apache Kafka, ksqlDB and Schema Registry on-device to record a temperature probe every second and materialize views… | ||
| Best of Stream + Batch: When Streaming Analytics Isn’t | Rohan Garg | 28m |
Demo claim: Apache Druid stores event streams and batch loads to deliver sub-second queries and serve dashboards without separate stream processors plus KV… | ||
| Panel discussion - Melbourne | Nitish Mathew, Joe Reis, Caroline Even, Zainab Maleki | 33m |
Panelists report real outcomes: a production lake house in production ~1 year offloads Redshift and lets ML jobs read from S3 without impacting BI, an… | ||
| nothing matches | ||