DataEngBytes Perth 2023
who it's for
data engineers, analytics engineers, data platform engineers, machine learning engineers, data architects, devops engineers, engineering managers
topics
YAML data contracts in Git; CI/CD GitHub Actions enforcement; Amundsen data catalog integration; PostGIS geospatial storage; pgRouting with Dijkstra routing; AWS Greengrass StreamManager; Apache Iceberg table storage; Delta Lake for production data; MLflow and prompt tracking; vector database for embeddings; real-time DB cache for recommendations; Redshift ML with SageMaker
| title | author | length |
|---|---|---|
| DataEngBytes 2023 - PER-T1-01 - Chad Sanderson | Chad Sanderson | 30m |
Claims that YAML contracts in Git plus CI/CD GitHub Actions enforcing tiers turn data quality from reactive toil into preventative control by validating… | ||
| DataEngBytes 2023 - PER-T1-02 - Josh Armitage | Josh Armitage | 36m |
Claim: a data operating model is the missing blueprint that actually links multi‑year strategy to day‑to‑day execution by creating a guiding structure, teams… | ||
| DataEngBytes 2023 - PER-T1-03 - Lizzie Macneill | Lizzie Macneill | 27m |
Live demo builds an electric-vehicle charging map using PostGIS as the single data store, ingests CSV charging points from Transport for NSW, converts… | ||
| DataEngBytes 2023 - PER-T1-04 - Nathan Glover | Nathan Glover | 28m |
Edge devices batch JSON into gzipped files on disk and use AWS Greengrass StreamManager export tasks to upload partitioned objects to S3; S3 object events feed… | ||
| DataEngBytes 2023 - PER-T1-05 - Ellie Gholami | Ellie Gholami | 28m |
States that 85% of ML apps never reach production and recommends a concrete CI/CD pattern: develop code on sample data, promote code through Dev→Staging→Prod… | ||
| DataEngBytes 2023 - PER-T1-06 - Kamal Hossain | Kamal Hossain | 20m |
Built a realtime recommendation pipeline that addresses listings changing by the minute and users whose preferences shift in seconds, because embedding models… | ||
| DataEngBytes 2023 - PER-T1-07 - Nadia Reyhani | Nadia Reyhani | 28m |
Demonstrates running ML entirely inside Amazon Redshift Serverless using Redshift ML to call SageMaker Autopilot, or with Auto=off to train an XGBoost model in… | ||
| DataEngBytes 2023 - PER-T1-08 - Sergio Zenatti Filho & Elizabeth Antoinne | Sergio Zenatti Filho, Elizabeth Antoinne | 27m |
Shows working Azure OpenAI integrations: ingest company docs (PDF/Markdown) with the add your own data feature so models answer questions about naming, design… | ||
| Panel Discussion | Joe Reis, Zainab Maleki, Jonathan Neo, Ash Smith, Peter Hanssens | 39m |
Panel explores practical pain and fixes: thousands of messy models (about 8,000 dbt models at one company, a cited case of 39,000 dbt models) driving cost and… | ||
| DataEngBytes 2023 PER-T1-10 - Matthew Carter | Matthew Carter | 20m |
CICD pipelines run automated checks that catch surprises before users see them: merge to GitHub triggers GitHub Actions (or CircleCI/Codefresh) which deploys… | ||
| DataEngBytes 2023 - PER-T1-11 - Mark Kitson | Mark Kitson | 32m |
Argues that a semantic layer is a linked set of business and technical definitions that prevents costly misinterpretation, supports governance, and connects… | ||
| nothing matches | ||