DataEngBytes 2020
who it's for
data engineers, platform engineers, analytics engineers, ml engineers, streaming engineers, cloud architects, data architects
topics
data mesh architecture; Azure Synapse Analytics pipelines; code generated data platforms; schema registry with Avro; governed data lake on Snowflake; change data capture with Debezium; Flink SQL stream processing; Kafka and ksqlDB applications; Kubernetes plus Airflow for ML; Snowpipe and external Parquet tables; Apache Druid serving layer; Prometheus and Grafana monitoring
| title | author | length |
|---|---|---|
| Introducing Data Mesh A Paradigm Shift In Analytical Data Management Architecture | Zhamak Dehghani | 36m |
Argues that centralized data warehouses and lakes hit scale limits and proposes domain-owned, product-minded datasets connected by a platform that provides… | ||
| Synapse Analytics | Charles Feddersen | 24m |
Demo of an end-to-end Azure Synapse Analytics pipeline that ties Synapse Link (Cosmos DB operational data indexed to column format) to Spark notebooks… | ||
| Building A Code Generated Data Platform | Dominic Colyer | 23m |
Shows a runnable pipeline that uses Fivetran to land source data into Snowflake then auto-generates DDL from information_schema to create staging views… | ||
| Ever Changing Data Model Schema Management For The Future | Vidya Venugopal | 23m |
Schema management centers on per-topic contracts: producers register schemas with a Confluent Schema Registry, send schema IDs with messages, and consumers use… | ||
| Building A Governed Data Lake | Louis Lee | 28m |
Claims a governed data lake built on Snowflake that keeps one copy of data, runs across clouds, and charges by the second while offering RBAC, dynamic masking… | ||
| Data Team as an Optimisation Problem | Mike Gouline | 23m |
Company frames building a data practice as an optimization problem with weighted metrics and immutable vs mutable requirements, then applies it to concrete… | ||
| Automating ML Pipelines With Kubernetes And Airflow | Yana Segal | 24m |
Pipeline moves from manual Docker producers/consumers to an automated system that reads configs from S3, uses Airflow to enqueue one queue per group, and… | ||
| Change Data Capture With Flink SQL And Debezium | Marta Paes | 30m |
Flink 1.11 consumes Debezium JSON changelogs from Kafka and Flink SQL deserializes them directly, proven with a Docker Compose demo wiring Postgres → Debezium… | ||
| Building a Telegram bot with Apache Kafka and ksqlDB | Robin Moffatt | 26m |
Builds a Telegram bot in Go that ingests a CSV REST feed every three minutes via curl and kcat into Apache Kafka, normalizes records to protobuf with Schema… | ||
| Some Like It Hot | Gian Merlino | 32m |
Explains why Apache Druid becomes a hot serving layer for interactive analytics by ingesting and indexing data ahead of queries, using columnar segments (a few… | ||
| Sweet Streams Are Made Of These | Caito Scherr | 28m |
Builds streaming apps with Apache Flink (and mentions Beam and Kafka) and uses concrete metrics to change development: tracks watermarks monitored and… | ||
| Solving Data Engineering Challenges | Joel Roland, Steve Lee | 14m |
Claims MongoDB is the persistence heart of event-driven systems and ships a MongoDB Kafka Connector that works as both sink and source using change streams to… | ||
| How Enables Real Time Data With Event Driven Architecture | Sam Harley | 24m |
Builds an accessible data story from a school-disability CSV using Google Sheets, exports SVG then edits in Adobe Illustrator and Sketch, checks color with… | ||
| Inclusive Storytelling With Data | Larene Le Gassick | 31m |
Shows concrete failures and fixes: a Google Translate demo flips “she is a doctor” to “he” because training corpora reflect historical gendered roles; COMPAS… | ||
| Artificial Intelligence More Like Artificial Stupidity | Aiko Klostermann | 30m |
Architecture that lands Salesforce contacts, RDS rows and synthetic web clicks into a single S3 bucket and catalogs them so queries return results in seconds… | ||
| Building Data Integration Services In Real Time | Kerry McRae | 21m |
Delta Lake on Databricks delivers ACID transactions and time travel to fix real failures: a demo pipeline merges Kafka streaming flight events, a slowly… | ||
| nothing matches | ||