DataEngBytes 2021
who it's for
data engineers, analytics engineers, platform engineers, data platform architects, engineering managers, business intelligence engineers
topics
Apache NiFi flows; container orchestration with Docker and Celery; streaming dbt with Materialize; incremental materialized views; Snowflake micro-partitions on S3; Snowpark and Java UDFs; Reverse ETL with Hightouch; BigQuery and dbt migrations; Snowflake zero copy cloning; Apache Druid for sub-second analytics; real-time PubNub market ingestion; Albero Azure service decision tree React app
| title | author | length |
|---|---|---|
| From telescope to data centre: adventures in astronomical data pipelines | Simon O'Toole | 45m |
Pipeline moves nightly AAT data from Siding Spring to Sydney and Canberra using a Minifi agent into an Apache NiFi flow that extracts FITS headers into… | ||
| Gone Streaming: dbt+Materialize | Jessica Laughlin | 25m |
Claim: dbt cannot keep models fresh for streaming data but paired with Materialize it does. The setup uses the dbt-materialize adapter to create incremental… | ||
| A Single Data Platform for All of Your Workloads | Stephen Weingartner | 24m |
Claims a single Snowflake platform replaces multiple silos by storing data as micro-partitions on S3/Blob with compute separated and spun up in seconds… | ||
| Putting your data warehouse to work - Reverse ETL & Operational Analytics | Tejas Manohar | 26m |
Claim: move modeled customer tables from the data warehouse into SaaS tools to power real workflows. Implementation uses BigQuery + dbt as source… | ||
| Snowflake and dbt -- Our Journey to the Cloud | Philipa Sidaway, Juliana Zhu | 21m |
Migration uses Snowflake plus dbt with Git and buildkite CI/CD, Snowpipe from S3 (kinesis notifications) and 5 tran connectors. Pipeline changes reduced 8… | ||
| Situational Intelligence with Analytics in Motion | Jag Dhillon | 22m |
Imply uses Apache Druid to combine streaming and batch data for interactive analytics with sub-second queries and support for millions of events per second… | ||
| Albero - Decision Tree “when to use what – the full Azure data estate | Elizabeth Antoine | 20m |
An open source decision tree named Albero maps read/write patterns across 20+ Azure data services and claims 10,000+ active users and ~110 GitHub issues… | ||
| What is a Data Mesh - And How Not To Mesh it Up | Barr Moses | 41m |
Defines Data Mesh as domain-owned data plus global governance and gives concrete rollout steps: use a self-serve data platform, assign ownership via a RACI… | ||
| Data Quality with Great Expectations and Airflow in a Reverse-ETL World | Sarah Krasnik | 22m |
Shows how Great Expectations plus Airflow gate reverse ETL pipelines into tools like Census after dbt models in Snowflake, preventing paid-ads from shipping… | ||
| Reliable data engineering made easy | Mahdi Askari | 22m |
Claims that Delta Live Tables turns SQL/pyETL into production pipelines on Databricks Lakehouse, automating dependency graphing, lineage, checkpointing, replay… | ||
| Data quality: the key to long term happiness | Mike Robins | 18m |
Argues that manufacturing-style statistical process control maps to data pipelines by versioning collection, processing and models, capturing anomalies instead… | ||
| Building a Data Platform at Assembly Payments | Roopak Madhu | 10m |
Assembly Payments builds a layered data lake using RDS and DynamoDB sources, AWS DMS/Firehose/Lambda for ingestion, Glue for landing-to-raw, DBT on Fargate… | ||
| Streaming data analytics with Apache Flink | Masudur Rahaman Sayem | 24m |
Claims that a serverless Flink on Kinesis Data Analytics runs real-time pipelines at sub‑second latency, auto‑scales and provides multi‑AZ high availability… | ||
| Teleport Data: The future of data | John Cosgrove | 18m |
Claims a new snapshot sync called Teleport that Fivetran acquires and integrates to avoid CDC and log dependence, cutting setup from hours to minutes while… | ||
| The Rise & Downfall of the Data Engineer REVISITED | Maxime Beauchemin | 38m |
Revisits 2017 claims and argues the data stack now favors hosted pay-as-you-go platforms (Snowflake, BigQuery), shrinking internal infra teams while ELT… | ||
| Data Engineers: Privacy Is Your Problem | Stephen Bailey | 28m |
Argues that data engineers must own privacy by building a metadata-driven access layer that abstracts governance from each warehouse and app, using tools like… | ||
| From experiment to production - a journey of a machine learning model | Xin Liang | 22m |
Shows a concrete ML pipeline that takes an IMDB sentiment model from experiments through CI/CD to production, using Weights & Biases for experiment and… | ||
| Chain-speed inference for Computer Vision Pipelines | Mercedes Speroni | 18m |
Proposes a concrete pipeline that separates a local storage, a message broker queue and a dedicated machine learning service that must return predictions… | ||
| Trust, Knowledge and your Data. Our approach at KADA to building a great data product | Dean Nguyen | 14m |
KADA reports that 95% fail of dashboards, models, apps and APIs never get used long term and prescribes five concrete practices: build an audience by pushing… | ||
| Shift-left testing : Building reliable Data Pipelines | Harmeet Sokhi | 20m |
Argues that shifting tests left—unit, component, flow and runtime quality checks—catches errors earlier and reduces cost (defects found early are 100x… | ||
| Beginners guide to Azure MLOps | Chris Benson | 27m |
Beginner-level walkthrough of Azure MLOps showing a real engineering-client pipeline: uses Azure Machine Learning workspace, Databricks, Jupyter/VS Code… | ||
| Intelligent Serverless and Scalable Real-Time Data Pipeline using Kinesis, Fargate and CFN | Yogesh Sharma | 37m |
Shows a working pipeline built with Kinesis Data Streams, Fargate tasks, and a CloudFormation stack where producers push into a one‑shard stream and Firehose… | ||
| Better, Faster, Stronger Streaming: Your First Dive into Flink SQL | Caito Scherr | 23m |
Claims that you can build a working stream pipeline in minutes using Flink SQL client and a Flink cluster (Java 8/11 required), demonstrating a local CSV… | ||
| Metrics-driven Data Architectures | Ned Letcher | 26m |
Argues that metric sprawl causes rework and bad decisions and that treating metrics as first-class, versioned artifacts fixes it. Surveys concrete tooling: dbt… | ||
| Modern Data Warehouse for Small and Medium Business | Galina Polyakova | 12m |
SMB customers with data warehouses in the 300–600 MB range and not expected to exceed 1 TB use a hybrid pattern: keep the relational warehouse on Azure SQL DB… | ||
| Kickstarting a greenfield data project | Vinay Kulkarni | 19m |
Greenfield platform centralizes fragmented systems into an immutable Azure Blob Storage data lake (parquet, hive-style time partitions, cold-tier after one… | ||
| Logging Apache Spark - How we made it easy | Simona Meriam | 14m |
Nielsen runs >50 TB and billions of events through Spark on EMR and needed real-time log access instead of SSH, YARN UI or delayed S3 dumps. The team installs… | ||
| nothing matches | ||