Software Best Practices for Data Pipelines: Form Energy's Multi-Day Storage R&D Pipeline 🌳

on YouTube ↗21m

" summary

Form Energy runs 100-hour iron-air batteries research with thousands of experiments and needs pipelines that deliver fresh correct data each morning. The stack uses Dagster, CircleCI regression tests, AWS EKS containers, infrastructure-as-code and standard Python logging, and adds structured events and tracing via OpenTelemetry/Honeycomb. Concrete wins include alerting on a Cron job that dropped daily rows to zero, spotting an API creeping toward a 30s Lambda timeout, and flame-graph traces that link pod errors to experiment IDs.

AI-written summary · about

" chapters (10)

1. 0:00Introduction to Form Energy
2. 2:10Form Energy’s Battery R&D
3. 3:32Data Pipeline Challenges
4. 5:18Quality Gaps in Pipelines
5. 7:23Pipeline Quality Stack
6. 10:51Structured Events and Tracing
7. 15:11OpenTelemetry in Pipelines
8. 18:42Open-Source Contribution
9. 19:50Reliable Data Delivery
10. 20:33Q&A

" related talks

• 30mDesigning Resilient Data PipelinesAndrew Bolin · SREcon19 Americas
• 34mData pipelines powered by Open source and funSayantika Banik · stackconf 2022
• 43mData Pipelines à la ModeTom Hall · Code Mesh LDN 2019
• 45mResilient Predictive Data PipelinesSiddharth "Sid" Anand · GOTO Chicago 2016
• 29mDagster: a Framework for Data Processing ApplicationsNick Shrock · Scale By The Bay 2019
• 19mBuilding Stateful Streaming PipelinesAnkit Jhalaria · Flink Forward Virtual 2020
• 45mDefining data pipelinesJuan Riaza · Commit Conf 2018
• 47mOpen Source for Energy Transition via Data Management - Eyes on a Major PrizeAarthi Thyagarajan · SustainabilityCon @ Open Source Summit North America 2023