Plowing Through Data Building Flexible Pipelines with Go
summary
Bayer Crop Science builds a reusable, compile‑time safe pipeline library in Go using parametric polymorphism (generics) that replaces hard‑coded soil‑sample handlers with a generic pipeline package supporting start/pipe/end handlers, feed structs, batching, context propagation, tracing and structured logging. Team of 18 operates 36 APIs, 4 distributed systems and 23 async processors; the library eases harvest spikes (month‑long peaks after 3–4 months of growth), reduces copy‑paste, and is production used with example code in the repo.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:32Crop science landscape
- 3:41Data pipeline challenges
- 7:57Pipeline requirements
- 9:21Hard-coded MVP problems
- 12:53Go generics solution
- 17:12Pipeline design comparison
- 17:56Generic architecture benefits
- 19:43Business challenges addressed
- 20:46Summary
- 21:49Q&A
related talks
28mManaging and Scaling Real Time Data Pipelines using Go
22mOne Billion Row Challenge in Golang
20mMaking Data Pipelines in R: A Story From A “Self-Taught” Perspective- 20mMaking Data Pipelines in R: A Story From A “Self-Taught” Perspective
30mFrom Service To Platform: A Ranking System in Go
24mA Gentle Introduction to Building Data-Intensive Applications
30mHow Go Helped Accelerate Software Delivery at the BBC
12mLeveraging Go for Efficient Infrastructure&Data Handling