Data Secrets from a Platform Engineer
summary
Claim that rewriting ORM queries to raw SQL cuts runtime from 8 hours to minutes; note warehouses lack DB guarantees like global IDs so entity‑resolution approaches break. Describe a deployed pipeline of two dozen scikit‑learn models serialized to S3, consumed via Kafka, invoked by Lambda, written into Elasticsearch, with numpy incompatibilities and checkpointing added because failures are normal. Recommend building pragmatic tools: an observability plane with Prometheus and Grafana, a historical downtime predictor, a custom anomaly detector, and synthetic test data from fine‑tuned LLMs.
AI-written summary · about
chapters (13)
- 0:00Welcome
- 1:03Data scientists as platform engineers
- 3:33Language modeling to platform engineering
- 5:57Queries took eight hours
- 8:25Databases vs warehouses
- 10:10NLP preprocessing work
- 13:01Debugging missing data
- 14:46Distributed systems quirks
- 20:24Models never reach production
- 24:00Platform engineers and data
- 27:35First deployed model lessons
- 31:10Collaborate with platform teams
- 37:13Call to curiosity
related talks
38mKeeping pace with data infrastructure so it doesn't outrun you
34mData Engineering Architecture
22mBuild Smarter, Don't Just Build
29mOwned By Statistics: How Kubeflow & MLOps Can Help Secure...
29mMachine learning in production
33mAccidental DataOps: How we matured our organization with data, and how you can too!
40mDesigning Data Driven Projects: Beyond ML Experimentation
39mHelp Me, Help You: Supporting Your Data