Processing Robot Data at Scale with R and Kubernetes

on YouTube ↗08m

" summary

R runs in Docker containers on Kubernetes to preprocess several hundred GB of compressed JSONL; Pachyderm pipelines version inputs and outputs so reruns propagate automatically. Data extraction performs nearest-timestamp joins to fuse sensor streams into single observations, then feature engineering uses a 42‑second lookback to label pre-failure windows. Parallelization relies on the future package and multicore containers, and tests use Pachyderm on an 80 CPU / 520 GB RAM cluster under $10/hr.

AI-written summary · about

" chapters (9)

1. 0:00Intro
2. 0:55Robots are everywhere
3. 1:23In Orbit
4. 1:44Problems
5. 2:43Feature Engineering
6. 3:40Data Pipeline
7. 5:47Parallelization
8. 6:37Implications
9. 7:11Outro

" related talks

• 34mPachyderm: Unlock the Power of Kubernetes for Big DataJoey Zwicker · KubeCon + CloudNativeCon North America 2016
• 22mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case StudyAnish Asthana · DevConf.CZ 2025
• 33mCompliant Data Management and Machine Learning on KubernetesDaniel Whitenack · KubeCon + CloudNativeCon Europe 2018
• 22mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case S...Anish Asthana, Mohammad Nassar · Open Source Summit North America 2025
• 21mScaling and automating R workflows with Kubernetes and AirflowIsaac Florence · rstudio::conf(2022)
• 15mSave an ocean of time: streamline data wrangling with RDanielle Dempsey · rstudio::conf 2019
• 27mGenerative AI Model Data Pre-Training on Kubernetes: A Use Case St...Anish Asthana, Alexey Roytman · KubeCon + CloudNativeCon Europe 2025
• 34mBuilding GPU-Accelerated Workflows with TensorFlow and Kubernetes [I]Daniel Whitenack · KubeCon + CloudNativeCon North America 2017