Enabling data-driven development for autonomous driving at BMW
summary
BMW ingests multi‑terabyte car logs (several TB per car per hour, tens of cars → petabytes) into a dedicated Hadoop cluster, copies SSDs via buffer stations into HDFS, sets 2 GB block size and uses a custom MDF input format to load fragmented vehicle bus messages into Spark. A two‑step, shuffle‑minimizing partition stitch rebuilds frames, sensor‑specific decoders turn byte blobs into typed rows, and results are stored as Parquet columnar storage for search, scenario extraction, synthetic perturbation and running car software in the datacenter.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 6:53Learning how to drive
- 7:37How humans learn
- 8:52Sensor setup
- 9:25Data collection
- 10:35Shortcut
- 13:18Deep Dive
- 13:44Concepts
- 14:38Loggers
- 15:14SSD
- 15:57HDFS
- 18:15MDF
- 20:26Decoding
- 21:09Spark Data Containers
- 22:04Sidestep
- 22:59Simulation
- 23:30combinatorial boost
- 23:51question time
- 24:52Questions
- 30:02Certificate
related talks
27mData-driven and cloud-native ecosystems in the automotive industry
16mData und AI at Scale: Unlocking Innovation
37mThe global data plane for autonomous driving from Day 0 to Day 2 — Stephan Reimann and Simon Putz
24mCreating a Data Platform for Software-Defined Electric Vehicles
06mData Driven Performance
40mFast Cars, Big Data - How Streaming Can Help Formula 1 #bbuzz
35mDataFrames in Apache Spark - the analysts perspective
30mData Platform Infrastructure at Rivian: A Deep Dive