So, You Want to "Do Big Data?"
summary
A ML-enabled fitness system that ingests 50 Hz sensors and ibeacons, classifies one-second setup movements, and ships 10 MB neural models to phones is described end-to-end and shown working. Uses Akka cluster and CQRS for user profiles and ingestion, stores events in Cassandra, runs training as Python jobs with Spark jobs on spot instances and Keras/TensorFlow backends, and delivers models via CDN to iOS where vectorized inference uses ~5% CPU. Lessons: naive event-journal training fails, need model/version storage and test splits.
AI-written summary · about
chapters (19)
- 0:00Demo
- 4:10Exercise
- 10:24Pebble
- 11:31Classification
- 15:05Mobile App
- 17:50Serialization
- 18:56Deferred Success
- 20:12The Right Way
- 23:03Technologies
- 23:39Acha
- 24:02Spark
- 26:22Machine Learning
- 27:49Python
- 33:55Cassandra
- 34:16Production
- 34:46Image generation
- 35:20Cassandra production
- 36:25Spot instances
- 37:48Future projects
related talks
01h 01mImplementing Exercise Wearables Analysis w/Spark and Akka
17mAI workflow for large scale deployment of far-edge ML devices
44mExercise Analysis
52mMuvr
31mRunning Machine Learning Analytics On Traces
04mInterview
17mCreating & operating ML models from event-based data using feature stor...
37mBig Data Analytics, Machine Learning, Real Time Processing