Data Science in the Internet of Things with Python and Spark
summary
Prototype smart office collects 200+ sensors over 36 rooms for ~3.5 months via MQTT → Kinesis Firehose → S3, sensors emit on change so timestamps are irregular. Pipeline pivots observations to time-series, upsample to 1m then downsample to 7m to regularize gaps, then trains multi-step classifiers by framing targets as arrays and using a multi-output RandomForest for 6‑hour occupancy prediction. Automation runs on EMR via boto3 run_job_flow with bootstrap pip installs, parallelizing per room and persisting models and predictions to DynamoDB.
AI-written summary · about
chapters (6)
- 0:00IoT data science with Python & Spark
- 3:18Smart office IoT setup
- 10:28Pre-processing sensor time series
- 14:05Time series forecasting
- 21:32Scaling with Spark on AWS
- 29:19Future directions
related talks
39mBuilding smart IoT applications with Python and Spark
39mPython and IoT: From Chips and Bits to Data Science
20mAutomating Data Science for the IOT
33mAnalytics in the age of the Internet of Things #bbuzz
35mData science and Internet of Things on the edge
44mMethods and Tools for Time Series Data Science Problems
02h 00mData Science Driven Business Insights with Python and PySp
37mFrom Sensors to Insights at the Edge: Applying AI to IoT Data