Apache Wayang: Bringing Big Data to the IoT
summary
Apache Wayang is presented as a scalable cross-platform orchestration layer that connects IoT storage and processing systems to run AI workflows without manual data plumbing. It demonstrates connectors for Apache IoTDB and Apache Cassandra and can move IoT data into Apache Spark or Flink for preprocessing, feed preprocessed data to an LSTM implemented via PySpark or other ML libraries, and persist models for inference with frameworks like PyTorch. A solar-panel energy-production use case illustrates end-to-end AIoT: raw IoT readings → Spark preprocessing → LSTM training → model stored in Cassandra → inference.
AI-written summary · about
chapters (26)
- 0:00Intro
- 2:28Big Data Landscape in 2012
- 2:56Data Analytics Back in 2010
- 3:30Data Analytics Today
- 4:14Big Data Landscape in 2020
- 4:34From an Elephant to a Jungle
- 5:22Tame Them to Live Together
- 6:36Targeted Running Example
- 7:24Roadmap
- 8:00Apache Wayang History
- 9:39Research Footprint
- 10:08Why Wayang?
- 11:00Apache Wayang's Goal
- 11:38Side-Effect Benefit
- 15:27Case 1: Platform Independence
- 18:19Polystores
- 22:11Opportunistic Cross-Platform
- 25:44Mandatory Cross-Platform
- 28:08Plan Inflation
- 29:45Cost-based Cross-Platform Optimizer
- 33:49Solar Energy Production Prediction
- 34:22Wayang's Plan (with Custom Ops)
- 35:22Wayang's Code Snippet
- 37:14Execution Plan Deployment
- 41:09Big Data-loT in Action
- 42:40Major Efforts On
related talks
42mWhy Choose One: Multi-Engine Analytics with Apache Wayang
20mBuilding realtime efficient queries in an IOT platform
41mApache Zeppelin, the missing GUI for your Big Data backend
50mData Science in the Enterprise
43mApache Zeppelin, the missing GUI for your Big Data back-end
43mHow Big Data is Powering the Internet of Things (IoT) Revolution
35mJVM based DeepLearing on IoT data with Apache Spark
01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live coding