Real Time Data Driving Innovation
summary
Shell describes RTDIP as an open source real-time data ingestion platform that already consumes over 3 million sensors globally and has stored about 4 trillion rows of time-series data. It uses Apache Spark and Python to pipeline messages from connectors (OPC UA, IEC 104, DNP3, 61850, Kafka, MQTT) into Delta Lake (S3/ADLS) and integrates with Databricks, Airflow/Dagster and secrets stores. More than 50 applications run on it, including predictive maintenance monitoring 17,000 pieces of equipment, and deployments span LNG, upstream, manufacturing and renewables.
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:45Definitions & Cautionary Note
- 1:55Two forces reshaping the energy system
- 4:49Open source as a key enabler
- 7:16Open Source Strategic Values
- 9:48Shell - LF Energy Collaboration
- 10:26Real Time Data Ingestion Platform Tangible Values
- 12:43Real Time Data Ingestion Platform (RTDIP) Pipelines SDK
- 15:47Real Time Data Ingestion Platform (RTDIP) Pipeline Comp
- 19:15Real Time Data Ingestion Platform (RTDIP) Pipelines Que
- 23:09Enabling the next generation of clean energy technology
related talks
47mOpen Source for Energy Transition via Data Management - Eyes on a Major Prize
24mCreating a Data Platform for Software-Defined Electric Vehicles
35mReal-time analytics at IoT scale
30mData Platform Infrastructure at Rivian: A Deep Dive
08mReal-Time Data Systems: Speed, Scale, and Resilience
38mRealtime Data Activation - Analytics, Intelligence & Decision Making
21mBuilding Data Integration Services In Real Time
10mData Rivers — The New Analytics Architecture