How To Remove Data Siloes: Tips & Considerations
summary
This describes a CDP built to collapse silos by ingesting streaming and batch sources with FluentD and Fluentbit, loading with Embulk and Digdag, and running ML via Apache Hivemall on Treasure Data’s platform. Claims: petabytes move daily, more than one million queries run per day, one customer ingests ~20 billion records a day, and 1,000+ connectors exist. Real outcomes include telematics-triggered service bookings and personalized offers that work in production; platform is backed by ~$234M funding and exposes Plasma DB for transparent querying.
AI-written summary · about
chapters (11)
- 0:00Introduction and speaker background
- 1:24Disconnected customer experiences explained
- 5:37Positive connected customer experience example
- 9:51Technologies enabling data unification
- 14:05Batch data ingestion with M-Bulk
- 14:47ETL pipelines with DigDag
- 16:53Machine learning with Apache Hivemall
- 18:16Treasure Data’s unified platform
- 20:21Commercial and compliance overview
- 22:48Scalability and real-world usage
- 23:52Q&A and booth information
related talks
32mConnecting Silos In Real-time With Data Virtualization
45mModern Data Architecture
29mSimplifying data flows
01h 35mDeep Dive into the Big Data Landscape
24mA Single Data Platform for All of Your Workloads
01h 03mDiscovering the value of a cloud data platform
38mKeynote: The Evolution of Data Infrastructure at Splunk
53mData Mesh in Practice - Assuring Data Quality at Scale