Building a scalable data platform

on YouTube ↗32m

" summary

Covers patterns and components used to build Hotstar’s scalable data platform, including specific technologies/practices for data ingestion (ingestion patterns and a unified ingestion proxy), schema definitions, in‑flight enrichments, and handling noisy neighbors. Describes storage and availability strategies—decoupling storage and compute, storage patterns, and high availability—plus query lineage/optimization, consumption patterns, parity between streaming and batch data, and providing both a single GUI and programmatic interfaces.

AI-written summary · about

" chapters (7)

1. 0:00Intro
2. 0:34About Hotstar
3. 2:15The Platform
4. 6:49Data Ingestion: Pattern #2
5. 16:23Data Ingestion Pattern #3
6. 19:26Data Storage: Pattern #1
7. 25:38Data Consumption: Pattern #1

" related talks

• 24mA Gentle Introduction to Building Data-Intensive ApplicationsJoe Karlsson · Conf42 Golang 2022
• 25mPlatform Engineering for Modern Data InfrastructureRahul Joshi · Conf42 Platform Engineering 2025
• 39mHigh-volume Data Streaming and StorageGleicon Moraes · QCon São Paulo 2019
• 30mPlatform Oriented Data Architecture for Acceler...Vladimir Bacvanski · Global Artificial Intelligence Conference 2020
• 05mThe Future of Data WarehousingAnupam Singh, Brian Coyne · Strata Data Conference 2018 - New York, NY
• 05mInterviewDavid Winters · Scale By The Bay 2018
• 52mData Engineering in PracticeElton Almeida · QCon São Paulo 2018
• 43mAnalytics capabilities architectural patterns: A journey for data-driven insightsJohn Mousa · Voxxed Days Brussels 2023