Democratizing data with an internal data pipeline platform

on YouTube ↗44m

" summary

Describes building an internal data pipeline platform at Indix to catalog and process varied datasets (HTML, Thrift, Avro, CSV, Google Sheets) using Spark. It covers dataset discovery with schemas, configurable pipelines via a wizard or JSON, predefined transformations (filter, project, sample, SQL), autoscaled dedicated Spark clusters, S3/SFTP/Sheets connectors, notebook client access, and operational scale (300+ pipelines, 100+ datasets).

AI-written summary · about

" chapters (5)

1. 0:00Start
2. 1:12Six Business Critical Indexes
3. 2:31Data Pipeline alndix
4. 4:34Scale of Data a Indix
5. 12:34Tech Stack

" related talks

• 36mApplying Functional Programming Principles to Large Scale Data ProcessingKishore Nallan · Functional Conf 2014
• 17mDemocratising data analyticsChristian Thiel, Edoardo Gatti · hub.berlin 2022
• 05mInterviewNimbus Goehausen · Scale By The Bay 2018
• 01h 35mDeep Dive into the Big Data LandscapeDean Wampler, Eva Andreasson · GOTO Copenhagen 2014
• 27mData Analytics Democratization: How ING Data Analytics Platform Bootstraps New...Krzysztof Adamski · Open Source Strategy Forum London 2021
• 04minterviewUmayah Abdennabi · Scale By The Bay 2019
• 31mOne Data Pipeline to Rule Them AllSam Kitajima-Kimbrel · PyCon AU 2017
• 42mBuilding Robust Streaming Data Pipelines with Apache SparkZak Hassan · Open Source Summit North America 2017