Applying Functional Programming Principles to Large Scale Data Processing

on YouTube ↗36m

" summary

Indix processes about 4 TB of fresh crawl data per day and 450 million product URLs, and replaces mutable shared DB rows with an append-only log on HDFS (or Kafka) plus read-time reconciliation so every update is recoverable. The pipeline runs on Hadoop using Cascading abstractions and Scalding on Hadoop for concise jobs, with Spark only for in-memory iterations. Aggregations are expressed as monoids for aggregation so parallel counts, max/min and metrics scale and debugging and retries work.

AI-written summary · about

" chapters (10)

1. 0:00Functional data processing intro
2. 5:19Large scale pipeline challenges
3. 8:08Why databases fail data systems
4. 10:16Immutability and append-only storage
5. 14:52Side effects and pure functions
6. 17:19Monoids for parallel operations
7. 20:06Scaling with Cascading Scalding
8. 27:29Spark versus Hadoop
9. 28:33Metrics and aggregations heartbeat
10. 34:31Summary of functional principles

" related talks

• 44mDemocratizing data with an internal data pipeline platformManoj Mahalingam · Scale By the Bay 2017
• 01h 35mDeep Dive into the Big Data LandscapeDean Wampler, Eva Andreasson · GOTO Copenhagen 2014
• 40mProductionizing Big Data - Stories from the TrenchesRoksolana Diachuk · Scala Days 2023 Madrid
• 11mBig Data Analytics: Immutability & IdempotenceKnut Nesheim · GOTO Berlin 2014
• 45mData pipelines from zero to solidLars Albertsson · Jfokus 2016
• 05mInterviewNimbus Goehausen · Scale By The Bay 2018
• 41mIntroducing Cascalog: Functional Data Processing for HadoopStefan Hübner · EuroClojure 2012
• 31mOne Data Pipeline to Rule Them AllSam Kitajima-Kimbrel · PyCon AU 2017