Pathikrit Bhowmick Interview

on YouTube ↗03m

" summary

Coatue Management runs Scala and Spark pipelines on multi‑billion‑row datasets and faces severe skew where some nodes overload and others sit idle, breaking even basic groupBy operations. Moving from tens of billions to hundreds of billions prompts a shift from batch to incremental processing to enable fast iteration. Code follows functional, immutable patterns so a small team can reason about systems; the claim is that immutability reduces logic bugs while operational data misunderstandings remain.

AI-written summary · about

" chapters (5)

1. 0:00Introduction
2. 0:28Key issues
3. 1:16Whats next
4. 1:42Functional programming idioms
5. 2:22Outro

" related talks

• 05mInterviewNimbus Goehausen · Scale By The Bay 2018
• 40mProductionizing Big Data - Stories from the TrenchesRoksolana Diachuk · Scala Days 2023 Madrid
• 42mBuilding a Company on ScalaDag Liodden · Scala Days Copenhagen 2017
• 04minterviewUmayah Abdennabi · Scale By The Bay 2019
• 38mScaling ScalaJulien Tournay, Filipe Regadas · Scala Love in the City 2021
• 47mKeeping the “fun” in Apache Spark Datasets and FPHolden Karau · Scala Days New York 2018
• 01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live codingOleg Klimenko · Java ON 2022
• 45mScala and the JVM as a Big Data Platform Lessons from Apache SparkDean Wampler · Scala Days Berlin 2016