In-memory data pipeline and warehouse at scale
summary
Live demo builds an in-memory pipeline that ingests millions of financial transactions from S3 → Tachyon (Tachyon backed by HDFS), transforms and balances partitions with zipWithIndex + repartition, and writes Parquet into Tachyon for interactive Spark SQL querying. A skewed join shows 70% of tasks finish <2s while one hits 17s; replacing the join with a broadcast country map drops task time to ~7s and makes runtime near-linear. They run jobs via a Spark Job REST server that spawns per-context JVMs and expose a Spark SQL REST UI called Joe's.
AI-written summary · about
chapters (7)
- 0:00Introduction and use case
- 2:11Data pipeline construction
- 8:11Data transformation techniques
- 11:54Spark job management
- 18:06Building an in-memory warehouse
- 23:52Tachyon and performance
- 29:06Jows and final demonstration
related talks
42mBuilding Robust Streaming Data Pipelines with Apache Spark
52mSolving Analytical Problems using Apache Spark
19mApache Beam pipelines at 100TB+ scale using Apache Spark
11mScaling R with Spark
24mImproving interactive querying experience on Spark SQL
24mTaming Data Skew in Production ML Pipelines
24mOrchestrating Scalable Data Pipelines with Apache Toree, YuniKorn, Spark, and Airflow
01h 20mSupercharge Your Real-Time Analytics: Snowpipe Streaming & Dynamic Tables in Snowflake