Apache Spark on planet scale
summary
Claims a working parallel PBF reader (Java 8) plus a Spark native data source that partitions planet OSM so every executor reads different blocks in parallel, cutting full-planet ingest from a single-thread ~45 minutes to ad hoc Spark runs of ~2.5 minutes. Supports local/S3/HDFS, partition count and threads per partition, JVM/Scala Spark 2.11/2.12 compatible, GPL on GitHub and Maven; includes helper routines for extract/merge, renders a public-transport coverage map, and costs ~€7 for a short EC2 cluster run.
AI-written summary · about
chapters (19)
- 0:00How it works
- 1:34Introduction to Spark
- 3:02What is Spark
- 4:51Importing data
- 8:28Data structure
- 12:37Parallel PDF reader
- 14:17Synthetic tests
- 15:48Parallel TDF
- 17:20Spark Data Source
- 20:18How to use it
- 22:11Data Source Schema
- 23:56Pure or Send
- 29:07Calculate public transport coverage
- 29:35Extract data
- 31:55Public Transport
- 33:09Parallel ADF
- 34:58Github
- 35:56Questions
- 38:47Parallel PDF
related talks
21mAvoiding Spark Pitfalls at Scale
52mSolving Analytical Problems using Apache Spark
19mApache Beam pipelines at 100TB+ scale using Apache Spark
01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live coding
50mIntroduction to Real-Time Big Data with Apache Spark
34mHive on Spark #bbuzz
36mNo more struggles with Apache Spark (PySpark) workloads in production
39mStorage-Partitioned Join for Apache Spark