Exploring Wikipedia with Apache Spark: A Live Coding Demo

on YouTube ↗59m

" summary

Live coding uses Apache Spark on a Databricks EC2 setup to analyze Wikipedia pagecounts, clickstream and the live edits IRC feed, with an 8-node Databricks cluster ~300 GB memory reading Parquet on S3 and Spark CSV. English Wikipedia is 5.1M articles (~54 GB); a cached PageCounts DataFrame has ~2.2M rows and shows ~43% of articles requested in one hour. Streaming runs 3 second micro-batches into a table, joins edits to pageviews, visualizes flows from the clickstream 1.2 GB file, runs shortest path in ~62s and PageRank in ~35s.

AI-written summary · about

" chapters (8)

1. 0:00Introduction and setup
2. 3:11Apache Spark overview
3. 6:43Analyzing traffic data
4. 12:19Data frames and operations
5. 18:16Real-time stream processing
6. 29:09Clickstream analytics
7. 40:11Graph processing applications
8. 47:42Conclusion and Q&A

" related talks

• 44mData Science at Scale with Apache SparkDean Wampler · GOTO Chicago 2015
• 01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live codingOleg Klimenko · Java ON 2022
• 52mSolving Analytical Problems using Apache SparkRohit Bhardwaj · Developer Summit 2020
• 34mKick-Ass Tools and Processes for Scalable Data ScienceHelge Reikerås · ScaleConf 2019
• 01h 00mGetting Started with SparkNikhil Nanivadekar · Devnexus 2020
• 01h 36mSpark for Scala DevelopersBrian M Clapper · Scala World 2017
• 01h 48mBig Data made easy with a Spark (3.0)Jean-Georges Perrin · All Things Open 2020
• 01h 25mApache Spark for Big Data ProcessingLudwine Probst, Ilayaperumal Gopinathan · SpringOne2GX 2015