Live-Hack: Analyzing 7 years of Buzzwords at Scale #bbuzz

on YouTube ↗39m

" summary

Scrapes 2010–2016 Berlin Buzzwords sites with Scrapy, stores line‑delimited JSON on S3, and analyzes it in Databricks + Spark using regex cleanup, a stopword list and crude lemmatization. Scraping still breaks on missing content‑type headers and changed HTML, but the crawler finishes (~6 minutes for these archives) and the pipeline produces counts: sessions per year (2010 shows ~10 pages, 2012 and 2014 peak), top speaker appears 9 times (Ted Dunning), and Flink overtakes Spark at this conference as streaming and Kafka climb.

AI-written summary · about

" chapters (16)

1. 0:00Introduction and welcome
2. 1:11Speaker introduction and motivation
3. 3:18Plan for live data scraping
4. 9:02Challenges in web scraping
5. 13:02Choosing analysis tools
6. 15:12Live coding setup
7. 18:29Loading and inspecting scraped data
8. 21:01Text cleaning for analysis
9. 27:31Preparing cleaned dataset
10. 29:00Counting sessions by year
11. 30:25Top speakers over the years
12. 33:47Analyzing buzzword trends
13. 35:14Streaming frameworks comparison
14. 36:41Kafka and batch vs streaming
15. 37:24Scalability and conclusions
16. 38:51Q&A

" related talks

• 01h 36mSpark for Scala DevelopersBrian M Clapper · Scala World 2017
• 39mAnalyzing Pwned Passwords with Apache SparkKelley Robinson · Scala Days New York 2018
• 41mUsing Clojure+Spark to Find All the Topics on the InterwebsHunter Kelly · Clojure/conj 2015
• 28mLarge Scale Graph Mining with Spark: What I learned from mapping 15 million websitesWin Suen · PyGotham 2018
• 28mA complete Tweet index on Apache Lucene #bbuzzMichael Busch · Berlin Buzzwords 2015
• 47mSpeeding Event Processing to 1.6M/sec: Badoo’s ExperienceAleksandr Krasheninnikov · HighLoad++ Moscow 2018
• 33mAnalytics in the age of the Internet of Things #bbuzzLudwine Probst · Berlin Buzzwords 2015
• 01h 35mDeep Dive into the Big Data LandscapeDean Wampler, Eva Andreasson · GOTO Copenhagen 2014