Analyzing Pwned Passwords with Apache Spark
summary
Uses Apache Spark 2.x and a local Zeppelin notebook to process Troy Hunt’s Pwned Passwords dump (originally ~500M breached credentials) after joining down to about 700,000 joined plaintexts. Demonstrates DataFrame/Dataset SQL, UDFs, length/count aggregation and filtering (bad-words list), finds 6-character passwords most common, shows "password" >3M occurrences and notes "123456" ~20M seen; writes Parquet to S3, shows the pipeline runs and highlights cluster operational pain points and obscure Spark error messages.
AI-written summary · about
chapters (34)
- 0:00Introduction
- 1:27What is Twilio
- 2:23Agenda
- 2:49What is Spark
- 3:35We dont need Spark
- 5:13Data Science Data Engineering
- 6:00RDD
- 6:33GroupByKey
- 9:00DataSets
- 12:12State of Password
- 13:22Have I Been Owned
- 14:22The Data
- 15:39Schema Check
- 18:15Most Popular Passwords
- 18:47Length Column
- 19:07Run Raw Sequel
- 20:42Filtering Passwords
- 21:58Password Data
- 22:28Schema Inference
- 24:30UserDefined Functions
- 25:14Results
- 25:43Dog Rights
- 26:00Benefits of Spark
- 26:37Challenges
- 27:30Nested Error Messages
- 28:08Apache Spark Documentation
- 29:05Security Implications
- 29:23Data Privacy
- 30:15Security
- 31:15What can you do
- 32:05Thank you
- 32:21Conclusion
- 32:45Closing
- 33:13Audience Questions
related talks
39mAnalyzing Pwned Passwords with Apache Spark
42mAnalyzing Pwned Passwords with Apache Spark
01h 02mI’m Pwned. You’re Pwned. We’re All Pwned
40mFantastic Passwords and Where to Find Them
50mHave I Been Pwned?
31m680,000 Plaintext Passwords: An Open Source Love Story in Three Acts
47mLessons From Billions of Breached Records
30mThink Like a Hacker