Analyzing Pwned Passwords with Apache Spark
summary
Demo loads ~500 million breached credential pairs (uses a ~289M truncated sample) into Apache Spark via a Zeppelin notebook, reads CSV/JSON with schema inference, converts to DataFrames/SQL DSL, filters out 67 banned words, adds length column, and runs joins and UDF normalization. Findings: "123456" appears >20 million times, "charlie" rises to ~278k after lowercasing. Runs locally for the demo, highlights opaque JVM stack traces, cluster deployment quirks, and that Spark is fast but operationally tricky.
AI-written summary · about
chapters (11)
- 0:00Start
- 3:40Introducing Apache Spark
- 4:03The Apache Spark Project
- 5:52Data Engineering
- 9:01Creating the Data Frames Api
- 13:27State of Passwords
- 17:01Read in the Password Data
- 22:42Where Clauses
- 22:56Variable Replacement
- 26:52Udf Operator
- 32:10Implications of Big Data Insecurity
