Exploratory Data Analysis with Spark and IntelliJ IDEA
summary
Demonstrates exploratory analysis on an IntelliJ IDEA code dump using Spark and an IntelliJ plugin that runs Zeppelin-style notebooks inside IDEA, converting RDDs to DataFrames and SQL views for fast queries and plots. Works with 8 million lines of code, finds ~7.5M Java lines and ~700k Kotlin lines, 76k Java files vs 6.5k Kotlin, median Java file 39 lines, largest file 30,451 lines, Pearson correlation between lines and bytes ≈ 0.77, and reports avg imports: Java 10.7, Kotlin 2.9.
AI-written summary · about
chapters (22)
- 0:00Introduction and speaker intro
- 1:16About the speaker and Spark basics
- 2:41What is exploratory data analysis
- 4:12Why use Spark for EDA
- 6:39Introducing the IntelliJ IDEA plugin
- 7:24Demo setup and data source
- 9:16Creating RDD and basic counts
- 10:20Converting RDD to DataFrame
- 13:37Filtering and descriptive stats
- 16:54File type distribution analysis
- 17:35Unique files and file counts
- 18:18Descriptive statistics for Java files
- 19:19Correlation and quantiles
- 21:08Exploring large Java files
- 22:33Excluding test data and re-analysis
- 23:43Visualizing file size distribution
- 25:34Comparing Kotlin file statistics
- 26:37Plotting lines vs bytes for Kotlin
- 28:08Analyzing import statements
- 30:20Advanced import pattern analysis
- 32:06Summary and next steps
- 33:31Resources and Q&A
related talks
29mData Engineers Shouldn’t Suffer, or Some Things About Big Data Tools for IntelliJ IDEA
01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live coding
43mApache Zeppelin, the missing GUI for your Big Data back-end
01h 36mSpark for Scala Developers
43mBoost Your Productivity with IntelliJ, Scala Plugin and More
41mApache Zeppelin, the missing GUI for your Big Data backend
27mIntelliJ IDEA Tips & Tricks
51mDataFrame: Kotlin's Innovative Approach to Data Structures