Case Law and ML on Spark
summary
Migration from Apache Pig to Spark powers a pipeline that processes a legal corpus of about 7 million opinions and 54 million citations, running NER, LDA topics, TextRank summarization, K-means clustering and other classifiers, exporting to MongoDB, Solr and an in‑memory graph store (Cav) and using Parquet archives for reproducibility. The change yields concrete wins: build time drops from 10 days to 2 days, easier local experimentation and reproducible inputs, all run on modest Spark clusters and deployed on AWS.
AI-written summary · about
chapters (11)
- 0:00Introduction to talk
- 1:18What is case law
- 3:05Product overview and features
- 5:34Spark pipeline requirements
- 7:43Switching from Pig to Spark
- 10:12Why Spark makes you happier
- 11:38Machine learning as software
- 12:42Optimizing the build pipeline
- 15:15Parquet archives benefits
- 16:20Hiring and demo teaser
- 17:57Q&A session
related talks
44mData Science at Scale with Apache Spark
39mMachine Learning with Scala on Spark
50mApache Spark vs Machine Learning
01h 25mApache Spark for Big Data Processing
35mPySpark : Combining Machine Learning & Big Data
26mFrom Spark to Elasticsearch and Back
23mRethinking scalable machine learning with Spark ecosystem
44mApache Spark and ML workflows