Blazing-Fast Serverless MapReduce Indexer for Apache Solr
summary
An indexer built as a serverless MapReduce pipeline uses AWS Lambda + Step Functions to fetch SQL query shards, write per-song intermediate files to S3, run parallel mappers that append to per-key files guarded by a DynamoDB lock table, and push batched updates to Apache Solr via SQS. It replaces 150 cumulative hours on three servers with a daily run under five hours, cutting runtime about 30x and cost to roughly 10% of the previous setup.
AI-written summary · about
chapters (15)
- 0:00Introduction and speaker intro
- 1:02Speaker background and company overview
- 2:32Problem of indexing relational data
- 4:00Trivial solutions and their flaws
- 5:28MapReduce pattern explained
- 6:32Word count example with MapReduce
- 9:03Applying MapReduce to database records
- 13:01AWS serverless implementation overview
- 15:14Fetcher component details
- 16:42Splitting data into multiple files
- 18:11Map function and concurrent file access
- 20:21Locking mechanism with DynamoDB
- 21:45Reducer function and SQS batching
- 23:56Production results and improvements
- 25:24Summary and Q&A
related talks
32mServerless Data Processing and Machine Learning
41mUnlocking Faster & Efficient Data Processing w/ Serverless
39mServerless in production, an experience report
49mUsing serverless functions for real-time observability
34mFunnel Rocket: A Serverless Query Engine
41mUsing Serverless & ARM64 for Real-Time Observability
17mBuild cost effective Asynchronous GraphQL APIs using Serverless.
32mUsing Serverless Functions for Real-time Observability