E-Commerce search at scale on Apache Lucene
summary
Amazon ports product search from a legacy C++ engine to Apache Lucene and proves it works in production, serving millions of shoppers; architecture uses near-real-time segment replication to copy write-once Lucene segments to deep replicas and concurrent searching to dispatch queries across segments on many cores. Index-time tricks include index sorting, index-time joins for offers, dimensional points for lightning deals, and automated nightly benchmarks that catch ~20% p99 regressions; a query-pattern index speeds QPS by ~30%. Multi-phase ranking applies expensive ML only to top candidates.
AI-written summary · about
chapters (23)
- 0:00Intro
- 0:42Outline
- 1:10Amazon
- 5:09Lucene features
- 7:19Upstream contributions
- 7:58Near realtime segment replication
- 9:33AWS features
- 13:28Sawtooth patterns
- 15:08Measuring performance
- 16:38Measuring query latency
- 18:19Query refresh times
- 19:26Query metrics
- 21:42Nightly benchmarks
- 22:36Text analysis
- 24:08Airplane
- 25:48Numbers
- 29:20Index Queries
- 31:00FB Growth Algorithm
- 31:54Full Text Results
- 33:22Lightning Deals
- 35:03Multiphase ranking
- 36:48Intermediate expense model
- 37:27Wrap up
related talks
41mScaling Search and Powering Vector-Based Retrieval With OpenSea...
24mKeeping Adobe Commerce Live Search results fresh in real time
30mIncrease text search speed & accuracy with Amazon OpenSearch Service
16mSolr Eclipse - Zomato's Journey to 64 Million Search Queries a Day
19mEnhance Search Performance At Scale with
30mCaching for Speed: Practical Performance Optimization in OpenSearch & Lucene
39mAccelerating Aggregations with Lucene-Aware Execution in OpenSearch
50mEvolution of an Ecommerce Application