Document Lake – Highly Scalable Document Repository
summary
Centralized repository stores 5 billion documents (≈3 PB) across regions, ingests ~10 million files/day, and enforces immutable WORM retention with feed monitoring and traceability tied to regulatory rules. Storage uses an S3-compatible API layered on ECS object storage with chunking for many small files; metadata and OCR go into an ElasticSearch index (150+ nodes) for fast search and a hot/cold strategy, with index reconstruction within 24 hours and intended indexing SLAs around five minutes.
AI-written summary · about
chapters (7)
- 0:00Introduction to Document Lake
- 2:18Purpose and evolution of Document Lake
- 8:56Use cases and regulatory requirements
- 13:52High-level architecture overview
- 18:06Storage and indexing deep dive
- 26:13Key learnings and conclusions
- 29:44Q&A session
related talks
19mAchieving Amazon S3 data lake resilience at LexisNexis
34mBest Practices for Implementing a Data Lake in Amazon S3
47mBuilding Data Lake Platform Fully Serverless
22mBig data tools
19mBuilding modern data lakes on Amazon S3
43mManaging a Data Lake on AWS
34mData Lakehouse: Who, Why and How
16mBreakdown data silos - Build a data lake on Amazon S3