Save The World And Money with MongoDB Data Lakes
summary
Demo shows an automatic pipeline that archives cold MongoDB data to AWS S3 using Serverless Triggers/Realm and Atlas configuration, moving identifiable batches (example run moves 240 documents out of ~24,000, freeing ~52 MB). Atlas Data Lake serves those S3 JSON files so the system queries archived JSON via MongoDB Atlas Data Lake with zero downtime. Also notes Atlas Online Archive as a GUI option and gives code for a Python archiver.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 2:25Data growth cost problem
- 3:47Hot and cold data with IoT
- 5:54What is a data lake
- 7:39MongoDB Data Lake features
- 9:01Setting up hot data
- 10:45Auto-archiving with serverless
- 11:49Archiving to S3 verification
- 12:55Querying archived data
- 13:16Summary and next steps
related talks
46mBuild an in-place Amazon S3 data lake with
44mAutomatically scale a serverless app with Amazon Textract & MongoDB
47mBuilding Data Lake Platform Fully Serverless
14mBuild your data lakehouse with Starburst Galaxy
26mBuilding generative AI–enriched applications with AWS & MongoDB Atlas
40mGoing serverless with MongoDB
21mBuild the next big thing with MongoDB Atlas on Microsoft Azure
42mBuild and automate a modern serverless data lake on