Horizontal Scaling of a Web3 system
summary
Migrated web3.storage off a slow bootstrap IPFS setup into a design using S3, SQS, Lambda, DynamoDB and EKS to become a fully stateless system that removes long DHT boot times (previously ~48 hours to bootstrap) and leverages Hydra and HTTP-based indexers to publish 10k-entry advertisements. Results: indexing latency drops from minutes to milliseconds and the deployment handles 180 million CAR files in six months (roughly 1M uploads/day, 16B blocks indexed), proving practical infinite horizontal scalability.
AI-written summary · about
chapters (13)
- 0:00Horizontal scaling in Web3
- 4:16Web3 Storage challenges
- 7:22Elastic IPFS architecture
- 10:35Data storage and indexing
- 13:06Indexing workflow
- 16:40DHT and Hydra nodes
- 21:57Indexer for HTTP ingestion
- 28:21Publishing concurrency
- 33:21PIV BitSwap stateless
- 35:26Performance metrics
- 37:59Elastic IPFS lessons
- 40:05Keep it simple stupid rule for Web3
- 41:52Final thoughts
related talks
27mCreate scaled customer experiences with digital assets
38mWeb3: Build, launch and scale your Web3 startup
45mHow to build decentralized serverless html5 applications
24mBest of Both Worlds: Exploring the Role of Centralisation in IPFS
43mFrom machine to digital services: Pioneering the future of manufacturing
28mThe hidden infrastructure powering our frontends
26mLaunch your dApp on decentralised Cloud
29mScaling data from the lake to the mesh via OSS