Decentralized Technologies at the Internet Archive
summary
Internet Archive archives the visible web since 1996 and faces scale limits: tens or hundreds of billions of records break naive IPFS assumptions because provider records expire in 24 hours and the APFS DHT cannot hold that cardinality, so deployments shift to giant nodes / IPNI and storing CARs in S3 with range requests. Filecoin use includes tens of thousands of on‑chain deals and a tiny in‑house miner (~100 tips of power). Preservation, gated ML access, and authenticated crawls remain active engineering problems.
AI-written summary · about
chapters (14)
- 0:00Internet Archive mission
- 1:32Why decentralized tech
- 2:58Motivations for decentralization
- 4:41Scaling IPFS challenges
- 7:11Filecoin for storage
- 8:39Filecoin mining strategies
- 9:45Other decentralized tools
- 10:28Pitfalls in adoption
- 11:52Overspecified implementations
- 13:15Privacy projects like Tor
- 14:01TS Notary authenticity
- 15:25Gated access ML data
- 16:09Web3 preservation
- 16:52Collaboration invitation
related talks
24mUniversal Access to All Knowledge: Decentralization Experiments
41mIPFS at Web Scale
26mSaving Data to the DWeb
25mBuckets of Data: Strategies for Leveraging Off-chain Data Storage
28mOpening Remarks / Filecoin: What's Now Possible
13mARG / Estuary.tech
01h 04mBuilding Web3 Startups
54mIPFS & Filecoin 101