Indexing the Planet For Good
summary
An indexer for IPFS and FileCoin called IPNI publishes compact signed advertisement chains and uses context ID pointers so providers add or remove huge groups of CIDs with tiny messages. Indexers pull immutable DAG entries, ingest ~3 billion hashes/day into encrypted key‑value records (using double hashing), and serve lookups that return protocols, endpoints and metadata. The system already indexes ~32% of FileCoin (~500 PB), uses Pebble with custom merges, edge caching and a federation plan, and offers reader privacy so indexers can’t snoop on queries.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:51Traditional lookup limits
- 3:55Scalable indexing demands
- 5:39IPNI replaces decentralized
- 6:21IPNI protocol roles
- 9:30Announcing content in IPNI
- 10:12Fetching and processing data
- 12:21Ingestion and encryption
- 14:07Lookups in practice
- 15:32IPNI efficiency features
- 16:37Real-world scale and integration
- 17:39Development and future plans
- 18:44Q&A
related talks
54mIPFS & Filecoin 101
27mDecentralized Storage with IPFS: How does it work under the hood?
14mThe Eternal Tussle: Exploring the Role of Centralization in IPFS
12mWhy Do We Need a Decentralized Storage System?
45mIPFS - The Practical Bits
26mPLN Ecosystem Overview: All the things you can build on
31m🛠 Building Applications with Filecoin & FEVM
37mStorage Provider Overview