Why we had to Build a Petabyte Data Lake to Fix Indexing
summary
Claims building a petabyte data lake (raised $50M) to replace RPC limits, running on a decentralized SP network of 3,000 nodes and ClickHouse-backed archives. A thin portal API unifies realtime and historical multi-chain queries with near-zero overhead, plus an open SDK and SP Cloud for hosted indexers. Production users (GMAX, Squid) load pipelines into ClickHouse, use the portal for unified events, and report token-launch detection within seconds and successful 100x signal captures.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:58RPC interface limitations
- 3:24Blockchain data interface design
- 5:11Early data solutions
- 6:57Centralized data drawbacks
- 7:41Decentralized data lake architecture
- 9:30SP Network as data lake
- 10:57Portal ecosystem and tools
- 12:00Morph case study
- 12:43GMAX trading case study
- 13:05Squid AI case study
- 14:55Portal documentation
- 15:37Q&A
related talks
17mBuilding Real-Time Data Lake with ClickHouse
40mUnbundling of the Cloud Data Warehouse: Open Source Databases and Data Lakes
45mClickHouse: what is behind the fastest columnar database
49mBuilding a 100‑Petabyte Analytics Warehouse
27mBuilding a PostgreSQL data warehouse
32mHow open source is re-shaping the cloud data warehouse landscape
28mRethinking Status Quo: Open Source and the Cloud Data Warehouse
51mWhat You Need to Know About ClickHouse Architecture