Making Big Datasets Searchable
summary
A fast API indexes Rapid7 Project Sonar DNS data so large lookups become file seeks not full-text queries: 183 GB / 1.8B records are parsed with a suffix-array parser, converted via scenaric robot, sorted, then a crowbar + Redis byte-offset index maps keys to exact file positions. Reverse-DNS and CIDR scans convert IPs to decimals and bucket offsets to avoid huge indexes. Old DB approach timed out under load; the new gRPC/HTTP server and Go streaming CLI return bulk results in under a second and sustain thousands of requests/sec.
AI-written summary · about
chapters (9)
- 0:00Start
- 0:36What?
- 1:46Extremely fast DNS API
- 2:03Reverse DNS queries
- 2:35Find Subdomains
- 6:55Sort time!
- 7:56Hash maps?
- 9:58Demo Time?
- 11:16Questions?
related talks
01h 00mHyper Speed: When Big Data Blooms
28mBuilding a Cutting Edge Search Engine in Go
36mLessons from building GitHub code search
31mExploring open data with BigQuery
44mSearching through large graphs using Elasticsearch
48mHyper Speed! When Big Data Blooms
39mWhen Rivers Speak Analyzing Massive Water Quality Datasets
47mFun with BIG Data with various databases