Low latency scalable web crawling on Apache Storm #bbuzz
summary
Storm-Crawler is an Apache Storm SDK that implements a multi-threaded fetcher bolt with per-host politeness queues, a parser pipeline (Tika for non‑HTML, HTML parser for links and metadata), and connectors for ElasticSearch + Kibana for indexing and metrics. It claims low latency and easy scaling, ships on Maven Central, grew from one committer to four in ~18 months, and runs in production on EC2 using RabbitMQ or SQS spouts and DynamoDB for state. Example topologies require only a few Java classes and configs.
AI-written summary · about
chapters (13)
- 0:00Introduction and speaker intro
- 1:16About Digital Pebble and Apache projects
- 2:42What is Storm Crawler
- 5:14Storm Crawler vs Nutch comparison
- 8:06Apache Storm basics
- 9:10Storm Crawler architecture and modules
- 12:22Basic topology example
- 13:50Handling recursive crawls
- 16:20Elasticsearch integration
- 17:27How to use Storm Crawler
- 18:10Monitoring with Storm UI
- 18:52Real-world use cases
- 19:12Future plans and resources
related talks
37mStorm, a Distributed Real-time Data Processing System
40mBrace yourself! Storm is coming!
46mClojure Scaling the Event Stream
04mEvan Chan Interview
26mHow to Pick Your Next Streaming Architecture
50mProcessing Billions of Events in Real-Time with Heron
18mStormdriver One Year Later
40mDecomposing a live application with confidence