Building a search engine with Clojure
summary
A small Clojure search engine is implemented end to end: a crawler (directed or weighted crawler with a max-depth limit), HTML extraction with Enlive, HTTP fetches via clj-http, storage in CouchDB, and indexing with Apache Lucene (demonstrates Lucene 4.0 features like byte storage and near‑realtime). Demo indexes Planet Clojure blogs and is runnable from GitHub; it works at small scale but notes multithreaded duplicate detection and field‑creation takeaways and warns it isn’t optimized for large distributed crawls.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:12Build search engine in Clojure
- 2:55Why Clojure for search
- 3:56Core components
- 7:27Crawling strategies
- 15:12Data extraction challenges
- 19:31Lucene 4.0 features
- 24:04Document handling
- 27:40Querying and search
- 31:55Demo and Q&A
related talks
41mAn introduction to search-index.js
28mBuilding a Cutting Edge Search Engine in Go
36mSearch on the fly: how to lighten your Big Data
42mBuilding a real-time news search engine #bbuzz
51mBuilding a Search Engine: From Thorns to Stars
34mSearch Engine in production with Rust
25mApache Lucene: From Text Indexing to Artificial Intelligence
27mPowerful Data Access in Clojure