How to write a search engine in 15 lines of code
summary
Claims a searchable engine core in 15 lines of code using the Unison language, backed by a local persistent index and a generic D index (~100 LOC) that uses rendezvous hashing for key-to-node assignment and a lazy intersect library (~70 LOC) to produce top-N results. Shows spawning and joining many nodes (example: spawn 10,000 nodes from a region), describes content-hash identified code distribution, and promises posted code and a demo.
AI-written summary · about
chapters (28)
- 0:00Intro
- 0:45Motivation
- 1:33Boring stuff
- 2:03What is a program
- 3:36History of computing
- 4:33Era of shrinkwrap software
- 5:05The internet
- 5:26Building large software systems
- 6:01Specialpurpose technologies
- 8:03Outline
- 8:46Unison Functions
- 11:39Unison Index
- 12:50Index Example
- 14:30Search Index
- 15:54Search Function
- 18:18Distributed Data Structures
- 21:39Creating Nodes
- 22:01Remote Node Spawn
- 23:04Remote Node Spawn Example
- 26:07Unison solution
- 27:59What is a code base
- 30:13Thank you
- 31:14Which advantages does Unison provide
- 31:32What is missing from Erlang
- 33:46Erlang vs Elixir
- 34:17Its not ready for production
- 35:01Is Unison inspired
- 35:52Applause
related talks
22mProgramming the worldwide elastic supercomputer with Unison
41mIntroduction to the Unison programming language
14mBringing Decentralized Search to Decentralized Services
40mNon-Obvious Things to Know When Coding in OpenSearch
01h 08mDistributed Java Databases Under the Hood
31mSharding 2.0: Unlocking Infinite Search Scalability in OpenSearch
15mUnconference Session: Dedicated Coordination Nodes - Why? When? How?
19mYou Know, for Search. Querying 24 Billion Records in 900ms