DuckDB Applications in Information Retrieval
summary
DuckDB ingests Protobuf/Parquet web-index dumps from the Open Web Search project and turns them into queryable tables in days rather than months, with Proto→Parquet→Duck tables created and exposed to the FTS extension (some Python ETL still needed). Inversion work runs on European supercomputers with parquet partitioning aligned to shard files. An experimental CRAFT-style pairwise document store is implemented and works for co‑occurrence queries, and vector search uses HNSW restricted by DuckDB filters so filtered candidate sets are passed to the ANN index. Code is available.
AI-written summary · about
related talks
28mAnalyze data easily with duckdb - and the implications on data architectures
40mDuckDB: Harnessing in-process analytics for data science and beyond
25mA quack of all trades: DuckDB, a versatile analytical database to keep in your toolkit
19mIn-Process Analytical Data Management with DuckDB
01h 02mAnalytics for not-so-big data with DuckDB
27mState of the Duck
12mGrep your lakehouse: Search-first retrieval for DuckDB-powered agents
27mIn Process Analytical Data Management with DuckDB