Building Efficient Web Scrapers
Lightningat RustConf 2026 · on Wednesday, 2026-09-09· at 17:15· Room 511 | Palais des Congrès de Montréal, Palais des Congrès de Montréal·
summary
Pipeline turns millions of raw fetches into structured data on a single machine by splitting responsibilities: Rust handles high‑throughput fetching with Tokio and Reqwest using all cores and streaming results into Redis as the queue, while Python workers run under multiprocessing to parse in parallel and emit Polars dataframes. Large binary blobs like Zarr use a claim‑check to spill to fast temp storage and pass pointers, preventing RAM exhaustion and keeping throughput steady.
AI-written summary · about
related talks
16mEmpowering Fearless Data Engineering
39mChallenges of Building Blazing Fast Data APIs
29mLessons Learned: Porting a Streaming Data Pipeline from Scala to Rust.
55mScaling out operational research with Rust
41mParsing Safely, from 500MB/s to 2GB/s
25mWTF, Rust for Web Development??
59mParallel Programming in Rust: Techniques for Blazing Speed
19mProduction Unsafe Rust