How to Extract Anything from the Entire Internet
summary
Presentation describes Yandex.Products, a system that crawls and indexes product offers across the entire internet, including large marketplaces (Ozon, Wildberries, Yandex.Market) and small shops with only dozens or hundreds of items. It covers the problems of finding and aggregating current offers from heterogeneous sites and the approach used to search, extract and present product data from both major marketplaces and long-tail online stores.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 3:38Task scope and requirements
- 5:46Price parsing problem
- 7:10Microdata and limits
- 8:14Selectors and manual parsers
- 10:42Manual approach issues
- 13:32DOM tree node classification
- 17:10Parsing flow optimization
- 21:06Price relevance and updates
- 24:58Fast price update loop
- 26:02Other parsing uses
- 26:59Q&A
related talks
45mOptimizations for Yandex Search Results
39mHow Data Delivery Evolved for Yandex’s Ad Engine
21mPreparing Yandex search data — the library and processes we built
51mSuperspeed. Yandex unified express delivery platform
41mReliability of Data Delivery in Stream Processing
41mSEO and the Future of Search
39mOn the Present and Future of the Browser
04mHighlights - Data and Business