How Web Data Infrastructure Powers the Next Generation of AI
summary
Oxylabs describes building a production web-data layer: a video API in under two weeks handling multi-petabyte jobs, iterating from transcripts to subtitles and metadata; a search pipeline redesigned from 4s to 650 ms P90 and 550 ms average by cutting page layout and tuning proxies parsers sessions and rendering; growth from 400 million to 6 billion daily requests and work toward 60k–100k RPS; major engineering pain points are browser-based scraping bottlenecks, observability at scale, and organic-load testing.
AI-written summary · about
chapters (12)
- 0:00Infrastructure matters for AI
- 1:06Static to real-time data needs
- 3:14Video API under extreme pressure
- 6:24Search data critical for AI
- 7:49Sub-second search delivery challenge
- 10:20Browser latency and speed hunt
- 13:09Speed defines AI products
- 13:53Adapting to massive scale
- 16:21Bottlenecks of web scrapers
- 18:11Scale keeps growing
- 19:38Oxylabs in AI infrastructure
- 20:19Better access powers AI
related talks
39mBig Data Information Architecture for AI
24mState of client side AI
41mBig Data Serving: The Last Frontier. Processing and Inference at Scale in Real Time
24mAI and Web Development: Hype or Reality
22mThe media processing pipelines behind AI
44mAI-Powered Search at Scale
24mWeb Apps of the Future With Web AI
46mPath of Data & AI Challenges