Detecting and Diagnosing Errors in Serving Archived Web Pages
summary
FidEx compares script-level execution traces between crawl and replay and matches pages using layout-tree + JS-write matching, not screenshots, to detect fidelity violations. The implementation traces which rewritten edits cause mismatches and automatically isolates problematic rewrites; evaluation on 80,000 pages from the top 1M shows a drop in reported false positives from ~70% to <10% and the applied fixes to PyWB reduce violating archived pages from 15% to 9%.
AI-written summary · about
chapters (11)
- 0:00Web archive fidelity issues
- 1:45How archives capture pages
- 4:31JavaScript rewriting challenges
- 6:35Detection methods and limitations
- 8:59Problem with screenshot comparisons
- 9:20Critical rendering path
- 10:43Fidelity detection with layout trees
- 12:06Fiddler versus baselines
- 13:53Diagnosing fidelity violations
- 14:57Fixing errors in PyWB
- 15:19Summary
related talks
47mReversing Engineering a Web Application - For Fun, Behavior & WAF Detection
48mPerformance Mistakes
47mHow to be a Performance Detective?
43mHinting at a better web
29mHigh Fidelity Web Crawling in Python
20mDon't Break The Web
35mModern Web Data Extraction: Techniques, Tools, Legal and Ethical Considerations
26mImproving Performance in Matu