Data Driven Cache Invalidation
summary
Demonstrates a concrete system: a Django app on PostgreSQL, Varnish fronting it, and DB-side invalidation using PostgreSQL triggers that either call PL/Python Varnish admin or enqueue events into pgq (SkyTools). A Python consumer reads pgq, issues regex Varnish purge via pgq calls, and retries across multiple Varnish servers. Example uses 1-hour TTL, 32MB Varnish cache, and targets “millions of hits a minute”; the setup works in demo, with pgq providing reliable async retries and scaling.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 1:09Caching challenges
- 3:39Caching layers
- 4:42Demo setup
- 6:50Django blog app
- 10:49Adding Varnish
- 14:46Fixed TTL problems
- 16:49On-demand invalidation
- 18:16App-only invalidation limits
- 19:42DB-driven invalidation
- 21:08Triggers with PL/Python
- 26:47Model mapping issues
- 28:12Async notifications with pgq
- 31:24Installing pgq
- 34:34Python pgq consumer
- 37:22Triggers using pgq
- 39:09Testing pgq invalidation
- 41:04Monitoring pgq queues
- 43:11Custom HTTP headers
- 45:20Invalidation best practices
- 46:50Q&A
related talks
47mReal World Django Q&A
47mReal World Django Q&A
35mCaching everywhere
39mHigh Performance Django: From Runserver to Reddit Hugs
45mCaching: A trip down the rabbit hole
46mA Hitchhiker’s Guide to Caching Patterns
56mCaching the uncacheable in Varnish
58mCaching the uncacheable: delivering personalized experiences without sacrificing performance