Getting to 300 million requests a month with a one man AI/Dev-Ops team
summary
Scale story graph to 300 million requests a month as a one‑person AI/DevOps team by swapping cloud for self‑hosting, using Redis queues + PG Bouncer for background ML jobs, and running a dozen custom models including self‑hosted LLMs (≈1M LLM requests/day) to avoid a $770k/month API bill. Migrated from Heroku to cheap Hetzner dedicated servers via Cloud66, moved Postgres workloads to a 10+ node Yugabyte DB cluster, and fixed the Jan 1 outage by removing HAProxy cookie persistence.
AI-written summary · about
chapters (13)
- 0:00Scaling with one-person team
- 1:15From dreams to AI and DevOps
- 10:23Perseverance and timing in tech
- 17:17Scalability with queues and self-taught solutions
- 18:02Cloud trap for growing apps
- 23:37Self-hosting cut costs and improved performance
- 26:46Self-hosting AI models for privacy
- 34:35Worst day lessons
- 35:17PostgreSQL for massive scale
- 41:44YugabyteDB as PostgreSQL-compatible
- 43:54Troubleshooting the January 1st 2025 outage
- 51:45Why outages matter
- 53:13Final takeaways and future
related talks
47mScaling: from 0 to 25 million users
40mHacking the Pachyderm: Scaling Servers and People
44mIntroduction to application scaling
35mScaling RAG Applications to serve millions of users
46mManaging a Billion Transactions with Tech, Tactics, and Teamwork
23mIt Takes a Nation of Millions to Hold Us Back
35mHow to Host a (Very) Popular Website for 30 Altairian...
26mHow We Prepped Our Infrastructure for a Super-Bowl Sized Traffic