Navigating Software Quality - From Chaos to Control
summary
Maps software quality to the Canavian spectrum and argues AI systems live between complex and chaotic domains, so testing uses context recall and precision for RAG retrievers, MR and precision@K for ranking, plus end-to-end relevance, groundedness and toxicity checks. Pipeline runs pre-commit, pre-release, continuous evals with human vibe checking and observability (Langfuse, LangSmith, Helicone, DataDog) feeding a live data flywheel. Recommends LLM-as-judge for automation (effective but costly), plus isolation and end-to-end tests and tracing across model chains.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 1:26Software quality evolution
- 3:11From CI/CD to DevOps
- 4:34Testing in production era
- 5:56AI-powered future hypothesis
- 6:58Introducing Cynefin framework
- 8:25Clear and complicated domains
- 12:33Complex and chaotic domains
- 15:44Traditional testing mindset
- 16:44Vector theory of change
- 18:28AI system characteristics
- 25:16Evaluation strategies for AI
- 26:40RAG system metrics
- 29:49Black box challenges in AI
- 35:08Isolation vs end-to-end testing
- 38:43Automated evals in pipeline
- 40:55Human evaluation importance
- 41:16Observability in AI systems
- 44:25Data flywheel concept
- 46:11Full-stack testing approach
- 48:39Coexistence of testing methods
- 50:02Product team responsibilities
related talks
41mWhen Software Starts Thinking: The Now of Quality Engineering
24mThe QA spectrum
50mThe End of Testing as We Know It
31mScaling Quality in a Decision Intelligence Platform: The Agentic QA Playbook
41mAutonomous Quality Engineering: AI Systems That Generate Execute Heal Learn and Govern Quality
28mWhy Software Quality Is the Key to Scaling AI Development
56mControlling Quality in Software Development
01h 02mThe Pursuit of Quality