Unlocking Smarter AI with Inference-Time Scaling
summary
Claims that running extra compute at test time boosts accuracy without retraining and that a Python library makes those methods plug-and-play. Supports OpenAI-compatible endpoints (best with VRM), exposes a Python SDK and an Inference as a Service FastAPI endpoint, and implements algorithms like self-consistency, beam/particle-decoding and a tool-focused multi-step self-consistency. Demonstrates a +10% end-to-end improvement on TopBench agent tool-calling, shows particle filtering using a reward model (reward_hub / inference_time_scaling), and requires only changing the inference call and a budget parameter.
AI-written summary · about
chapters (18)
- 0:00Inference time scaling intro
- 1:07Why scaling works for LLMs
- 3:11Computation benefits proof
- 4:16Scaling algorithm types
- 5:39Graph-based self-consistency
- 6:21Decoding with particle filtering
- 7:06IT Hub library purpose
- 8:34IT Hub design principles
- 10:03IT Hub features and algorithms
- 11:05Python SDK self-consistency
- 12:32Particle filtering with IT Hub
- 13:17Inference as a Service
- 15:24IT Hub FastAPI setup
- 16:51Curl requests to IT Hub
- 17:35IT Hub with LangGraph
- 19:03Multi-step self-consistency
- 21:34TopBench real-world example
- 24:29Q&A on budget and voting
related talks
13mTailor-Made Inference at Scale
14mReliable Agentic AI, the Open Way
01h 08mInference Scaling: A New Frontier for AI Capability
20mFireworks AI: Go from Hype to High - Value AI
28mScaling AI Infrastructure
24mKeynote: Next-generation frameworks for Large-scale AI
15mThis or That with Takara AI
19mNow in AI - How We Got Here