Building Real-Time AI Decision Systems with LLMs
summary
Practical blueprint for running LLMs as real‑time decision engines: ingest events with Kafka, Kinesis, Flink, feed a RAG LLM reasoning layer, evaluate outputs in a rules‑based decision engine, and log everything to an observability and tracing layer. Calls out four bottlenecks — token throughput, context window, cold starts, stateful sessions — and prescribes decoupling inference, aggressive caching, async paths, circuit breakers, and semantic caching with hybrid model routing to avoid the sub-100 ms SLA impossible trap.
AI-written summary · about
chapters (15)
- 0:00Intro and Goals
- 0:35From Experiments to Intelligence
- 2:36Then vs Now Decisions
- 4:02Core Architecture Layers
- 5:43Evolution Roadmap
- 7:25Data Problems in Orgs
- 9:32Governance and Human Loop
- 11:10Scaling LLM Infrastructure
- 14:55RAG for Operations
- 16:52Feedback and Reinforcement
- 17:36Observability and Tracing
- 19:24Fail Safes and Fallbacks
- 20:45Guardrails and Evaluation
- 21:33Cost and Latency Budgeting
- 22:35Key Takeaways and Wrap Up
related talks
32mBeyond LLMs: Building an AI-Powered Decision System at Scale
24mLLM Inference Optimization From Simple To Sophisticated
28mSymbolic Reasoning in telecom oriented LLMs
28mNavigating LLM deployment: Tips, tricks, and techniques
43mLLMs in the Wild - Streaming, RAG, and Real Time GenAI at Scale
39mSelf-Service Analytics using LLMs
36mLLMs and AI agents demystified
12mKeynote: How LLMs Have Changed the Way Organizations Do AI