LLM Telemetry as a First-Class Rails Concern
summary
Rails app records every LLM call with Ruby LLM and stores prompts, responses, and token usage in Tracebook so cost spikes like a one day $150 bill become traceable and auditable the Rubric LLM gem runs deterministic evals in CI using custom domain rubrics plus an optional frontier judge LLM for comparison fine tuned models outperform general models on Texas Q A the privacy filter model from OpenAI helps but still leaks responses so regex alone fails and prompt engineering remains necessary
AI-written summary · about
chapters (9)
- 0:00LLM telemetry in Rails
- 1:15Texas Q&A with LLMs
- 4:16Evaluating LLM responses
- 15:46Monitoring LLM interactions
- 19:02PII filtering for user data
- 21:14Telemetry tools in Rails
- 23:01Feedback loops for models
- 28:06Rubric LLM examples
- 31:05Next steps and resources
related talks
24mRubyLLM: Making AI Development Beautiful Again
01h 49mAI engineering with open access LLMs that lie, curse, and steal
23mObservability for LLMs: Understanding What’s Happening Under the Hood
17mLLM Observability, Evaluation, Experimentation Platform
09mRuby startup demo – Wyatt Ades
44mPractical LLM Evals: Down-to-Earth Tooling for Everyone
21mRubyLLM
38mMaking LLMs reliable - A practical framework