Debugging tools for LLM apps in JS
summary
Overview of debugging tools for JavaScript LLM/AI-agent applications, comparing LangGraph Studio, LangSmith, LangFuse, Arize Phoenix and Lunary. Explains each tool’s debugging approach, limitations and strengths, how they affect development workflow and error-finding/behavior analysis, and whether tools work framework-agnostically. Aims to help choose the right tool for a project to save time and effort.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:23What is LLM app
- 4:16LLM debugging problems
- 6:47Sentry for LLM observability
- 9:57Debugging with LangGraph Studio
- 12:49Handling states in LangGraph
- 14:36Debugging prompts in LangGraph Studio
- 17:26LangChain testing datasets
- 20:40LangGraph Studio vs Sentry
- 22:25Tool selection for team
- 24:53LangFuse and Phoenix alternatives
- 27:00Neural network nuances
- 30:12Optimizing neural calls
- 31:38Cosine similarity and drift
- 34:07Local LLM with GPT2gi
- 35:33Why LangFuse won
- 37:18Q&A
related talks
27mBuilding Useful Apps With Small, Local LLMs
44mAccelerating Software Development: The LLM (R)evolution
01h 03mAgents Everywhere
30mBuilding Robust, Secure LLM and Agentic AI Workflows
38mMaking LLMs reliable - A practical framework
28mGetting More Out of LLMs for Ruby Developers Justin Wienckowski, Jake Gribschaw We're all hearing about LLM tools that are changing on a daily, if not hourly, basis- 44mJoint Keynote Address: Accelerating Software Development: The LLM (R)evolution
17mLLM Observability, Evaluation, Experimentation Platform