How to Be Correct at Scale With No Ground Truth
summary
Mirakl Catalog Transformer is an LLM-based pipeline that enforces deterministic correctness for fixed-value lists and required attributes, e.g. mapping dark blue → navy so products aren’t rejected. The method uses three concrete steps: define crisp, nonoverlapping metrics, decompose tasks (locale detection, extraction, format) and run bespoke judges with refined prompts plus weekly human annotations. Hard languages get a stronger model, judges power investigations, drive code fixes and catch drift, and the system operates continuously in production.
AI-written summary · about
related talks
39mMaking AI Reliable in the SDLC: The Role of Deterministic Context at Scale
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
06mMeasuring Generative AI Solutions
01h 26mShipping AI That Works: An Evaluation Framework for PMs
31mBuilding High-Quality & Trustworthy FMware
47mValidating LLMs – A True Story
35mImproving Credibility of ML Solutions
32mEvaluating multi-turn conversations