Model Evaluation for Custom Datasets With Open Models: Multi-Model Comparison With...
Talkat Open Source Summit Japan + AI_dev: Open Source GenAI & ML Summit + Automotive Linux Summit 2025 · 2025-12-08..12-10·
summary
A Streamlit-based UI compares three open embedding models (all-MiniLM L6, BGE small m1.5, Ruri/B3M) on MTB tasks and a custom Japanese corpus using retrieval metrics like nDCG@10. Code and data are on GitHub and run locally (large models need >36 GB RAM so demo uses small variants). Results: BGE-small tops MTB but Ruri/B3M beats others on the Japanese dataset, showing language-specialized models can reverse leaderboard order; synthetic queries are partly generated with LLMs and require manual quality checks.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:01Embedding models and uses
- 2:28Embedding models history
- 6:01Open vs enterprise tradeoffs
- 7:47Evaluation methodologies
- 10:56Custom benchmarking setup
- 13:30Demo and model selection
- 18:20Running evaluation
- 26:01Open model challenges
- 27:46Key takeaways
related talks
19mHow to look at your data
28mFine-Tuning Small Language Models with Python
32mReproducibility in Embedding Benchmarks
27mDo you know how well your model is doing? Evaluate your LLMs
33mBuild Better AI with First-Party Data and Fine-Tuning
33mThe Right Model for the Right Job: Cutting LLM Costs Without Cutting Quality
27mModel Evaluation for Humans
25mState of Open Source AI Coding Models