Improve LLMs Alignment with Complete and Robust Preference Data
summary
Claims a concrete pipeline called KURA that transforms pairwise scores with a BTL logit link, imputes missing comparisons via matrix completion (LRPR), and removes adversarial labels with RPCA (ReTRon). Builds reconstructed preference matrices from 80 prompts × ~34 responses, injects five dismissive responses, and measures policy effects: uncorrupted DPO finetune wins ~82% vs GPT‑4, corrupted data drops to 0%, reconstruction recovers to 10.84%. Experiments show augmentation improves recovery and the method yields usable cleaned preferences for downstream DPO/RLHF.
AI-written summary · about
chapters (8)
- 0:00Improving LLM alignment
- 3:38Alignment critical for AI
- 10:05Collecting preference data challenges
- 14:05Introducing KURA
- 20:20KURA data processing
- 25:24KURA experimental results
- 28:56Key takeaways and future
- 29:38Q&A
related talks
34mBeyond Scoring: Rethinking Ranking in the LLM Era
53mIterative preference learning methods for large language model post training
30mLLM-Based Recommendation Systems
46mEfficient Policy Optimization Techniques for LLMs
45mOn Spurious Associations and LLM Alignment
29mReinforcement Learning with Human Feedback
42mRethinking AI Alignment and Evaluation under Heterogeneous Preferences
31mBeyond LLMs and Maturing RAG use cases