PyDex: Repairing Bugs in Introductory Python Assignments using LLMs
summary
PyDex is an APR pipeline that uses Codex with iterative prompts, parser diagnostics, test-case filtering, and peer-program few-shots to repair student Python submissions. It splits programs into localized chunks, applies a separate iterative syntax phase then a semantics phase, and selects candidates by test-pass and minimal token edit distance. Evaluated on 286 real student programs it repairs about 86% of programs and produces smaller patches on average compared to Refactory and other baselines.
AI-written summary · about
chapters (14)
- 0:00PyDex introduction
- 1:25Automated feedback motivation
- 2:50Problem definition example
- 4:57Minimal edits goal
- 6:00Two-phase repair overview
- 7:27Syntax repair chunking
- 9:57Semantic repair with tests
- 12:05Peer programs as examples
- 13:12Evaluation setup
- 14:17Comparison with systems
- 15:39Key results findings
- 16:42Takeaways for LLMs
- 17:25Education opportunities
- 17:46Q&A
related talks
20mLeroy: Library Learning for Imperative Programming Languages
01h 07mSufficiently Advanced Testing
02h 52mLow Level Technicals of LLMs
19mMAGPY: Compiling Eager Mode DNN Programs by Monitoring Execution States
19mTowards Mining Robust Coq Proof Patterns
21mPersonalization of Programming Education: An NLP-based Bi-dimensional(…)
13mScheduled Partial-Credit RL for Reliable Code Generation with Small Language Models (WIP)
40mJudgement Day: Benchmarking "Black Box" LLMs With Open Legal Datasets