Program Repair and Hint Generation

Programming feedback has two different jobs. A repair system must produce code that is demonstrably correct; a teaching system must help a learner discover that correction. This project studies both jobs on the same introductory Python exercises, then asks how model scale, candidate sampling, LoRA rank, and joint training change the result.

The Learning Problem

The input is a buggy program, its exercise context, and a test suite. The model produces either an executable patch or a natural-language hint. Those outputs share the same diagnosis, but they are judged differently: one by tests and the other by pedagogical restraint.

Evaluation Setup

INTROPYNUS sample

The experiments use five introductory Python tasks with five buggy submissions per task. Each record includes the exercise context, a failing implementation, a reference solution, and executable tests.

What counts as a good answer

Repair quality is binary and executable. Hint quality is multi-dimensional: a useful hint can be technically correct and still fail if it gives away too much or assumes knowledge the learner does not yet have.

What Changed Performance

Foundation-model baseline

Prompting alone produces a large model-size gap. GPT-4o-mini repairs 92% of the evaluated programs, while Phi-3-mini reaches 36%. The smaller model therefore becomes the useful test case: can inference-time search or parameter-efficient training close that gap?

Candidate sampling

Sampling several possible repairs gives the smaller model more chances to escape an incorrect first approach. Most of the improvement arrives by five candidates; beyond ten, each additional sample buys progressively less.

Parameter-efficient fine-tuning

LoRA produces a much larger shift than sampling. Increasing rank improves repair accuracy, but the parameter and memory costs grow rapidly. The medium configuration, $$(r=16, \alpha=32)$$, is the operating point used for the remaining experiments: it stays within eight points of the highest-rank result at one quarter of its trainable parameters.

Joint repair and hint learning

Training the two tasks together improves both outcomes. Shared representations help the model connect a diagnosis to an executable fix and then translate the same diagnosis into restrained guidance.

A Repair in Context

A factorial implementation exposes the distinction between patching and teaching. The recursive call repeats the same argument, so execution never approaches the base case. The repair changes one expression; the hint names the invariant the learner should inspect.

1def factorial(n):
2    if n == 0:
3        return 1
4    return n * factorial(n)

1def factorial(n):
2    if n == 0:
3        return 1
4    return n * factorial(n - 1)

Generated hint “Think about what happens to the parameter in each recursive call. What should change to eventually reach the base case?”

$ python student_solution.py
RecursionError: maximum recursion depth exceeded
$ repair-hint --task factorial --mode repair-and-hint
Inspecting the recursive call and its base-case path…
Patch: recursive argument n → n - 1
Hint: What should change in each call to reach the base case?
$ python repaired_solution.py
5 / 5 tests passed

How Hint Quality Is Scored

A hint is not treated as a single scalar until the end of evaluation. Keeping the dimensions separate makes failure modes visible: a fluent answer may be wrong, and a correct answer may disclose too much.

What the Experiments Show

Findings

Limits and next steps

The evaluation is deliberately small and Python-only. The next useful test is not another rank sweep; it is a broader study of whether the same behavior survives new languages, unseen exercise families, and real student interaction.