Armando Solar-Lezama, Jianfeng Gao, Jeevana Priya Inala, Chenglong Wang, Theo X. Olausson
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large language models have shown remarkable aptitude in code generation, but still struggle to perform complex tasks. Self-repair-in which the model debugs and repairs its own code-has recently become a popular way to boost performance in these settings. However, despite its increasing popularity, existing studies of self-repair have been limited in scope; in many settings, its efficacy thus remains poorly understood. In this paper, we analyze Code Llama, GPT-3.5 and GPT-4's ability to perform self-repair on problems taken from HumanEval and APPS. We find that when the cost of carrying out repair is taken into account, performance gains are often modest, vary a lot between subsets of the data, and are sometimes not present at all. We hypothesize that this is because self-repair is bottlenecked by the model's ability to provide feedback on its own code; using a stronger model to artificially boost the quality of the feedback, we observe substantially larger performance gains. Similarly, a small-scale study in which we provide GPT-4 with feedback from human participants suggests that even for the strongest models, self-repair still lags far behind what can be achieved with human-level debugging.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2306.09896")
get_code_for_paper("2306.09896")
have("2306.09896")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2306.09896.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2306.09896)
Connect an agent — have() is free.