We lifted 11 functions out of this paper's own repositories and ran 9 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| tddpytester/pytester | canonical | 9 of 11 |
| Function | Status | Where it lives |
|---|---|---|
| check_test_case_syntax | Ran | tddpytester/pytester/handlers/testing_util.py pointer only (licence: NONE) · get_code("d4025094cd1b7d1b") |
| load_pickle | Ran | tddpytester/pytester/handlers/utils.py pointer only (licence: NONE) · get_code("70db7c9cbe9e680d") |
| post | Ran | tddpytester/pytester/baselines/Copilot/query_apps.py pointer only (licence: NONE) · get_code("c479a69c9318680d") |
| read_json_file | Ran | tddpytester/pytester/handlers/utils.py pointer only (licence: NONE) · get_code("bfa04064fa4b10d8") |
| split_test_cases | Ran | tddpytester/pytester/handlers/testing_util.py pointer only (licence: NONE) · get_code("fa42b92f92918cb1") |
| split_test_cases | Ran | tddpytester/pytester/handlers/testing_util_v2.py pointer only (licence: NONE) · get_code("48927d42c199464b") |
| test_function | Ran | tddpytester/pytester/handlers/testing_util.py pointer only (licence: NONE) · get_code("cd5ea938ecaf9b22") |
| transform_to_code_and_test | Ran | tddpytester/pytester/handlers/testing_util_v2.py pointer only (licence: NONE) · get_code("23e99f1aaaf91949") |
| transform_to_input | Ran | tddpytester/pytester/handlers/testing_util_v2.py pointer only (licence: NONE) · get_code("76a2b0b9d12a137d") |
| DecodeIds | Not yet run | tddpytester/pytester/handlers/code_processing.py pointer only (licence: NONE) · get_code("e47c17a9039c79db") |
| copilot | Not yet run | tddpytester/pytester/baselines/Copilot/api.py pointer only (licence: NONE) · get_code("0cba2737fdabc294") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Test-driven development (TDD) is a widely-employed software development practice that mandates writing test cases based on requirements before writing the actual code. While writing test cases is the centerpiece of TDD, it is time-consuming, expensive, and often shunned by developers. To address these issues associated with TDD, automated test case generation approaches have recently been investigated. Such approaches take source code as input, but not the requirements. Therefore, existing work does not fully support true TDD, as actual code is required to generate test cases. In addition, current deep learning-based test case generation approaches are trained with one learning objective, i.e., to generate test cases that are exactly matched with the ground-truth test cases. However, such approaches may limit the model's ability to generate different yet correct test cases. In this paper, we introduce PyTester, a Text-to-Testcase generation approach that can automatically generate syntactically correct, executable, complete, and effective test cases while being aligned with a given natural language requirement. We evaluate PyTester on the public APPS benchmark dataset, and the results show that our Deep RL approach enables PyTester, a small language model, to outperform much larger language models like GPT3.5, StarCoder, and InCoder. Our findings suggest that future research could consider improving small over large LMs for better resource efficiency by integrating the SE domain knowledge into the design of reinforcement learning architecture.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2401.07576")
get_code_for_paper("2401.07576")
have("2401.07576")
Connect an agent — have() is free.