Oleksandr Polozov, Tal Schuster, Adam Kalai
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We introduce a new type of programming challenge called programming puzzles, as an objective and comprehensive evaluation of program synthesis, and release an open-source dataset of Python Programming Puzzles (P3). 1 Each puzzle is defined by a short Python program f , and the goal is to find an input which makes f return True. The puzzles are objective in that each one is specified entirely by the source code of its verifier f , so evaluating f is all that is needed to test a candidate solution. They do not require an answer key or input/output examples, nor do they depend on natural language understanding. The dataset is comprehensive in that it spans problems of a range of difficulties and domains, ranging from trivial string manipulation problems, to classic programming puzzles (e.g., Tower of Hanoi), to interview/competitive-programming problems (e.g., dynamic programming), to longstanding open problems in algorithms and mathematics (e.g., factoring). We develop baseline enumerative program synthesis, GPT-3 and Codex solvers that are capable of solving puzzles-even without access to any reference solutions-by learning from their own past solutions. Codex performs best, solving up to 18% of 397 test problems with a single try and 80% of the problems with 1,000 tries per problem. In a small user study, we find a positive correlation between puzzlesolving performance and coding experience, and between the puzzle difficulty for humans and AI solvers. Therefore, further improvements on P3 could have a significant impact on many program synthesis areas.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2106.05784")
get_code_for_paper("2106.05784")
have("2106.05784")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2106.05784.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2106.05784)
Connect an agent — have() is free.