SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2107.03374 · arXiv.org · 2021

Evaluating Large Language Models Trained on Code

William Guss, Jie Tang, Jan Leike, Alec Radford, Dario Amodei, Lukasz Kaiser, Elizabeth Barnes, Christopher Hesse, Igor Babuschkin, Nick Ryder, Greg Brockman, Mark Chen, and 46 more

arXiv · PDF · Open in the Atlas

Code that ran

26 of 39 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 1 honoured the contract we drafted, 0 violated it and 18 have no contract verdict; 7 failures of our instrument; 13 no recorded run), measured 2026-11-21; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2107.03374.json · badge: /api/ran/2107.03374.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
openai/code-align-evals-data canonical (classifier label)
classifier basis: census
9 ran · 3 no recorded run
codedotal/gpt-code-clippy community (classifier label)
classifier basis not recorded on the link
11 ran · 0 no recorded run
openai/human-eval marked official in Papers with Code
Papers with Code archive
2 ran · 3 no recorded run
2796gaurav/human-eval community (classifier label)
classifier basis not recorded on the link
2 ran · 0 no recorded run
my-other-github-account/llm-humaneval-benchmarks community (classifier label)
classifier basis not recorded on the link
1 ran · 6 no recorded run
microsoft/PythonProgrammingPuzzles community
Papers with Code archive
1 ran · 1 no recorded run
FunctionStatusWhere it lives
add_elements Ran
ran on a synthesized input
openai/code-align-evals-data/bad-solutions/add_elements.py
code served (permissive licence) · get_code("2cc1153eb2697bbf")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Based on the function signature and implementation:
# - arr: a list/array of integers
# - k: an integer index
# The function accesses arr[k] and iterates through it, checking len(str(elem)) <= 2

# Looking at the function body, it appears there's a bug in the provided code.
# The example shows arr = [111,21,3,4000,5,6,7,8,9], k = 4, output = 24
# But the current implementation does arr[k] which would be arr[4] = 5 (a single element)
# The intended behavior seems to be: find first k elements with at most 2 digits and sum them

# However, I need to construct valid arguments for the FUNCTION AS WRITTEN.
# The function does: sum((elem for elem in arr[k] if len(str(elem)) <= 2))
# This means arr[k] must be iterable (like a list/string)
# If arr[k] is a list of integers, it will check each integer's string representation

# Valid setup: arr is a list of lists, where each sublist contains integers
arr = [
    [111, 222, 333],      # arr[0]
    [1000, 2000],         # arr[1]
    [5, 6, 7],            # arr[2]
    [21, 3, 4000, 5],     # arr[3] - first k elements with <= 2 digits: 21, 3, 5 -> sum = 29
    [8, 9, 10]            # arr[4]
]

k = 3  # valid k value (1 <= k <= len(arr))

ARGS = {
    "arr": arr,
    "k": k
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
all_prefixes Ran
ran on a synthesized input
openai/code-align-evals-data/bad-solutions/all_prefixes.py
code served (permissive licence) · get_code("5e866b54fbf03be3")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Construct arguments for all_prefixes function
# The function takes a single string parameter and returns all prefixes

ARGS = {
    "string": "abc"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
anti_shuffle Ran
ran on a synthesized input
openai/code-align-evals-data/bad-solutions/anti_shuffle.py
code served (permissive licence) · get_code("1a25d7256e5f4bb5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Analyze the function: it takes a string s, splits by spaces,
# sorts characters in each word, and joins back with spaces

torch.manual_seed(0)

# The function expects a string parameter
ARGS = {
    's': 'Hello World!!!'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
any_int Ran
ran on a synthesized input
openai/code-align-evals-data/bad-solutions/any_int.py
code served (permissive licence) · get_code("c3efafd6833cfa4b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

# Analyzing the function, it takes 3 numeric parameters x, y, z
# The function checks if all are integers and if sum of any two equals the third

# Test cases to construct valid arguments
ARGS = {
    "x": 5,
    "y": 2,
    "z": 7
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
below_threshold Ran
ran on a synthesized input
openai/code-align-evals-data/bad-solutions/below_threshold.py
code served (permissive licence) · get_code("db66d9a5e3e1895b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Analyze the function:
# - below_threshold takes a list `l` and an int `t`
# - It iterates through elements in `l` and checks if any are >= t
# - Returns True if all elements are < t (based on the docstring examples)

# Create test arguments
ARGS = {
    "l": [1, 2, 4, 10],
    "t": 100
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
below_zero Ran
ran on a synthesized input
openai/code-align-evals-data/bad-solutions/below_zero.py
code served (permissive licence) · get_code("92fc2114d3e87e85")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

from typing import List

# Test case 1: operations that keep balance non-negative
operations_valid = [1, 2, 3]

# Test case 2: operations that cause balance to go below zero
operations_invalid = [1, 2, -4, 5]

# Use the case where balance goes below zero
ARGS = {
    "operations": operations_invalid
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
bf Ran
ran on a synthesized input
openai/code-align-evals-data/bad-solutions/bf.py
code served (permissive licence) · get_code("2ef1476740b6e3dd")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

ARGS = {
    "planet1": "Jupiter",
    "planet2": "Neptune"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
by_length Ran
ran on a synthesized input
openai/code-align-evals-data/bad-solutions/by_length.py
code served (permissive licence) · get_code("f63bbaac0df6c6fe")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Construct valid arguments for by_length function
arr = [2, 1, 1, 4, 5, 8, 2, 3]

ARGS = {
    "arr": arr
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family int — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
car_race_collision Ran
ran on a synthesized input
openai/code-align-evals-data/bad-solutions/car_race_collision.py
code served (permissive licence) · get_code("968da8e551331977")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# The function car_race_collision takes a single parameter n (an integer)
# and returns n ^ 2 (XOR operation on n with itself, which is always 0,
# but more importantly it demonstrates the collision count calculation)

ARGS = {
    "n": 5
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
chdir Ran
ran; our contract draft was wrong, not the code
openai/human-eval/human_eval/execution.py
code served (permissive licence) · get_code("4ed7a6255ffcd3e2")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned _GeneratorContextManager. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import contextlib
import os
import tempfile

@contextlib.contextmanager
def chdir(root):
    if root == '.':
        yield
        return
    cwd = os.getcwd()
    os.chdir(root)
    try:
        yield
    except BaseException as exc:
        raise exc
    finally:
        os.chdir(cwd)

# Create a temporary directory to use as the root parameter
temp_dir = tempfile.mkdtemp()

ARGS = {
    'root': temp_dir
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type _GeneratorContextManager — a _GeneratorContextManager output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a _GeneratorContextManager output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
create_tempdir Ran
ran on a synthesized input
openai/human-eval/human_eval/execution.py
code served (permissive licence) · get_code("1214e44aeb74f364")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned _GeneratorContextManager. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import contextlib
import os
import tempfile

@contextlib.contextmanager
def chdir(root):
    if root == '.':
        yield
        return
    cwd = os.getcwd()
    os.chdir(root)
    try:
        yield
    except BaseException as exc:
        raise exc
    finally:
        os.chdir(cwd)

@contextlib.contextmanager
def create_tempdir():
    with tempfile.TemporaryDirectory() as dirname:
        with chdir(dirname):
            yield dirname

# create_tempdir() takes no parameters
ARGS = {}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type _GeneratorContextManager — a _GeneratorContextManager output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a _GeneratorContextManager output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
estimate_pass_at_k Ran
ran; our contract draft was wrong, not the code
2796gaurav/human-eval/human_eval/evaluation.py
pointer only (licence: MIT); the stored copy get_code serves, from rucaibox/bamboo, is pointer only (licence: NONE) · get_code("e080db2eaa81fdb3")
How it ran

The body of this sample is withheld: MIT, as GitHub detects it on this paper's repository, 2796gaurav/human-eval, is in the permissive set, but a licence decision keeps the body a pointer: the reason for that decision is not recorded with it. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

find_indentation Ran
ran; our contract draft was wrong, not the code
codedotal/gpt-code-clippy/reindent.py
pointer only (licence: Apache-2.0); the stored copy get_code serves, from xwin-lm/xwin-lm, is pointer only (licence: NONE) · get_code("f05983c38acfa250")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (Apache-2.0), but get_code serves the stored copy, first harvested from xwin-lm/xwin-lm, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

generate_prompt Ran
ran on a synthesized input
codedotal/gpt-code-clippy/evaluation/apps_utils/generate_gpt_codes.py
code served (permissive licence) · get_code("5bcc0e96c2ae8311")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
import os

# Create temporary files for the function to read
temp_dir = tempfile.mkdtemp()

# Create a prompt file
prompt_path = os.path.join(temp_dir, "prompt.txt")
with open(prompt_path, 'w') as f:
    f.write("Write a function that adds two numbers.")

# Create a test case file with fn_name
test_case_path = os.path.join(temp_dir, "test_cases.json")
with open(test_case_path, 'w') as f:
    json.dump({"fn_name": "add", "test_cases": [{"input": [1, 2], "output": 3}]}, f)

# Create a starter code file
starter_path = os.path.join(temp_dir, "starter.py")
with open(starter_path, 'w') as f:
    f.write("def add(a, b):\n    pass")

# Create a minimal tokenizer stand-in
class MinimalTokenizer:
    def __init__(self):
        pass

tokenizer = MinimalTokenizer()

# Define ARGS
ARGS = {
    "test_case_path": test_case_path,
    "prompt_path": prompt_path,
    "solutions_path": os.path.join(temp_dir, "solutions.py"),
    "tokenizer": tokenizer,
    "starter_path": starter_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
generate_prompt Ran
ran on a synthesized input
codedotal/gpt-code-clippy/data_processing/apps.py
code served (permissive licence) · get_code("73c40ea3df62ea10")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
import os

# Create temporary directory and files for the function to read
temp_dir = tempfile.mkdtemp()

# Create test case file
test_case_path = os.path.join(temp_dir, "test_case.json")
with open(test_case_path, 'w') as f:
    json.dump({"fn_name": "solution"}, f)

# Create prompt file
prompt_path = os.path.join(temp_dir, "prompt.txt")
with open(prompt_path, 'w') as f:
    f.write("Write a function that adds two numbers.\n")

# Create solutions file (not used by function but listed as parameter)
solutions_path = os.path.join(temp_dir, "solutions.txt")
with open(solutions_path, 'w') as f:
    f.write("def add(a, b):\n    return a + b\n")

# Create optional starter file
starter_path = os.path.join(temp_dir, "starter.txt")
with open(starter_path, 'w') as f:
    f.write("def solution(a, b):\n    pass\n")

# Create a minimal tokenizer stand-in
class MinimalTokenizer:
    def __init__(self):
        pass

tokenizer = MinimalTokenizer()

ARGS = {
    "test_case_path": test_case_path,
    "prompt_path": prompt_path,
    "solutions_path": solutions_path,
    "tokenizer": tokenizer,
    "starter_path": starter_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_question Ran
ran; our contract draft was wrong, not the code
codedotal/gpt-code-clippy/evaluation/apps_eval_util.py
code served (permissive licence) · get_code("ed8331339b4de97c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile
import shutil

# Create temporary directories with question.txt files
temp_dir1 = tempfile.mkdtemp()
temp_dir2 = tempfile.mkdtemp()

# Write a question file to the first directory
with open(os.path.join(temp_dir1, 'question.txt'), 'w') as f:
    f.write("What is the capital of France?\n")
    f.write("Is it Paris?\n")

# Second directory has no question.txt (will test the else branch)
# (no file written)

problem_list = [temp_dir1, temp_dir2]

ARGS = {
    'problem_list': problem_list,
    'prob_index': 0
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_solutions Ran
ran; our contract draft was wrong, not the code
codedotal/gpt-code-clippy/evaluation/apps_eval_util.py
code served (permissive licence) · get_code("0c83bb0f43b8dc57")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import os
import tempfile
import shutil

# Create a temporary directory structure with a solutions.json file
temp_dir = tempfile.mkdtemp()
problem_dir = os.path.join(temp_dir, "problem_0")
os.makedirs(problem_dir, exist_ok=True)

# Create a solutions.json file
solutions_data = {
    "solution_1": {"code": "print('hello')", "score": 0.95},
    "solution_2": {"code": "print('world')", "score": 0.87}
}

solutions_file = os.path.join(problem_dir, "solutions.json")
with open(solutions_file, 'w') as f:
    json.dump(solutions_data, f)

# Create problem_list with the temporary directory
problem_list = [problem_dir]
prob_index = 0

ARGS = {
    "problem_list": problem_list,
    "prob_index": prob_index
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_valid_problems Ran
ran; our contract draft was wrong, not the code
codedotal/gpt-code-clippy/evaluation/apps_eval_util.py
code served (permissive licence) · get_code("8bbb265e59e23293")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import json
import tempfile
import shutil

# Create a temporary directory structure for testing
temp_dir = tempfile.mkdtemp()

# Create subdirectories and files to match what the function expects
data_subdir = os.path.join(temp_dir, 'data')
os.makedirs(data_subdir, exist_ok=True)

# Create a few problem folders with the expected files
problem_1 = os.path.join(data_subdir, 'problem_1')
os.makedirs(problem_1, exist_ok=True)
with open(os.path.join(problem_1, 'input_output.json'), 'w') as f:
    json.dump({"test": "data"}, f)

problem_2 = os.path.join(data_subdir, 'problem_2')
os.makedirs(problem_2, exist_ok=True)
with open(os.path.join(problem_2, 'sols.json'), 'w') as f:
    json.dump({"solution": "code"}, f)

# Create a problem folder without valid files (should be skipped)
problem_3 = os.path.join(data_subdir, 'problem_3')
os.makedirs(problem_3, exist_ok=True)
with open(os.path.join(problem_3, 'other_file.txt'), 'w') as f:
    f.write("some content")

ARGS = {
    'data_dir': temp_dir
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
is_digit Ran
ran on a synthesized input
codedotal/gpt-code-clippy/data_processing/download_license_info.py
code served (permissive licence) · get_code("257503e1cf9966ff")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

ARGS = {
    'x': '5'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
pass_at_k Ran
ran, honoured the contract we drafted
microsoft/PythonProgrammingPuzzles/solvers/codex/run_codex_experiments.py
code served (permissive licence) · get_code("2a310ceef0a63386")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Analyzing pass_at_k function:
# - k: int - number of iterations in loop
# - successes: int - number of successful attempts
# - attempts: int - total number of attempts
# The function calculates probability of success when sampling k times
# All parameters are simple integers

ARGS = {
    "k": 3,
    "successes": 7,
    "attempts": 10
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
read_problems Ran
ran on a synthesized input
2796gaurav/human-eval/human_eval/data.py
pointer only (licence: MIT) · get_code("d3d6eed5a7c21746")
How it ran

The body of this sample is withheld: MIT, as GitHub detects it on this paper's repository, 2796gaurav/human-eval, is in the permissive set, but a licence decision keeps the body a pointer: the reason for that decision is not recorded with it. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

reindent_code Ran
ran on a synthesized input
codedotal/gpt-code-clippy/data_processing/apps.py
code served (permissive licence) · get_code("b6404d68982b4e0b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import io

# Analyze the function: reindent_code takes a single parameter 'codestr' which is a string
# The function reads from it line by line, so we need a valid Python code string

ARGS = {
    'codestr': '''def hello_world():
    print("Hello")
    if True:
        x = 1
        y = 2
    return x'''
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
replace_inline_tabs Ran
ran; our contract draft was wrong, not the code
codedotal/gpt-code-clippy/reindent.py
pointer only (licence: Apache-2.0); the stored copy get_code serves, from xwin-lm/xwin-lm, is pointer only (licence: NONE) · get_code("27f23c533d62061a")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (Apache-2.0), but get_code serves the stored copy, first harvested from xwin-lm/xwin-lm, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

run_async Ran
ran on a synthesized input
my-other-github-account/llm-humaneval-benchmarks/benchmark_utils.py
code served (permissive licence) · get_code("f7666066f684b48f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned coroutine. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import asyncio
import json

# Define ARGS dict with all parameters for run_async
ARGS = {
    'prompt': 'Hello, how are you?',
    'seed': 42,
    'port': 8000,
    'deterministic': True,
    'host': 'localhost'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
websockets — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type coroutine — a coroutine output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a coroutine output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
split_into_chunks Ran
ran on a synthesized input
codedotal/gpt-code-clippy/data_processing/download_license_info.py
code served (permissive licence) · get_code("d27e3a9c64e09294")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

def split_into_chunks(l, n):
    n = max(1, n)
    return [l[i:i + n] for i in range(0, len(l), n)]

# Setup
torch.manual_seed(0)

# Create a simple list to split
test_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

# Create chunk size
chunk_size = 3

ARGS = {
    "l": test_list,
    "n": chunk_size
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list — a list output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
timeout Ran
ran on a synthesized input
codedotal/gpt-code-clippy/data_processing/download_license_info.py
code served (permissive licence) · get_code("2e22b41539e9cf87")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Define a simple function to pass as the first argument
def sample_function(x, y):
    return x + y

# Define ARGS dict with all parameters of the timeout function
ARGS = {
    "func": sample_function,
    "args": (5, 3),
    "kwargs": {},
    "timeout_duration": 10,
    "default": None
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
add Not yet run
unverified (harvested, no recorded run)
openai/code-align-evals-data/bad-solutions/add.py
code served (permissive licence) · get_code("9f6bcf31e58add54")
How it ran

no invocation recorded.

can_arange Not yet run
unverified (harvested, no recorded run)
openai/code-align-evals-data/bad-solutions/can_arange.py
code served (permissive licence) · get_code("d3bd061f24d302d4")
How it ran

no invocation recorded.

change_base Not yet run
unverified (harvested, no recorded run)
openai/code-align-evals-data/bad-solutions/change_base.py
code served (permissive licence) · get_code("63c649d1e83343f6")
How it ran

no invocation recorded.

check_correctness Not yet run
unverified (harvested, no recorded run)
openai/human-eval/human_eval/execution.py
code served (permissive licence) · get_code("87135198de7ec14b")
How it ran

no invocation recorded.

get_output_and_retry Not yet run
unverified (harvested, no recorded run)
my-other-github-account/llm-humaneval-benchmarks/launch_pod.py
code served (permissive licence) · get_code("32636ddfa64b46bc")
How it ran

no invocation recorded.

launch_pod Not yet run
unverified (harvested, no recorded run)
my-other-github-account/llm-humaneval-benchmarks/launch_pod.py
code served (permissive licence) · get_code("8339ab6beb446371")
How it ran

no invocation recorded.

reliability_guard Not yet run
unverified (harvested, no recorded run)
openai/human-eval/human_eval/execution.py
code served (permissive licence) · get_code("53e4784adcaf6602")
How it ran

no invocation recorded.

run Not yet run
unverified (harvested, no recorded run)
microsoft/PythonProgrammingPuzzles/solvers/codex/run_codex_experiments.py
code served (permissive licence) · get_code("aa03c033d83047b0")
How it ran

no invocation recorded.

run Not yet run
unverified (harvested, no recorded run)
my-other-github-account/llm-humaneval-benchmarks/benchmark_utils.py
code served (permissive licence) · get_code("de45bd615801ae81")
How it ran

no invocation recorded.

run_sync Not yet run
unverified (harvested, no recorded run)
my-other-github-account/llm-humaneval-benchmarks/benchmark_utils.py
code served (permissive licence) · get_code("4d5e8113a82b291c")
How it ran

no invocation recorded.

start_server Not yet run
unverified (harvested, no recorded run)
my-other-github-account/llm-humaneval-benchmarks/benchmark_manager.py
code served (permissive licence) · get_code("5263f985bd2e84c0")
How it ran

no invocation recorded.

unsafe_execute Not yet run
unverified (harvested, no recorded run)
openai/human-eval/human_eval/execution.py
code served (permissive licence) · get_code("3c91bdab594f03c5")
How it ran

no invocation recorded.

wait_for_pod Not yet run
unverified (harvested, no recorded run)
my-other-github-account/llm-humaneval-benchmarks/launch_pod.py
code served (permissive licence) · get_code("8d76e5a80e6c961c")
How it ran

no invocation recorded.

Also present in other papers' code

Of 39 samples lifted from this paper's linked repositories, 39 carry a function identity and 0 do not. Of the 38 distinct functions with an identity, 10 are also present in other papers whose code we hold (5 with at least one paper dated earlier, 0 the same month, 10 later) and 28 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
estimate_pass_at_k 20 other papers
20 later
2207.14255 (2022-07)
Efficient Training of Language Models to Fill in the Middle
2210.14868 (2022-10)
Multi-lingual Evaluation of Code Generation Models
2306.03438 (2023-06)
Large Language Models of Code Fail at Completing Code with Potential Bugs
and 17 more not named here
get_solutions 4 other papers
1 dated earlier, 3 later
2105.09938 (2021-05)
Measuring Coding Challenge Competence With APPS
2203.08597 (2022-03)
Less is More: Summary of Long Instructions is Better for Program Synthesis
2403.04706 (2024-03)
Common 7B Language Models Already Possess Strong Math Capabilities
and 1 more not named here
get_valid_problems 4 other papers
1 dated earlier, 3 later
2105.09938 (2021-05)
Measuring Coding Challenge Competence With APPS
2203.08597 (2022-03)
Less is More: Summary of Long Instructions is Better for Program Synthesis
2403.04706 (2024-03)
Common 7B Language Models Already Possess Strong Math Capabilities
and 1 more not named here
get_question 3 other papers
1 dated earlier, 2 later
2105.09938 (2021-05)
Measuring Coding Challenge Competence With APPS
2403.04706 (2024-03)
Common 7B Language Models Already Possess Strong Math Capabilities
2602.01685 (2026-02)
Semantic-aware Wasserstein Policy Regularization for Large Language Model Alignment
check_correctness
brick of this paper's assembly: entry
2 other papers
2 later
2402.14852 (2024-02)
HumanEval on Latest GPT Models -- 2024
2606.05859 (2026-06)
TARPO: Token-Wise Latent-Explicit Reasoning via Action-Routing Policy Optimization
find_indentation 2 other papers
1 dated earlier, 1 later
2105.09938 (2021-05)
Measuring Coding Challenge Competence With APPS
2403.04706 (2024-03)
Common 7B Language Models Already Possess Strong Math Capabilities
replace_inline_tabs 2 other papers
1 dated earlier, 1 later
2105.09938 (2021-05)
Measuring Coding Challenge Competence With APPS
2403.04706 (2024-03)
Common 7B Language Models Already Possess Strong Math Capabilities
chdir
brick of this paper's assembly: helper
1 other paper
1 later
2604.12268 (2026-04)
CODESPECBENCH: Benchmarking LLMs for Executable Behavioral Specification Generation
create_tempdir
brick of this paper's assembly: helper
1 other paper
1 later
2604.12268 (2026-04)
CODESPECBENCH: Benchmarking LLMs for Executable Behavioral Specification Generation
reliability_guard
brick of this paper's assembly: helper
1 other paper
1 later
2604.12268 (2026-04)
CODESPECBENCH: Benchmarking LLMs for Executable Behavioral Specification Generation

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-21.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

We introduce Codex, a GPT language model finetuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2107.03374")
get_code_for_paper("2107.03374")
have("2107.03374")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2107.03374.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2107.03374.svg)](https://syntology.ai/paper/2107.03374)

Connect an agent — have() is free.