SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2005.14165 · Neural Information Processing Systems · 2020

Language Models are Few-Shot Learners

Daniel Ziegler, Alec Radford, Christopher Hesse, Nick Ryder, Aditya Ramesh, Mark Chen, Gretchen Krueger, Scott Gray, Ariel Herbert-Voss, Sam Mccandlish, Prafulla Dhariwal, Ilya Sutskever, and 18 more

arXiv · PDF · Open in the Atlas

Code that ran

45 of 65 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 6 honoured the contract we drafted, 2 violated it and 28 have no contract verdict; 9 failures of our instrument; 20 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2005.14165.json · badge: /api/ran/2005.14165.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
neuralmagic/lm-evaluation-harness canonical (classifier label)
classifier basis: census
5 ran · 0 no recorded run
asahi417/relbert community (classifier label)
classifier basis not recorded on the link
10 ran · 5 no recorded run
vilm-ai/viet-llm-eval community (classifier label)
classifier basis not recorded on the link
6 ran · 1 no recorded run
juletx/lm-evaluation-harness community (classifier label)
classifier basis not recorded on the link
4 ran · 0 no recorded run
Sypherd/lm-evaluation-harness community (classifier label)
classifier basis not recorded on the link
4 ran · 0 no recorded run
volcengine/vegiantmodel community (classifier label)
classifier basis not recorded on the link
3 ran · 4 no recorded run
ramanakshay/nanogpt community (classifier label)
classifier basis not recorded on the link
2 ran · 0 no recorded run
EleutherAI/lm_evaluation_harness community (classifier label)
classifier basis not recorded on the link
2 ran · 0 no recorded run
opengptx/lm-evaluation-harness community (classifier label)
classifier basis not recorded on the link
2 ran · 0 no recorded run
abhaskumarsinha/Corpus2GPT extension (classifier label)
classifier basis: classifier_2025
1 ran · 1 no recorded run
insait-institute/lm-evaluation-harness-bg community (classifier label)
classifier basis not recorded on the link
1 ran · 1 no recorded run
VachanVY/gpt.jax community (classifier label)
classifier basis not recorded on the link
1 ran · 1 no recorded run
sambanova/lm-evaluation-harness community (classifier label)
classifier basis not recorded on the link
1 ran · 1 no recorded run
nlx-group/overlapy community (classifier label)
classifier basis not recorded on the link
1 ran · 0 no recorded run
karpathy/llm.c reimplementation (classifier label)
classifier basis: classifier_2025
1 ran · 0 no recorded run
abhaskumarsinha/MinimalGPT reimplementation (classifier label)
classifier basis: classifier_2025
1 ran · 0 no recorded run
copy not recorded community
not in the paper's declared repositories
0 ran · 4 no recorded run
mbzuai-paris/lm-evaluation-harness-atlas-chat community (classifier label)
classifier basis not recorded on the link
0 ran · 2 no recorded run
FunctionStatusWhere it lives
FLOP Ran
ran, violated the contract we drafted
abhaskumarsinha/Corpus2GPT/models/GPT.py
code served (permissive licence) · get_code("f0fc0ac7590a2d39")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Sonnet 4.5, prompt python_fixture_v1) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# The function FLOP calculates FLOPs for a transformer model
# It takes scalar integer parameters representing model dimensions

input_len = 128  # sequence length
vocab_size = 5000  # vocabulary size
embed_dim = 256  # embedding dimension
num_heads = 8  # number of attention heads
num_decoders = 6  # number of decoder layers
fc_dim_factor = 4  # feedforward dimension multiplier (typically 4x embed_dim)

ARGS = {
    'input_len': input_len,
    'vocab_size': vocab_size,
    'embed_dim': embed_dim,
    'num_heads': num_heads,
    'num_decoders': num_decoders,
    'fc_dim_factor': fc_dim_factor
}
Setup block drafted by
a language model, Claude Sonnet 4.5, prompt python_fixture_v1
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
column_parallel_load_hook Ran
ran on a synthesized input
volcengine/vegiantmodel/src/veGiantModel/module/dense.py
code served (permissive licence) · get_code("904fac6af5735d5a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned function. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

class ColumnParallelLinear(nn.Module):
    """Minimal ColumnParallelLinear module for testing."""
    def __init__(self, in_features, out_features, mp_rank, mp_size):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.mp_rank = mp_rank
        self.mp_size = mp_size
        # Each rank gets a slice of out_features
        self.weight = nn.Parameter(torch.randn(out_features, in_features))
        self.bias = nn.Parameter(torch.randn(out_features))

# Create a minimal module that uses mp_rank and out_features
module = ColumnParallelLinear(
    in_features=3,
    out_features=8,
    mp_rank=1,
    mp_size=2
)

# Create a simple logging function
def log_fn(msg):
    print(msg)

ARGS = {
    'module': module,
    'log_fn': log_fn,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type function — a function output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a function output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
column_serial_load_hook Ran
ran on a synthesized input
volcengine/vegiantmodel/src/veGiantModel/module/dense.py
code served (permissive licence) · get_code("64087979238e1d41")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned function. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

class ColumnSerialLinear(nn.Module):
    def __init__(self, in_features, out_features, model_parallel_size):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.model_parallel_size = model_parallel_size
        self.weight = nn.Parameter(torch.randn(out_features * model_parallel_size, in_features))
        self.bias = nn.Parameter(torch.randn(out_features * model_parallel_size))
    
    def forward(self, x):
        return torch.nn.functional.linear(x, self.weight, self.bias)

def mock_log_fn(msg):
    pass

module = ColumnSerialLinear(
    in_features=3,
    out_features=4,
    model_parallel_size=2
)

log_fn = mock_log_fn

ARGS = {
    "module": module,
    "log_fn": log_fn
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type function — a function output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a function output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
configure_optimizers Ran
ran on a synthesized input
ramanakshay/nanogpt/src/algorithm/optimizer.py
code served (permissive licence) · get_code("451bc0edacfc317d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned AdamW. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Create a minimal model with parameters of different dimensions
class MinimalModel(nn.Module):
    def __init__(self):
        super().__init__()
        # 2D parameter (will be in decay_params)
        self.linear = nn.Linear(4, 3)
        # 1D parameter (will be in nodecay_params)
        self.bias_only = nn.Parameter(torch.randn(2))
    
    def forward(self, x):
        return self.linear(x)

model = MinimalModel()

# Create a minimal state object with required attributes
class State:
    def __init__(self):
        self.device_type = 'cpu'
        self.is_master_process = True

state = State()

# Define all arguments
ARGS = {
    'model': model,
    'weight_decay': 0.1,
    'learning_rate': 1e-3,
    'betas': (0.9, 0.999),
    'eps': 1e-8,
    'state': state
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type AdamW — a AdamW output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a AdamW output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
cosine_similarity Ran
ran on a synthesized input
asahi417/relbert/relbert/evaluation/analogy_questions.py
code served (permissive licence) · get_code("3055a2ccdae837c7")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# The function expects two iterables (a and b) that:
# 1. Can be iterated over to get numeric values
# 2. Support map() operations
# 3. Can be zipped together
# 4. Support arithmetic operations

# Create simple list arguments (iterables of numbers)
a = [1.0, 2.0, 3.0, 4.0]
b = [0.5, 1.5, 2.5, 3.5]

ARGS = {
    "a": a,
    "b": b,
    "zero_vector_mask": -100
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
cosine_similarity Ran
ran on a synthesized input
asahi417/relbert/relbert/evaluation/relation_mapping.py
code served (permissive licence) · get_code("3e6ff0c8581ee623")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float64, ndim 0, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

torch_seed = 0

# Create two small numpy arrays for cosine similarity computation
a = np.array([1.0, 2.0, 3.0, 4.0])
b = np.array([2.0, 1.0, 4.0, 3.0])

ARGS = {
    "a": a,
    "b": b,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float64, ndim 0, dtype family float — no output shape is stated: observe_signature.py recorded float ndim 0, which disagrees with the fixture run
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
create_image_prompt Ran
ran on a synthesized input
neuralmagic/lm-evaluation-harness/lm_eval/models/api_models.py
code served (permissive licence) · get_code("ffcac685bcb61388")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

from PIL import Image
import numpy as np

# Create a small test image
img_array = np.random.randint(0, 256, (10, 10, 3), dtype=np.uint8)
test_image = Image.fromarray(img_array)

# Create another test image with different dimensions
img_array_2 = np.random.randint(0, 256, (15, 12, 3), dtype=np.uint8)
test_image_2 = Image.fromarray(img_array_2)

# Create a chat dict with the expected structure
chat = [
    {"role": "user", "content": "What is in this image?", "type": "text"}
]

ARGS = {
    "imgs": [test_image, test_image_2],
    "chat": chat,
    "fmt": "PNG"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
PIL, numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
create_image_prompt Ran
ran on a synthesized input
EleutherAI/lm_evaluation_harness/lm_eval/models/api_models.py
code served (permissive licence) · get_code("a20ac4af0aec2beb")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

from PIL import Image
import numpy as np

# Create small PIL images for testing
img1 = Image.new('RGB', (2, 2), color='red')
img2 = Image.new('RGB', (2, 2), color='blue')

# Create a chat dict with the expected structure
chat = [
    {
        'role': 'user',
        'content': 'What is in this image?',
        'type': 'text'
    }
]

# Define arguments
ARGS = {
    'imgs': [img1, img2],
    'chat': chat,
    'fmt': 'PNG'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
PIL, numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
create_image_prompt Ran
ran on a synthesized input
juletx/lm-evaluation-harness/lm_eval/models/api_models.py
code served (permissive licence) · get_code("ecbbe3133a0f68b3")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

from PIL import Image
import numpy as np

# Create small test images
img1 = Image.new('RGB', (4, 3), color='red')
img2 = Image.new('RGB', (4, 3), color='blue')
imgs = [img1, img2]

# Create a chat structure with list content
chat = [
    {
        'role': 'user',
        'type': 'text',
        'content': [
            {'type': 'text', 'text': 'What is in this image?'}
        ]
    }
]

fmt = 'PNG'

ARGS = {
    'imgs': imgs,
    'chat': chat,
    'fmt': fmt
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
PIL, numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
custom_prompter Ran
ran on a synthesized input
asahi417/relbert/relbert/lm.py
code served (permissive licence) · get_code("5307f83d52fe15d1")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Analyze the function:
# - word_pair: must be a sequence of length 2 with two strings (subj, obj)
# - template: a string containing '<subj>', '<obj>', and optionally '<mask>'
# - mask_token: optional string to replace '<mask>' in template

ARGS = {
    "word_pair": ("cat", "animal"),
    "template": "The <subj> is a type of <obj>. <mask>",
    "mask_token": "[MASK]"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
default_gen_kwargs Ran
ran on a synthesized input
neuralmagic/lm-evaluation-harness/lm_eval/defaults.py
code served (permissive licence) · get_code("03edfbaef45e2c28")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Test case 1: until is None
ARGS_1 = {
    'until': None,
    'max_gen_toks': 256,
}

# Test case 2: until is a string
ARGS_2 = {
    'until': "stop_token",
    'max_gen_toks': 256,
}

# Test case 3: until is a list of strings
ARGS_3 = {
    'until': ["token1", "token2"],
    'max_gen_toks': 512,
}

# Use the most basic case for ARGS
ARGS = ARGS_1
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
escaped_split Ran
ran; our contract draft was wrong, not the code
vilm-ai/viet-llm-eval/lm_eval/utils.py
code served (permissive licence) · get_code("57e732507bad40db")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

# The function escaped_split takes text (a string), sep_char (a single character string),
# and maxsplit (an integer). Let's construct valid arguments.

ARGS = {
    "text": "hello\\,world,foo\\,bar,baz",
    "sep_char": ",",
    "maxsplit": -1
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
euclidean_distance Ran
ran on a synthesized input
asahi417/relbert/relbert/evaluation/analogy_questions.py
code served (permissive licence) · get_code("3866cbd2c9f5fd58")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# The function euclidean_distance takes two iterables (a, b)
# and computes the Euclidean distance by:
# 1. Zipping them together
# 2. Computing squared differences for each pair
# 3. Summing all squared differences
# 4. Taking the square root

# Valid arguments can be lists, tuples, or any iterable of numbers
a = [1.0, 2.0, 3.0, 4.0]
b = [5.0, 6.0, 7.0, 8.0]

ARGS = {
    "a": a,
    "b": b
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_lr Ran
ran on a synthesized input
ramanakshay/nanogpt/src/algorithm/optimizer.py
code served (permissive licence) · get_code("b1e0531f8c17d5d6")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import math

ARGS = {
    "it": 500,
    "warmup_iters": 100,
    "lr_decay_iters": 1000,
    "min_lr": 1e-5,
    "lr": 1e-3,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_result Ran
ran, honoured the contract we drafted
neuralmagic/lm-evaluation-harness/lm_eval/models/gguf.py
pointer only (licence: MIT); the stored copy get_code serves, from MrGGLS/BlockPruner, is pointer only (licence: NONE) · get_code("53354f082d9addc6")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from MrGGLS/BlockPruner, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

get_result Ran
ran, honoured the contract we drafted
insait-institute/lm-evaluation-harness-bg/lm_eval/models/openai_completions.py
code served (permissive licence) · get_code("5decf8cec7692d54")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

class MockLogprobs:
    def __init__(self):
        self.token_logprobs = [-0.5, -0.3, -0.8, -0.2, -0.6, -0.4]
        self.top_logprobs = [
            {"token_a": -0.5, "token_b": -1.2},
            {"token_c": -0.3, "token_d": -0.9},
            {"token_e": -0.8, "token_f": -1.5},
            {"token_g": -0.2, "token_h": -1.1},
            {"token_i": -0.6, "token_j": -1.3},
            {"token_k": -0.4, "token_l": -0.7},
        ]

class MockResponse:
    def __init__(self):
        self.logprobs = MockLogprobs()

response = MockResponse()
ctxlen = 2

ARGS = {
    "response": response,
    "ctxlen": ctxlen,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_result Ran
ran, honoured the contract we drafted
vilm-ai/viet-llm-eval/lm_eval/models/openai_completions.py
code served (permissive licence) · get_code("89b22f3da55a855c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Construct a valid response dict matching OpenAI API structure
response = {
    'logprobs': {
        'token_logprobs': [
            -0.5,   # context token 0
            -0.3,   # context token 1
            -0.2,   # context token 2
            -1.2,   # continuation token 0
            -0.8,   # continuation token 1
            -1.5    # continuation token 2
        ],
        'tokens': [
            'hello',
            'world',
            'this',
            'is',
            'a',
            'test'
        ],
        'top_logprobs': [
            {'hello': -0.5, 'hi': -1.2},           # context token 0
            {'world': -0.3, 'there': -0.9},       # context token 1
            {'this': -0.2, 'that': -0.7},         # context token 2
            {'is': -1.2, 'was': -1.5},            # continuation token 0 (greedy match)
            {'a': -0.8, 'the': -1.1},             # continuation token 1 (greedy match)
            {'test': -1.5, 'example': -1.8}       # continuation token 2 (greedy match)
        ]
    }
}

ctxlen = 3

ARGS = {
    'response': response,
    'ctxlen': ctxlen
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_sample_size Ran
ran, honoured the contract we drafted
Sypherd/lm-evaluation-harness/lm_eval/evaluator_utils.py
code served (permissive licence) · get_code("00f5e7d9ee6636ba")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import math
from typing import List, Optional, Tuple, Union

class MockTask:
    def __init__(self):
        self.eval_docs = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

task = MockTask()

ARGS = {
    "task": task,
    "limit": 0.5
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_sample_size Ran
ran, honoured the contract we drafted
opengptx/lm-evaluation-harness/lm_eval/evaluator_utils.py
code served (permissive licence) · get_code("5b315604fb608f3b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import math
from typing import Dict, List, Optional, Tuple, Union

class MockTask:
    def __init__(self, num_eval_docs: int):
        self.eval_docs = list(range(num_eval_docs))

task = MockTask(num_eval_docs=100)
limit = 0.5

ARGS = {
    "task": task,
    "limit": limit
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_sample_size Ran
ran on a synthesized input
neuralmagic/lm-evaluation-harness/lm_eval/evaluator_utils.py
code served (permissive licence) · get_code("0db35c379a280bd8")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import math

class MockTask:
    def __init__(self, num_eval_docs):
        self.eval_docs = list(range(num_eval_docs))

# Create a task with 100 eval docs
task = MockTask(num_eval_docs=100)

# Test with different limit values
# Case 1: limit as a fraction (< 1.0)
limit_fraction = 0.5

# Case 2: limit as an absolute number (>= 1.0)
limit_absolute = 25

# Case 3: limit as None
limit_none = None

ARGS = {
    "task": task,
    "limit": limit_fraction
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_sample_size Ran
ran on a synthesized input
juletx/lm-evaluation-harness/lm_eval/evaluator_utils.py
code served (permissive licence) · get_code("0fb487fd2495f33f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import math

class MockTask:
    def __init__(self, num_eval_docs):
        self.eval_docs = list(range(num_eval_docs))

# Create a task with 100 eval documents
task = MockTask(num_eval_docs=100)

# Test with a fractional limit (< 1.0)
limit_fractional = 0.5

ARGS = {
    "task": task,
    "limit": limit_fractional
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
handle_arg_string Ran
ran, violated the contract we drafted
vilm-ai/viet-llm-eval/lm_eval/utils.py
code served (permissive licence) · get_code("0ee5d4d20cf990bd")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

def handle_arg_string(arg):
    if arg.lower() == 'true':
        return True
    elif arg.lower() == 'false':
        return False
    elif arg.isnumeric():
        return int(arg)
    try:
        return float(arg)
    except ValueError:
        return arg

ARGS = {
    'arg': 'true'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
hash_args Ran
ran; our contract draft was wrong, not the code
Sypherd/lm-evaluation-harness/lm_eval/api/model.py
code served (permissive licence) · get_code("20a7cc804eb22661")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import hashlib
import json

# Construct valid arguments for hash_args function
# The function takes attr (attribute name) and args (tuple/list of arguments)
# and creates a JSON string to hash

attr = "test_attribute"
args = ("value1", 42, [1, 2, 3], {"key": "value"})

ARGS = {
    "attr": attr,
    "args": args
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
hash_args Ran
ran; our contract draft was wrong, not the code
vilm-ai/viet-llm-eval/lm_eval/api/model.py
pointer only (licence: MIT); the stored copy get_code serves, from opengvlab/videochat-r1, is pointer only (licence: NONE) · get_code("ea06eaae4fc1eaf0")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from opengvlab/videochat-r1, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

hash_args Ran
ran on a synthesized input
neuralmagic/lm-evaluation-harness/lm_eval/api/model.py
code served (permissive licence) · get_code("26519d22d8af6d48")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np
from typing import Iterable, Any

torch.manual_seed(0)

# Analyze the function:
# - hash_args takes attr (str) and args (Iterable[Any])
# - It converts them to JSON with [attr] + list(args)
# - JSON serialization requires hashable/serializable types
# - The function doesn't actually use the values beyond JSON encoding

ARGS = {
    "attr": "example_attribute",
    "args": ["value1", 42, 3.14, True, None]
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
hash_args Ran
ran on a synthesized input
EleutherAI/lm_evaluation_harness/lm_eval/api/model.py
code served (permissive licence) · get_code("72f5b387b64e9d9b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import hashlib
import json
from collections.abc import Iterable
from typing import Any

# Setup torch for potential use
import torch
torch.manual_seed(0)

# Define the function (provided)
def _multimodal_cache_default(obj: Any) -> Any:
    """JSON ``default`` handler for cache-key hashing.

    Replaces multimodal payloads with type-tagged content digests so that
    request keys stay deterministic and content-sensitive:

    - separate image objects with identical content map to the same key;
    - different content maps to a different key;
    - anything unsupported keeps failing loudly (no unstable object reprs).
    """
    if isinstance(obj, (bytes, bytearray)):
        return {'__bytes_sha256__': hashlib.sha256(bytes(obj)).hexdigest()}
    try:
        from PIL import Image
        if isinstance(obj, Image.Image):
            from lm_eval.utils import convert_pil_to_hash
            return {'__pil_image_sha256__': convert_pil_to_hash(obj)}
    except ImportError:
        pass
    raise TypeError(f'Object of type {type(obj).__name__} is not JSON serializable')

def hash_args(attr: str, args: Iterable[Any]) -> str:
    dat = json.dumps([attr] + list(args), default=_multimodal_cache_default)
    return hashlib.sha256(dat.encode('utf-8')).hexdigest()


# Construct valid arguments
ARGS = {
    'attr': 'test_method',
    'args': [
        'string_value',
        42,
        3.14,
        {'key': 'value'},
        [1, 2, 3],
        True,
        None,
        bytes(b'test_bytes'),
        bytearray(b'test_bytearray'),
    ]
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
PIL, lm_eval, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
hash_args Ran
ran on a synthesized input
juletx/lm-evaluation-harness/lm_eval/api/model.py
code served (permissive licence) · get_code("3e9d38ecacd03fa7")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import hashlib
import json
from typing import Any, Iterable

def hash_args(attr: str, args: Iterable[Any]) -> str:
    dat = json.dumps([attr] + list(args))
    return hashlib.sha256(dat.encode('utf-8')).hexdigest()

torch.manual_seed(0)

ARGS = {
    "attr": "test_attribute",
    "args": [1, "string_value", 3.14, {"key": "value"}, [1, 2, 3]]
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
is_even Ran
ran on a synthesized input
volcengine/vegiantmodel/src/veGiantModel/engine/engine.py
code served (permissive licence) · get_code("2e6ad295f92e3d9d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

ARGS = {
    "number": 4
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
list_split Ran
ran on a synthesized input
nlx-group/overlapy/overlapy.py
code served (permissive licence) · get_code("deba15e38b692b23")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Construct arguments for list_split function
# The function takes a list and splits it into N sections

# Example 1: Split a list of 10 elements into 3 sections
lst_example1 = list(range(10))
sections_example1 = 3

# Example 2: Split a list of 20 elements into 4 sections
lst_example2 = list(range(20))
sections_example2 = 4

# Use the first example as the main test case
ARGS = {
    "lst": list(range(10)),
    "sections": 3
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list — a list output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_object Ran
ran on a synthesized input
VachanVY/gpt.jax/model/model.py
code served (permissive licence) · get_code("2ee40684ab58dcb9")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pickle
import tempfile
import os

# Create a temporary file with a pickled object
temp_dir = tempfile.mkdtemp()
temp_path = os.path.join(temp_dir, "test_object.pkl")

# Create a simple test object and pickle it
test_object = {"key": "value", "number": 42, "list": [1, 2, 3]}
with open(temp_path, 'wb') as f:
    pickle.dump(test_object, f)

ARGS = {
    "path": temp_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
loss_nce Ran
ran on a synthesized input
asahi417/relbert/relbert/util.py
code served (permissive licence) · get_code("fdde66f5d1999801")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import nn

torch.manual_seed(0)

# Create small tensors for positive and negative samples
# Shape: (batch_size, feature_dim)
batch_size_positive = 2
batch_size_negative = 3
feature_dim = 4

tensor_positive = torch.randn(batch_size_positive, feature_dim)
tensor_negative = torch.randn(batch_size_negative, feature_dim)

# Create a simple linear layer for the optional linear parameter
# The function concatenates [tensor_positive[i], tensor_positive[j], abs_diff]
# which is feature_dim + feature_dim + feature_dim = 3 * feature_dim
# Output should be 1 (for sigmoid in bce loss)
linear_model = nn.Linear(3 * feature_dim, 1)

# Define all arguments
ARGS = {
    'tensor_positive': tensor_positive,
    'tensor_negative': tensor_negative,
    'temperature': 1.0,
    'info_loob': False,
    'linear': linear_model,
    'device': 'cpu'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
loss_triplet Ran
ran on a synthesized input
asahi417/relbert/relbert/util.py
code served (permissive licence) · get_code("bd67f40255aa71d3")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import nn

torch.manual_seed(0)

# Create small tensors with distinct dimensions
batch_size = 2
embedding_dim = 4

tensor_anchor = torch.randn(batch_size, embedding_dim)
tensor_positive = torch.randn(batch_size, embedding_dim)
tensor_negative = torch.randn(batch_size, embedding_dim)
tensor_positive_parent = torch.randn(batch_size, embedding_dim)
tensor_negative_parent = torch.randn(batch_size, embedding_dim)

# Create a simple linear model for classification loss
# The feature dimension is 3 * embedding_dim (anchor, other, difference)
feature_dim = 3 * embedding_dim
linear_model = nn.Linear(feature_dim, 1)

ARGS = {
    'tensor_anchor': tensor_anchor,
    'tensor_positive': tensor_positive,
    'tensor_negative': tensor_negative,
    'tensor_positive_parent': tensor_positive_parent,
    'tensor_negative_parent': tensor_negative_parent,
    'margin': 1.0,
    'linear': linear_model,
    'device': 'cpu'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
oa_chat_completion Ran
ran on a synthesized input
vilm-ai/viet-llm-eval/lm_eval/models/openai_completions.py
code served (permissive licence) · get_code("7cacc3b113e10007")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Mock. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import time
from unittest.mock import Mock, AsyncMock, MagicMock

# Create a mock OpenAI client
client = Mock()

# Create a mock response object
mock_response = Mock()
mock_response.choices = [Mock(message=Mock(content="test response"))]

# Set up the client to return the mock response when create is called
client.chat = Mock()
client.chat.completions = Mock()
client.chat.completions.create = Mock(return_value=mock_response)

# Construct kwargs that would be passed to the OpenAI API
kwargs = {
    "model": "gpt-3.5-turbo",
    "messages": [
        {"role": "user", "content": "Hello, how are you?"}
    ],
    "temperature": 0.7,
    "max_tokens": 100
}

ARGS = {
    "client": client,
    **kwargs
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
openai, tiktoken — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Mock — a Mock output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
pad_vocab Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
karpathy/llm.c/train_gpt2.py
code served (permissive licence) · get_code("cc82794b6ec72130")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Sonnet 4.5, prompt python_fixture_v1) and returned Tensor, ndim 2, dtype family float, shape (50304, 8) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function expects a 2D tensor with shape (50257, C) where C is some embedding dimension
# We'll create a tensor with the exact vocab size of 50257 and a small embedding dimension
V = 50257
C = 8  # small embedding dimension

tensor = torch.randn(V, C)

ARGS = {
    'tensor': tensor,
    'multiple': 128,
    'value': 0
}
Setup block drafted by
a language model, Claude Sonnet 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (50304, 8) — shape (50304, 8) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
parse_eval_args Ran
ran; our contract draft was wrong, not the code
Sypherd/lm-evaluation-harness/lm_eval/__main__.py
pointer only (licence: MIT); the stored copy get_code serves, from BatsResearch/crosslingual-test-time-scaling, is pointer only (licence: NONE) · get_code("400ea4f64ccce5ef")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from BatsResearch/crosslingual-test-time-scaling, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

positional_function Ran
ran, honoured the contract we drafted
abhaskumarsinha/MinimalGPT/GPT.py
code served (permissive licence) · get_code("e60f912106425ef4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Sonnet 4.5, prompt python_fixture_v1) and returned ndarray, ndim 2, dtype family float, shape (8, 16) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np
import math

# The function takes two integer parameters:
# - words: number of words/positions
# - embedding: embedding dimension

# Looking at the function body:
# - It creates a numpy array of shape (words, embedding)
# - It iterates over words and embedding dimensions
# - The formula references 512 as a hardcoded value (likely the original embedding dim)

# We'll use small values for testing
words = 8
embedding = 16

ARGS = {
    'words': words,
    'embedding': embedding
}
Setup block drafted by
a language model, Claude Sonnet 4.5, prompt python_fixture_v1
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 2, dtype family float, shape (8, 16) — shape (8, 16) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
prepare_print_tasks Ran
ran; our contract draft was wrong, not the code
opengptx/lm-evaluation-harness/lm_eval/evaluator_utils.py
code served (permissive licence) · get_code("809bc9208dd99dce")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import collections

# Construct valid arguments for prepare_print_tasks

# task_hierarchy: Dictionary representing the group hierarchy of tasks
# Each key is a group name and its value is a list of task names
task_hierarchy = {
    'group1': ['task1', 'task2'],
    'task1': [],
    'task2': []
}

# results: Dictionary containing the results of each task
# Each key is a group name and its value is a dictionary of task results
results = {
    'group1': {
        'accuracy': 0.95,
        'loss': 0.05,
        'samples': 100,
        'alias': 'Group 1'
    },
    'task1': {
        'accuracy': 0.92,
        'loss': 0.08,
        'samples': 50
    },
    'task2': {
        'accuracy': 0.98,
        'loss': 0.02,
        'samples': 50
    }
}

# tab: The indentation level for printing (default is 0)
tab = 0

ARGS = {
    'task_hierarchy': task_hierarchy,
    'results': results,
    'tab': tab
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
rand_sample Ran
ran on a synthesized input
asahi417/relbert/relbert/trainer.py
code served (permissive licence) · get_code("63b45afd720a808e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import random

def rand_sample(_list):
    return _list[random.randint(0, len(_list) - 1)]

# Setup code
sample_list = [1, 2, 3, 4, 5, "apple", "banana", [10, 20], {"key": "value"}]

ARGS = {
    "_list": sample_list
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
remove_none_pattern Ran
ran; our contract draft was wrong, not the code
sambanova/lm-evaluation-harness/lm_eval/logging_utils.py
code served (permissive licence) · get_code("de43c7cdbf869350")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re
from typing import Any, Dict, List, Literal, Optional, Tuple, Union

def remove_none_pattern(input_string: str) -> Tuple[str, bool]:
    """Remove the ',none' substring from the input_string if it exists at the end.

    Args:
        input_string (str): The input string from which to remove the ',none' substring.

    Returns:
        Tuple[str, bool]: A tuple containing the modified input_string with the ',none' substring removed
                          and a boolean indicating whether the modification was made (True) or not (False).
    """
    pattern = re.compile(',none$')
    result = re.sub(pattern, '', input_string)
    removed = result != input_string
    return (result, removed)


ARGS = {
    "input_string": "test_value,none"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
request_caching_arg_to_dict Ran
ran; our contract draft was wrong, not the code
Sypherd/lm-evaluation-harness/lm_eval/evaluator.py
code served (permissive licence) · get_code("c44830a7722ff5f7")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

ARGS = {
    'cache_requests': 'true'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
restore_list Ran
ran on a synthesized input
asahi417/relbert/relbert/list_keeper.py
code served (permissive licence) · get_code("99b9d9d7a4d6ca29")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np
from copy import deepcopy
from typing import List

torch.manual_seed(0)

# Construct arguments for restore_list function
# The function takes:
# 1. original_list: List - the template structure to restore
# 2. flatten_value: List - the flat values to fill in

# Create a nested list structure with various nesting levels
original_list = [1, [2, 3], [[4, 5], 6], 7]

# Create a flat list of replacement values
# Count the leaf nodes in original_list to know how many values we need
# Leaves: 1, 2, 3, 4, 5, 6, 7 = 7 leaves
flatten_value = [10, 20, 30, 40, 50, 60, 70]

ARGS = {
    "original_list": original_list,
    "flatten_value": flatten_value,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list — a list output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
safe_json_load Ran
ran on a synthesized input
asahi417/relbert/relbert/relbert_cl/push_to_hub.py
code served (permissive licence) · get_code("c7a7a9dca42a1347")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import json
import tempfile

# Create a temporary JSON file for testing
temp_dir = tempfile.mkdtemp()
temp_file = os.path.join(temp_dir, "test.json")

# Write a valid JSON file
test_data = {"key": "value", "number": 42, "list": [1, 2, 3]}
with open(temp_file, 'w') as f:
    json.dump(test_data, f)

ARGS = {
    "_file": temp_file
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
simple_parse_args_string Ran
ran; our contract draft was wrong, not the code
vilm-ai/viet-llm-eval/lm_eval/utils.py
code served (permissive licence) · get_code("dfed6e9291316231")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Analyze the function:
# simple_parse_args_string takes args_string parameter
# It processes a string like "args1=val1,arg2=val2"
# Returns a dictionary with parsed key-value pairs
# handle_arg_string converts string values to appropriate types (bool, int, float, or str)

# Test cases to construct valid arguments:

ARGS = {
    "args_string": "learning_rate=0.001,epochs=10,use_dropout=true,model_name=resnet,batch_size=32,debug=false"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
steer Ran
ran on a synthesized input
juletx/lm-evaluation-harness/lm_eval/models/hf_steered.py
code served (permissive licence) · get_code("cabc905de3f72170")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned _GeneratorContextManager. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from transformers import PreTrainedModel
from peft.peft_model import PeftModel
from unittest.mock import MagicMock

torch.manual_seed(0)

# Create a minimal model with named_modules that includes hookpoints
class MinimalTransformer(nn.Module):
    def __init__(self):
        super().__init__()
        self.embedding = nn.Linear(4, 8)
        self.layer1 = nn.Linear(8, 8)
        self.layer2 = nn.Linear(8, 4)
    
    def forward(self, x):
        x = self.embedding(x)
        x = self.layer1(x)
        x = self.layer2(x)
        return x

# Create a mock PreTrainedModel with base_model attribute
class MockPeftModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.base_model = MinimalTransformer()
    
    def forward(self, x):
        return self.base_model(x)

model = MockPeftModel()

# Create steering functions that transform the output tensor
def steer_func_1(output):
    return output * 2.0

def steer_func_2(output):
    return output + 1.0

# Create hook_to_steer dict mapping module names to steering functions
hook_to_steer = {
    "layer1": steer_func_1,
    "layer2": steer_func_2,
}

ARGS = {
    "model": model,
    "hook_to_steer": hook_to_steer,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
peft, torch, transformers — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type _GeneratorContextManager — a _GeneratorContextManager output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a _GeneratorContextManager output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
to_tensor Ran
ran on a synthesized input
asahi417/relbert/relbert/trainer.py
code served (permissive licence) · get_code("11b1822611370e59")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

ARGS = {
    'name': 'attention_mask',
    'data': [[1, 0, 1, 1], [1, 1, 0, 1]]
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
add_byte_giant_model_customize_args Not yet run
unverified (harvested, no recorded run)
volcengine/vegiantmodel/src/veGiantModel/initialize.py
code served (permissive licence) · get_code("3bb8fcfcb4c12121")
How it ran

no invocation recorded.

bps_recv Not yet run
unverified (harvested, no recorded run)
volcengine/vegiantmodel/src/veGiantModel/engine/p2p.py
code served (permissive licence) · get_code("e66a197775436632")
How it ran

no invocation recorded.

bps_send Not yet run
unverified (harvested, no recorded run)
volcengine/vegiantmodel/src/veGiantModel/engine/p2p.py
code served (permissive licence) · get_code("7ca8b4cd5f558e02")
How it ran

no invocation recorded.

build_model Not yet run
unverified (harvested, no recorded run)
VachanVY/gpt.jax/model/model.py
code served (permissive licence) · get_code("9dd643ee2669464c")
How it ran

no invocation recorded.

em_accuracy_helper Not yet run
unverified (harvested, no recorded run)
this paper's copy was not recorded; identical code first harvested from tonyzhaozh/few-shot-learning
pointer only · get_code("e1d89d7579eaaeec")
How it ran

The body of this sample is withheld: this paper's copy of the code was not recorded (its link names no repository or source path), so no licence was read for it. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

eval_accuracy Not yet run
unverified (harvested, no recorded run)
this paper's copy was not recorded; identical code first harvested from tonyzhaozh/few-shot-learning
pointer only · get_code("7753000f9482cac5")
How it ran

The body of this sample is withheld: this paper's copy of the code was not recorded (its link names no repository or source path), so no licence was read for it. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

evaluate_analogy Not yet run
unverified (harvested, no recorded run)
asahi417/relbert/relbert/evaluation/analogy_questions.py
code served (permissive licence) · get_code("34f65890e3e58ec4")
How it ran

no invocation recorded.

evaluate_classification Not yet run
unverified (harvested, no recorded run)
asahi417/relbert/relbert/evaluation/lexical_relation_classification.py
code served (permissive licence) · get_code("87cfebcf19ab803b")
How it ran

no invocation recorded.

evaluate_relation_mapping Not yet run
unverified (harvested, no recorded run)
asahi417/relbert/relbert/evaluation/relation_mapping.py
code served (permissive licence) · get_code("c7271f2f2ce3c4fa")
How it ran

no invocation recorded.

flatten_list Not yet run
unverified (harvested, no recorded run)
asahi417/relbert/relbert/list_keeper.py
code served (permissive licence) · get_code("c56b548854fce3e6")
How it ran

no invocation recorded.

get_commit_from_path Not yet run
unverified (harvested, no recorded run)
sambanova/lm-evaluation-harness/lm_eval/logging_utils.py
code served (permissive licence) · get_code("f6825b4eb5fe78c8")
How it ran

no invocation recorded.

get_distribution_scope Not yet run
unverified (harvested, no recorded run)
abhaskumarsinha/Corpus2GPT/training_utils/distribution_utils.py
code served (permissive licence) · get_code("306b51e6dcfb30ef")
How it ran

no invocation recorded.

get_label_probs Not yet run
unverified (harvested, no recorded run)
this paper's copy was not recorded; identical code first harvested from tonyzhaozh/few-shot-learning
pointer only · get_code("77347b697708c203")
How it ran

The body of this sample is withheld: this paper's copy of the code was not recorded (its link names no repository or source path), so no licence was read for it. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

get_p_content_free Not yet run
unverified (harvested, no recorded run)
this paper's copy was not recorded; identical code first harvested from tonyzhaozh/few-shot-learning
pointer only · get_code("d09226e56c2085ea")
How it ran

The body of this sample is withheld: this paper's copy of the code was not recorded (its link names no repository or source path), so no licence was read for it. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

get_readme Not yet run
unverified (harvested, no recorded run)
asahi417/relbert/relbert/relbert_cl/readme_template.py
code served (permissive licence) · get_code("8e3012dd13fbb68a")
How it ran

no invocation recorded.

Showing the first 60. An agent gets every one from get_harvested_code_for_paper("2005.14165").

Also present in other papers' code

Of 65 samples lifted from this paper's linked repositories, 65 carry a function identity and 0 do not. Of the 61 distinct functions with an identity, 20 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 20 later) and 41 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
hash_args 44 other papers
43 later, 1 undated
2110.04374 (2021-10)
A Few More Examples May Be Worth Billions of Parameters
2303.17003 (2023-03)
Evaluating GPT-3.5 and GPT-4 Models on Brazilian University Admission Exams
2304.01089 (2023-04)
RPTQ: Reorder-based Post-training Quantization for Large Language Models
and 41 more not named here
request_caching_arg_to_dict 17 other papers
17 later
2405.07938 (2024-05)
EconLogicQA: A Question-Answering Benchmark for Evaluating Large Language Models in Economic Sequential Reasoning
2411.07176 (2024-11)
More Expressive Attention with Negative Weights
2411.10606 (2024-11)
AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deployment
and 14 more not named here
parse_eval_args 15 other papers
14 later, 1 undated
2024.findings-emnlp.86 (2024 (year only))
Tokenization Falling Short: On Subword Robustness in Large Language Models
2405.07938 (2024-05)
EconLogicQA: A Question-Answering Benchmark for Evaluating Large Language Models in Economic Sequential Reasoning
2406.11687 (2024-06)
Tokenization Falling Short: On Subword Robustness in Large Language Models
and 12 more not named here
get_sample_size 12 other papers
12 later
2405.07938 (2024-05)
EconLogicQA: A Question-Answering Benchmark for Evaluating Large Language Models in Economic Sequential Reasoning
2412.14171 (2024-12)
Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces
2025.findings-acl.744 (2025 (year only))
Ranked Voting based Self-Consistency of Large Language Models
and 9 more not named here
escaped_split 9 other papers
9 later
2305.14493 (2023-05)
Do prompt positions really matter?
2307.16039 (2023-07)
Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback
2310.05620 (2023-10)
LAiW: A Chinese Legal Large Language Models Benchmark
and 6 more not named here
get_result 4 other papers
4 later
2312.15838 (2023-12)
SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Security
2405.07938 (2024-05)
EconLogicQA: A Question-Answering Benchmark for Evaluating Large Language Models in Economic Sequential Reasoning
2406.10594 (2024-06)
BlockPruner: Fine-grained Pruning for Large Language Models
and 1 more not named here
get_label_probs 3 other papers
3 later
2102.09690 (2021-02)
Calibrate Before Use: Improving Few-Shot Performance of Language Models
2401.12406 (2024-01)
Enhancing In-context Learning via Linear Probe Calibration
2411.02988 (2024-11)
Confidence Calibration of Classifiers with Many Classes
get_p_content_free 2 other papers
2 later
2102.09690 (2021-02)
Calibrate Before Use: Improving Few-Shot Performance of Language Models
2312.04021 (2023-12)
A Study on the Calibration of In-context Learning
simple_parse_args_string 2 other papers
2 later
2312.15838 (2023-12)
SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Security
2503.12496 (2025-03)
Does Your Vision-Language Model Get Lost in the Long Video Sampling Dilemma?
em_accuracy_helper 1 other paper
1 later
2102.09690 (2021-02)
Calibrate Before Use: Improving Few-Shot Performance of Language Models
eval_accuracy 1 other paper
1 later
2102.09690 (2021-02)
Calibrate Before Use: Improving Few-Shot Performance of Language Models
get_result 1 other paper
1 later
2312.15838 (2023-12)
SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Security
get_result 1 other paper
1 later
2504.08165 (2025-04)
Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
handle_arg_string 1 other paper
1 later
2312.15838 (2023-12)
SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Security
load_model 1 other paper
1 later
2405.07938 (2024-05)
EconLogicQA: A Question-Answering Benchmark for Evaluating Large Language Models in Economic Sequential Reasoning
pad_vocab 1 other paper
1 later
2305.07759 (2023-05)
TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
prepare_print_tasks 1 other paper
1 later
2405.07938 (2024-05)
EconLogicQA: A Question-Answering Benchmark for Evaluating Large Language Models in Economic Sequential Reasoning
remove_none_pattern 1 other paper
1 later
2405.07938 (2024-05)
EconLogicQA: A Question-Answering Benchmark for Evaluating Large Language Models in Economic Sequential Reasoning
run_inference_one_model 1 other paper
1 later
2312.15838 (2023-12)
SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Security
setup_distributed_environment 1 other paper
1 later
2405.07938 (2024-05)
EconLogicQA: A Question-Answering Benchmark for Evaluating Large Language Models in Economic Sequential Reasoning

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

More are linked than shown here.

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions -something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art finetuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2005.14165")
get_code_for_paper("2005.14165")
have("2005.14165")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2005.14165.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2005.14165.svg)](https://syntology.ai/paper/2005.14165)

Connect an agent — have() is free.