SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2402.14992 · ICML · 2024

tinyBenchmarks: evaluating LLMs with fewer examples

Mikhail Yurochkin, Yuekai Sun, Leshem Choshen, Felipe Polo, Lucas Weber, Gongjun Xu

arXiv · PDF · Open in the Atlas

Code that ran

25 of 39 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 5 honoured the contract we drafted, 1 violated it and 15 have no contract verdict; 4 failures of our instrument; 14 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2402.14992.json · badge: /api/ran/2402.14992.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
felipemaiapolo/efficbench canonical (classifier label)
classifier basis: census
7 ran · 10 no recorded run
felipemaiapolo/tinybenchmarks canonical (classifier label)
classifier basis: census
3 ran · 2 no recorded run
mariushobbhahn/swebench-verified-mini community (classifier label)
classifier basis not recorded on the link
15 ran · 2 no recorded run
FunctionStatusWhere it lives
check_file_exists Ran
ran on a synthesized input
mariushobbhahn/swebench-verified-mini/run_all.py
code served (permissive licence) · get_code("3e37b7d40582ad9d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

from pathlib import Path
import tempfile
import os

# Create a temporary file that exists
with tempfile.NamedTemporaryFile(delete=False) as tmp:
    existing_file = Path(tmp.name)

# Create a Path object to a file that does not exist
non_existing_file = Path(tempfile.gettempdir()) / "this_file_definitely_does_not_exist_12345.txt"

ARGS = {
    "file_path": existing_file
}

# Clean up
try:
    os.unlink(existing_file)
except:
    pass
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
compute_kmeans Ran
ran on a synthesized input
mariushobbhahn/swebench-verified-mini/generate_subsets.py
code served (permissive licence) · get_code("44ffb79d93d3714c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd
import numpy as np
from sklearn.cluster import KMeans

# Create a small DataFrame with numeric data
np.random.seed(42)
data = pd.DataFrame(
    np.random.randn(10, 4),
    columns=['feature_1', 'feature_2', 'feature_3', 'feature_4']
)

ARGS = {
    'data': data,
    'n_clusters': 3,
    'random_state': 42
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, pandas, sklearn — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
convert_to_gb Ran
ran on a synthesized input
mariushobbhahn/swebench-verified-mini/get_docker_image_sizes.py
code served (permissive licence) · get_code("607e7cb82032195e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Test cases for convert_to_gb function
ARGS = {
    "size_str": "77.9MB"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
count_test_results Ran
ran on a synthesized input
mariushobbhahn/swebench-verified-mini/extract_data_from_logs.py
code served (permissive licence) · get_code("8ae80ca052410f1d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 2, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Construct a test_output string that contains both PASS_TO_PASS and FAIL_TO_PASS sections
test_output = '''
PASS_TO_PASS:
Test case 1: "PASSED"
Test case 2: "PASSED"
Test case 3: "FAILED"
Test case 4: "PASSED"
FAIL_TO_PASS:
Test case 5: "PASSED"
Test case 6: "FAILED"
Test case 7: "FAILED"
'''

ARGS = {
    'test_output': test_output
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 2, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
draw_representative_sample Ran
ran on a synthesized input
mariushobbhahn/swebench-verified-mini/generate_subsets.py
code served (permissive licence) · get_code("ac93c19afba2b273")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd
import numpy as np
from sklearn.cluster import KMeans
from sklearn.utils import resample

# Create a small sample DataFrame with the expected columns
np.random.seed(1)
data = pd.DataFrame({
    'id': ['task_1', 'task_2', 'task_3', 'task_4', 'task_5', 'task_6'],
    'difficulty': ['<15 min fix', '15 min - 1 hour', '1-4 hours', '>4 hours', '<15 min fix', '15 min - 1 hour'],
    'size_in_gb': [1.5, 2.3, 0.8, 3.2, 1.1, 2.0],
    'environment': ['prod', 'staging', 'prod', 'dev', 'staging', 'prod'],
    'priority': [1, 2, 3, 4, 1, 2],
    'complexity_score': [0.5, 0.7, 0.9, 0.3, 0.4, 0.6]
})

# Create labels via KMeans clustering
data_numeric_temp = data.drop(columns=['id', 'size_in_gb', 'environment']).copy()
difficulty_map = {'<15 min fix': 0, '15 min - 1 hour': 1, '1-4 hours': 2, '>4 hours': 3}
data_numeric_temp['difficulty'] = data_numeric_temp['difficulty'].map(difficulty_map)
data_numeric_temp = data_numeric_temp.fillna(0)

kmeans_temp = KMeans(n_clusters=2, random_state=1)
labels = kmeans_temp.fit_predict(data_numeric_temp)

ARGS = {
    'data': data,
    'labels': labels,
    'n': 4,
    'random_state': 1
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, pandas, sklearn — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
filter_dataset_by_ids Ran
ran on a synthesized input
mariushobbhahn/swebench-verified-mini/make_new_huggingface_dataset.py
code served (permissive licence) · get_code("bac16ddf5e843b55")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DatasetDict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import datasets
from datasets import Dataset, DatasetDict

# Create a small dataset with instance_id column
data = {
    'instance_id': ['id_1', 'id_2', 'id_3', 'id_4'],
    'text': ['sample 1', 'sample 2', 'sample 3', 'sample 4'],
    'label': [0, 1, 0, 1]
}

# Create a DatasetDict with train and test splits
dataset_dict = DatasetDict({
    'train': Dataset.from_dict(data),
    'test': Dataset.from_dict({
        'instance_id': ['id_5', 'id_6'],
        'text': ['sample 5', 'sample 6'],
        'label': [0, 1]
    })
})

# List of IDs to filter by
ids_to_keep = ['id_1', 'id_3', 'id_5']

ARGS = {
    'dataset': dataset_dict,
    'ids': ids_to_keep
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
datasets — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DatasetDict — a DatasetDict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DatasetDict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_data Ran
ran; our contract draft was wrong, not the code
felipemaiapolo/efficbench/run_experiment.py
pointer only (licence: NONE) · get_code("f79b7ee882bf1224")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, felipemaiapolo/efficbench when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

get_datas Ran
ran, honoured the contract we drafted
felipemaiapolo/efficbench/generating_data/download-openllmleaderboard/download_model_names/openllm.py
pointer only (licence: NONE) · get_code("3f37f4d4e8b9e152")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, felipemaiapolo/efficbench when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

get_lambda Ran
ran, honoured the contract we drafted
felipemaiapolo/efficbench/utils.py
pointer only (licence: NONE) · get_code("c030bb619c641f8c")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, felipemaiapolo/efficbench when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

get_score_and_test_counts_from_sample Ran
ran on a synthesized input
mariushobbhahn/swebench-verified-mini/extract_data_from_logs.py
code served (permissive licence) · get_code("479a232346a8cdc4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

class MockScorer:
    def __init__(self, value, explanation):
        self.value = value
        self.explanation = explanation

class MockScores:
    def __init__(self, scorer):
        self.scores = {'swe_bench_scorer': scorer}

class MockSample:
    def __init__(self, sample_id, scorer):
        self.id = sample_id
        self.scores = scorer.scores

# Create a valid sample with proper test output format
scorer = MockScorer(
    value=0.75,
    explanation='PASS_TO_PASS:\nTest 1: "PASSED"\nTest 2: "PASSED"\nTest 3: "FAILED"\nFAIL_TO_PASS:\nTest 4: "PASSED"\nTest 5: "FAILED"\nTest 6: "FAILED"'
)
sample = MockSample('sample_001', MockScores(scorer))

ARGS = {
    'sample': sample
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
item_curve Ran
ran, honoured the contract we drafted
felipemaiapolo/efficbench/utils.py
pointer only (licence: NONE) · get_code("d173066d5ff1fb67")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, felipemaiapolo/efficbench when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

item_curve Ran
ran, honoured the contract we drafted
felipemaiapolo/tinybenchmarks/demo/utils.py
code served (permissive licence) · get_code("921caf5fc3c2b540")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (2,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# theta is used in: a * theta
# a is used in: a * theta - b  (element-wise operations, then .sum(axis=1))
# b is used in: a * theta - b

# From the operation "a * theta - b" and ".sum(axis=1)", we can infer:
# - theta should be 2D (batch_size, feature_dim)
# - a should be 1D or broadcastable to theta's shape
# - b should be scalar or broadcastable

# Let's use:
# theta: shape (2, 4) - batch of 2, 4 features
# a: shape (4,) - coefficients for each feature
# b: scalar

theta = np.random.randn(2, 4)
a = np.random.randn(4)
b = 0.5

ARGS = {
    "theta": theta,
    "a": a,
    "b": b,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (2,) — shape (2,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_irt_parameters Ran
ran; our contract draft was wrong, not the code
felipemaiapolo/efficbench/irt.py
pointer only (licence: NONE) · get_code("a744fa187a860f72")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, felipemaiapolo/efficbench when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

load_sample_ids Ran
ran on a synthesized input
mariushobbhahn/swebench-verified-mini/make_new_huggingface_dataset.py
code served (permissive licence) · get_code("6a02dcfbdb7bea7d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
from pathlib import Path

# Create a temporary JSON file with sample IDs
with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f:
    sample_ids = ['sample_001', 'sample_002', 'sample_003', 'sample_004']
    json.dump(sample_ids, f)
    temp_path = f.name

ARGS = {
    'path': Path(temp_path)
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
map_environments Ran
ran on a synthesized input
mariushobbhahn/swebench-verified-mini/get_docker_image_sizes.py
code served (permissive licence) · get_code("f81ca1d465c962aa")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd
from pathlib import Path
import json
import tempfile

# Create a temporary directory to hold the mapping file
temp_dir = tempfile.TemporaryDirectory()
temp_path = Path(temp_dir.name)

# Create a mapping file
mapping_data = {
    "image_1": "production",
    "image_2": "staging",
    "image_3": "development"
}
mapping_file = temp_path / "mapping.json"
with open(mapping_file, 'w') as f:
    json.dump(mapping_data, f)

# Create a sample DataFrame with Docker image IDs
df = pd.DataFrame({
    'ID': ['image_1', 'image_2', 'image_3'],
    'NAME': ['app1', 'app2', 'app3'],
    'VERSION': ['1.0', '2.0', '3.0']
})

ARGS = {
    'df': df,
    'mapping_path': mapping_file
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
merge_data_with_metadata Ran
ran on a synthesized input
mariushobbhahn/swebench-verified-mini/add_metadata_to_data.py
code served (permissive licence) · get_code("d7d8fe87aefb67aa")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd
import numpy as np

# Create sample data DataFrame with 'id' column and some performance metrics
data = pd.DataFrame({
    'id': [101, 102, 103],
    'accuracy': [0.85, 0.92, 0.78],
    'loss': [0.15, 0.08, 0.22]
})

# Create sample annotation DataFrame with 'instance_id' and 'difficulty' columns
annotation = pd.DataFrame({
    'instance_id': [101, 102, 103, 104],
    'difficulty': ['easy', 'hard', 'medium', 'hard']
})

ARGS = {
    'data': data,
    'annotation': annotation
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
merge_model_scores Ran
ran on a synthesized input
mariushobbhahn/swebench-verified-mini/extract_data_from_logs.py
code served (permissive licence) · get_code("a2fa41c1789608ea")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd

# Create sample dataframes with model scores
df1 = pd.DataFrame({
    'id': [1, 2, 3],
    'score': [0.85, 0.92, 0.78],
    'model': ['model_a', 'model_a', 'model_a']
})

df2 = pd.DataFrame({
    'id': [1, 2, 3],
    'score': [0.88, 0.90, 0.81],
    'model': ['model_b', 'model_b', 'model_b']
})

df3 = pd.DataFrame({
    'id': [1, 2, 3],
    'score': [0.91, 0.87, 0.84],
    'model': ['model_c', 'model_c', 'model_c']
})

ARGS = {
    'df_list': [df1, df2, df3]
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
merge_with_environment_data Ran
ran on a synthesized input
mariushobbhahn/swebench-verified-mini/add_metadata_to_data.py
code served (permissive licence) · get_code("ef12fdf35ff27c06")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd
import numpy as np

# Create sample dataframes for the function
df = pd.DataFrame({
    'id': [1, 2, 3, 4],
    'model_performance': [0.85, 0.92, 0.78, 0.88],
    'difficulty': [5, 7, 3, 6]
})

size_env_data = pd.DataFrame({
    'ID': [1, 2, 3, 4],
    'SIZE_IN_GB': [2.5, 3.2, 1.8, 2.9],
    'ENVIRONMENT': ['prod', 'dev', 'test', 'prod']
})

ARGS = {
    'df': df,
    'size_env_data': size_env_data
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
prepare_data Ran
ran; our contract draft was wrong, not the code
felipemaiapolo/tinybenchmarks/tutorials/utils.py
code served (permissive licence) · get_code("ef3515d9063526b8")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Construct a minimal data dictionary that matches the expected structure
# The function accesses data['data'][sub]['correctness'].shape[0]
# So we need correctness arrays for each subscenario

data = {
    'data': {}
}

# Add correctness data for each subscenario
scenarios_dict = {
    'harness_truthfulqa_mc_0': ['harness_truthfulqa_mc_0'],
    'gsm8k': ['harness_gsm8k_5'],
    'winogrande': ['harness_winogrande_5'],
    'arc': ['harness_arc_challenge_25'],
    'hellaswag': ['harness_hellaswag_10'],
    'mmlu': ['harness_hendrycksTest_abstract_algebra_5', 'harness_hendrycksTest_anatomy_5']
}

# Create correctness arrays with different sizes for each subscenario
subscenario_list = []
for scenario_name, subscenarios in scenarios_dict.items():
    for subscenario in subscenarios:
        subscenario_list.append(subscenario)

# Assign different sizes to each subscenario to make them distinct
for idx, subscenario in enumerate(subscenario_list):
    # Use idx + 2 to create different batch sizes (2, 3, 4, 5, 6, 7, ...)
    size = idx + 2
    data['data'][subscenario] = {
        'correctness': np.ones((size,), dtype=np.float32)
    }

ARGS = {
    'scenarios': scenarios_dict,
    'data': data
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
prepare_data Ran
ran on a synthesized input
mariushobbhahn/swebench-verified-mini/generate_subsets.py
code served (permissive licence) · get_code("cc7ed1b21213f9b3")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd

# Create a sample DataFrame with the columns that the function expects
# The function drops 'id', 'size_in_gb', 'environment' and maps 'difficulty'
data = pd.DataFrame({
    'id': [1, 2, 3],
    'difficulty': ['<15 min fix', '15 min - 1 hour', '1-4 hours'],
    'size_in_gb': [10, 20, 30],
    'environment': ['prod', 'dev', 'staging'],
    'priority': [1, 2, 3],
    'impact': [0.5, 0.7, 0.9]
})

ARGS = {
    'data': data
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
prepare_numeric_data Ran
ran on a synthesized input
mariushobbhahn/swebench-verified-mini/compare_subsets.py
code served (permissive licence) · get_code("f20343e9cf5ded69")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd
import numpy as np

# Create a sample DataFrame with the columns that the function expects to handle
data = pd.DataFrame({
    'id': [1, 2],
    'difficulty': ['<15 min fix', '1-4 hours'],
    'environment': ['prod', 'dev'],
    'size_in_gb': [10.5, 20.3],
    'metric1': [1.5, 2.3],
    'metric2': [np.nan, 4.5],
    'metric3': [5.0, np.nan],
})

ARGS = {
    'data': data,
    'replace_difficulty_with_number': False,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
process_docker_output Ran
ran on a synthesized input
mariushobbhahn/swebench-verified-mini/get_docker_image_sizes.py
code served (permissive licence) · get_code("fb3c33baacb5cdd5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd
import numpy as np

# Create a sample DataFrame that mimics raw Docker output
df = pd.DataFrame({
    'REPOSITORY': ['sweb.eval.x86_64.repo1', 'sweb.eval.x86_64.repo2', 'sweb.eval.x86_64.repo3'],
    'TAG': ['latest', 'v1.0', 'v2.0'],
    'IMAGE': ['sha256:abc123', 'sha256:def456', 'sha256:ghi789'],
    'ID': ['2023-01-15', '2023-02-20', '2023-03-10'],
    'CREATED': ['2 weeks ago', '3 weeks ago', '4 weeks ago'],
    'SIZE': ['77.9MB', '2.58GB', '500kB'],
    'SHARED': ['10.5MB', '1.2GB', '50kB'],
    'SIZE.1': ['67.4MB', '1.38GB', '450kB'],
    'UNIQUE': ['512MB', '256MB', '100MB'],
    'SIZE.2': [2, 1, 3]
})

ARGS = {
    'df': df
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
sigmoid Ran
ran, violated the contract we drafted
felipemaiapolo/tinybenchmarks/demo/utils.py
code served (permissive licence) · get_code("a911f9cacbc9faf4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# Create a small numpy array to pass as argument
z_value = np.array([0.0, 1.0, -1.0, 2.0])

ARGS = {
    "z": z_value
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
sigmoid Ran
ran, honoured the contract we drafted
felipemaiapolo/efficbench/utils.py
pointer only (licence: NONE) · get_code("613d5fa522ce2fe7")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, felipemaiapolo/efficbench when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

winrate Ran
ran; our contract draft was wrong, not the code
felipemaiapolo/efficbench/plots.py
pointer only (licence: NONE) · get_code("f9a2099f222c55c0")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, felipemaiapolo/efficbench when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

calculate_accuracies Not yet run
unverified (harvested, no recorded run)
felipemaiapolo/efficbench/acc.py
pointer only (licence: NONE) · get_code("665d15ac51c28fd5")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, felipemaiapolo/efficbench when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

compare_difficulty_distributions Not yet run
unverified (harvested, no recorded run)
mariushobbhahn/swebench-verified-mini/compare_subsets.py
code served (permissive licence) · get_code("bf1d584f11529d41")
How it ran

no invocation recorded.

compare_means Not yet run
unverified (harvested, no recorded run)
mariushobbhahn/swebench-verified-mini/compare_subsets.py
code served (permissive licence) · get_code("fd966a7ef33f7a71")
How it ran

no invocation recorded.

compute_acc_pirt Not yet run
unverified (harvested, no recorded run)
felipemaiapolo/efficbench/acc.py
pointer only (licence: NONE) · get_code("db63652efa566108")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, felipemaiapolo/efficbench when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

estimate_ability_parameters Not yet run
unverified (harvested, no recorded run)
felipemaiapolo/efficbench/irt.py
pointer only (licence: NONE) · get_code("a0e44adf60096817")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, felipemaiapolo/efficbench when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

estimate_ability_parameters Not yet run
unverified (harvested, no recorded run)
felipemaiapolo/tinybenchmarks/tutorials/irt.py
code served (permissive licence) · get_code("5cfb956fb08b8f0e")
How it ran

no invocation recorded.

evaluate_scenarios Not yet run
unverified (harvested, no recorded run)
felipemaiapolo/efficbench/experiments.py
pointer only (licence: NONE) · get_code("1153f14380c21738")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, felipemaiapolo/efficbench when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

fit_theta Not yet run
unverified (harvested, no recorded run)
felipemaiapolo/tinybenchmarks/demo/utils.py
code served (permissive licence) · get_code("bfcec22b9dae71a3")
How it ran

no invocation recorded.

get_commit_dates_models Not yet run
unverified (harvested, no recorded run)
felipemaiapolo/efficbench/generating_data/download_alpaca/preprocess_alpaca_eval.py
pointer only (licence: NONE) · get_code("9562e2171d59db3a")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, felipemaiapolo/efficbench when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

get_last_commit_date Not yet run
unverified (harvested, no recorded run)
felipemaiapolo/efficbench/generating_data/download_alpaca/preprocess_alpaca_eval.py
pointer only (licence: NONE) · get_code("5c4864e2b480cdcb")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, felipemaiapolo/efficbench when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

get_random Not yet run
unverified (harvested, no recorded run)
felipemaiapolo/efficbench/selection.py
pointer only (licence: NONE) · get_code("68f19e5ceaec8296")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, felipemaiapolo/efficbench when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

load_scores Not yet run
unverified (harvested, no recorded run)
felipemaiapolo/efficbench/plots.py
pointer only (licence: NONE) · get_code("9d2ac225248cfcc2")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, felipemaiapolo/efficbench when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

run_adaptive_selection Not yet run
unverified (harvested, no recorded run)
felipemaiapolo/efficbench/selection.py
pointer only (licence: NONE) · get_code("6620cc858a942cb4")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, felipemaiapolo/efficbench when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

select_initial_adaptive_items Not yet run
unverified (harvested, no recorded run)
felipemaiapolo/efficbench/selection.py
pointer only (licence: NONE) · get_code("5c7d27e94d6596c8")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, felipemaiapolo/efficbench when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

Also present in other papers' code

Of 39 samples lifted from this paper's linked repositories, 39 carry a function identity and 0 do not. Of the 37 distinct functions with an identity, 5 are also present in other papers whose code we hold (1 with at least one paper dated earlier, 0 the same month, 5 later) and 32 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
sigmoid 7 other papers
3 dated earlier, 4 later
2006.06887 (2020-06)
Stochastic Optimization for Performative Prediction
2007.13819 (2020-07)
MULTI-LEVEL LOCAL SGD: DISTRIBUTED SGD FOR HETEROGENEOUS HIERARCHICAL NETWORKS
2207.09176 (2022-07)
Self-Supervision Can Be a Good Few-Shot Learner
and 4 more not named here
item_curve 3 other papers
3 later
2409.03563 (2024-09)
100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instances
2502.10436 (2025-02)
MERGE 3 : Efficient Evolutionary Merging on Consumer-grade GPUs
2506.07673 (2025-06)
How Benchmark Prediction from Fewer Data Misses the Mark
estimate_ability_parameters 1 other paper
1 later
2506.07673 (2025-06)
How Benchmark Prediction from Fewer Data Misses the Mark
load_irt_parameters 1 other paper
1 later
2409.03563 (2024-09)
100 instances is all you need: predicting the success of a new LLM on unseen data by testing on a few instances
prepare_data 1 other paper
1 later
2503.10657 (2025-03)
RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

The versatility of large language models (LLMs) led to the creation of diverse benchmarks that thoroughly test a variety of language models' abilities. These benchmarks consist of tens of thousands of examples making evaluation of LLMs very expensive. In this paper, we investigate strategies to reduce the number of evaluations needed to assess the performance of an LLM on several key benchmarks. For example, we show that to accurately estimate the performance of an LLM on MMLU, a popular multiple-choice QA benchmark consisting of 14K examples, it is sufficient to evaluate this LLM on 100 curated examples. We release evaluation tools and tiny versions of popular benchmarks: Open LLM Leaderboard, MMLU, HELM, and AlpacaEval 2.0. Our empirical analysis demonstrates that these tools and tiny benchmarks are sufficient to reliably and efficiently reproduce the original evaluation results 1 .

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2402.14992")
get_code_for_paper("2402.14992")
have("2402.14992")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2402.14992.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2402.14992.svg)](https://syntology.ai/paper/2402.14992)

Connect an agent — have() is free.