SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2605.05973 · 2026

Towards Reliable LLM Evaluation: Correcting the Winner's Curse in Adaptive Benchmarking

Yang Xu, Zihan Zhou, Vaneet Aggarwal, Tianyu Cao, Haixiang Sun, Jiefu Zhang

arXiv · PDF · Open in the Atlas

Code that ran

24 of 24 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 2 honoured the contract we drafted, 5 violated it and 0 have no contract verdict; 17 failures of our instrument; 0 no recorded run), measured 2026-11-22; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2605.05973.json · badge: /api/ran/2605.05973.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
jznmsl/siren canonical (classifier label)
classifier basis: census
24 ran · 0 no recorded run
FunctionStatusWhere it lives
compute_ground_truth Ran
ran; our contract draft was wrong, not the code
jznmsl/siren/03_analysis_scripts/compute_ground_truth.py
code served (permissive licence) · get_code("c6409fc87ba95a34")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Set random seed for reproducibility
np.random.seed(42)
rng = np.random.RandomState(42)

# Create a small binary score matrix [M, K]
# M = 10 samples, K = 4 classes
scores = np.random.randint(0, 2, size=(10, 4)).astype(np.float32)

# rho: fraction of data to use for development set
rho = 0.6

# tau: temperature parameter for softmax
tau = 0.5

# n_mc: number of Monte Carlo iterations
n_mc = 5

# rng: numpy RandomState for reproducibility
rng = np.random.RandomState(42)

ARGS = {
    "scores": scores,
    "rho": rho,
    "tau": tau,
    "n_mc": n_mc,
    "rng": rng,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
compute_ground_truth Ran
ran, honoured the contract we drafted
jznmsl/siren/03_analysis_scripts/exp3_optimism.py
code served (permissive licence) · get_code("657c4fbeeb6dac34")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np
from scipy.special import expit

# Create a random number generator
rng = np.random.RandomState(42)

# Define arguments for compute_ground_truth
ARGS = {
    "q_0": 0.5,
    "diff_low": -1.0,
    "diff_high": 1.0,
    "N_gt": 100,
    "rng": rng
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, scipy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_answer Ran
ran; our contract draft was wrong, not the code
jznmsl/siren/03_analysis_scripts/study_e_multimodel_api.py
code served (permissive licence) · get_code("99767d2dbf24d8cf")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def extract_answer(text):
    if not text:
        return '?'
    if '</think>' in text:
        text = text.split('</think>')[-1]
    text = text.strip()
    if text and text[0] in 'ABCDEFGHIJ':
        rest = text[1:].strip()
        if not rest or rest[0] in '.\n,) ':
            return text[0]
    m = re.search('(?:answer|Answer|ANSWER)\\s*(?:is|:)\\s*([A-J])', text)
    if m:
        return m.group(1)
    matches = re.findall('\\b([A-J])\\b', text)
    return matches[-1] if matches else '?'

ARGS = {
    'text': 'The answer is: B'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_answer Ran
ran; our contract draft was wrong, not the code
jznmsl/siren/03_analysis_scripts/study_e_multimodel_law.py
code served (permissive licence) · get_code("16e4d0149abd853f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import argparse

def extract_answer(text):
    if not text:
        return '?'
    if '</think>' in text:
        text = text.split('</think>')[-1]
    text = text.strip()
    if text and text[0] in 'ABCDEFGHIJ':
        rest = text[1:].strip()
        if not rest or rest[0] in '.\n,) ':
            return text[0]
    m = re.search('(?:answer|Answer|ANSWER)\\s*(?:is|:)\\s*([A-J])', text)
    if m:
        return m.group(1)
    matches = re.findall('\\b([A-J])\\b', text)
    return matches[-1] if matches else '?'

import re

ARGS = {
    'text': 'The answer is: B'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_answer Ran
ran; our contract draft was wrong, not the code
jznmsl/siren/03_analysis_scripts/build_tensor.py
code served (permissive licence) · get_code("bec005b22dd07b60")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

VALID = set('ABCDEFGHIJ')

def extract_answer(text):
    if '</think>' in text:
        text = text.split('</think>')[-1]
    text = text.strip()
    if text and text[0] in VALID:
        return text[0]
    m = re.search('(?:answer|Answer|ANSWER)\\s*(?:is|:)\\s*([A-J])', text)
    if m:
        return m.group(1)
    matches = re.findall('\\b([A-J])\\b', text)
    if matches:
        return matches[-1]
    for ch in reversed(text):
        if ch in VALID:
            return ch
    return '?'


# Test cases for different code paths
test_cases = [
    "A is the answer",
    "</think>The answer is B",
    "The final choice: C",
    "Looking at options D and E, E is best",
    "No valid answer here",
    "F",
]

ARGS = {
    "text": "Answer: G"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_letter Ran
ran; our contract draft was wrong, not the code
jznmsl/siren/03_analysis_scripts/study_e_dspy_randsearch_law.py
code served (permissive licence) · get_code("ab86878635cab737")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def extract_letter(s):
    if not s:
        return '?'
    s = str(s).strip()
    if '</think>' in s:
        s = s.split('</think>')[-1].strip()
    if s and s[0] in 'ABCDEFGHIJ':
        rest = s[1:].strip()
        if not rest or rest[0] in '.\n,) ':
            return s[0]
    m = re.search('(?:answer|Answer|ANSWER)\\s*(?:is|:)\\s*([A-J])', s)
    if m:
        return m.group(1)
    matches = re.findall('\\b([A-J])\\b', s)
    return matches[-1] if matches else '?'

# Test case 1: Simple string with leading letter
test1 = "A. This is the answer"

# Test case 2: String with answer keyword
test2 = "The answer is B"

# Test case 3: String with think tags
test3 = "<think>some reasoning</think>C) correct"

# Test case 4: Empty string
test4 = ""

# Test case 5: String with multiple letters
test5 = "Between A and D, the answer is D"

# Test case 6: No valid letter
test6 = "No letter here"

# Test case 7: None input (will be converted to string)
test7 = None

# Test case 8: Answer with colon
test8 = "Answer: F"

ARGS = {
    "s": test1
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_letter Ran
ran; our contract draft was wrong, not the code
jznmsl/siren/03_analysis_scripts/study_e_miprov2_law.py
code served (permissive licence) · get_code("34e5242449c228b1")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def extract_letter(s):
    """Extract a single A-J letter from model output."""
    if not s:
        return '?'
    s = str(s).strip()
    if '</think>' in s:
        s = s.split('</think>')[-1].strip()
    if s and s[0] in 'ABCDEFGHIJ':
        rest = s[1:].strip()
        if not rest or rest[0] in '.\n,) ':
            return s[0]
    m = re.search('(?:answer|Answer|ANSWER)\\s*(?:is|:)\\s*([A-J])', s)
    if m:
        return m.group(1)
    matches = re.findall('\\b([A-J])\\b', s)
    return matches[-1] if matches else '?'


ARGS = {
    's': 'Answer is: B'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
fit_rasch Ran
ran; our contract draft was wrong, not the code
jznmsl/siren/03_analysis_scripts/study_h_prompteval_law.py
code served (permissive licence) · get_code("a76e46d8778190cc")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Create small test data for the Rasch model
# Y_full: binary response matrix (I items × J persons)
# mask: binary mask indicating which entries are observed
np.random.seed(0)
I, J = 5, 4  # 5 items, 4 persons
Y_full = np.random.randint(0, 2, size=(I, J))
mask = np.ones((I, J), dtype=bool)
# Mask out a few entries to test the masking logic
mask[0, 0] = False
mask[2, 3] = False

ARGS = {
    "Y_full": Y_full,
    "mask": mask,
    "n_iter": 10,
    "lr": 0.5,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, scipy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
format_options Ran
ran; our contract draft was wrong, not the code
jznmsl/siren/03_analysis_scripts/build_tensor.py
code served (permissive licence) · get_code("20ef5b27be91837f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Analyzing the function:
# - format_options takes a single parameter 'options'
# - It iterates over enumerate(options), so options must be iterable
# - Each element is used as a string in an f-string (opt)
# - The function returns a formatted string with letters A-J as prefixes

# Valid test case: a list of strings (option descriptions)
options = [
    "First option",
    "Second option",
    "Third option",
    "Fourth option",
    "Fifth option"
]

ARGS = {
    "options": options
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
format_options Ran
ran; our contract draft was wrong, not the code
jznmsl/siren/03_analysis_scripts/study_e_dspy_randsearch_law.py
code served (permissive licence) · get_code("7f9f073eb2c03fd6")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function takes 'options' as a parameter
# It iterates through enumerate(options) and formats each element
# So options needs to be an iterable of strings (or string-convertible objects)

options = [
    "Option A",
    "Option B", 
    "Option C",
    "Option D"
]

ARGS = {
    "options": options
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
format_question Ran
ran; our contract draft was wrong, not the code
jznmsl/siren/03_analysis_scripts/study_e_multimodel_api.py
code served (permissive licence) · get_code("7dc3245651ee6f5e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

ARGS = {
    'item': {
        'question': 'What is the capital of France?',
        'options': ['London', 'Paris', 'Berlin', 'Madrid']
    }
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
generate_scores Ran
ran; our contract draft was wrong, not the code
jznmsl/siren/03_analysis_scripts/exp1_coverage.py
code served (permissive licence) · get_code("42d8f6af2837c5f6")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 2, dtype family float, shape (5, 6) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Set random seed for reproducibility
rng = np.random.default_rng(seed=42)

# Define parameters for generate_scores
M = 5  # number of items
qualities = np.array([0.5, -0.3, 1.2, -0.8, 0.1, -1.5])  # K=6 quality values
diff_low = -2.0
diff_high = 2.0

ARGS = {
    "M": M,
    "qualities": qualities,
    "rng": rng,
    "diff_low": diff_low,
    "diff_high": diff_high,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, scipy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 2, dtype family float, shape (5, 6) — shape (5, 6) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
generate_scores Ran
ran; our contract draft was wrong, not the code
jznmsl/siren/03_analysis_scripts/exp2_near_tie.py
code served (permissive licence) · get_code("63018ddc189ff1cd")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 2, dtype family float, shape (2, 3) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Set random seed for reproducibility
np.random.seed(42)

# Create a random number generator
rng = np.random.default_rng(seed=42)

# Create qualities array: qualities should be a 1D array of quality scores
# Using shape (3,) to have a distinct size from M
qualities = np.array([0.5, -0.3, 1.2])

# M: number of items/difficulties
M = 2

# Optional parameters for difficulty range
diff_low = -2.0
diff_high = 2.0

ARGS = {
    "M": M,
    "qualities": qualities,
    "rng": rng,
    "diff_low": diff_low,
    "diff_high": diff_high,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, scipy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 2, dtype family float, shape (2, 3) — shape (2, 3) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
generate_scores Ran
ran; our contract draft was wrong, not the code
jznmsl/siren/03_analysis_scripts/exp3_optimism.py
code served (permissive licence) · get_code("9c912271312561f5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 2, dtype family float, shape (5, 3) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np
from scipy.special import expit

# Set random seed for reproducibility
np.random.seed(42)

# Create a random number generator
rng = np.random.RandomState(42)

# Define parameters
M = 5  # number of items
K = 3  # number of artifacts
q_0 = 1.5  # quality parameter
difficulties = np.array([0.2, 0.5, 1.0, 1.5, 2.0])  # M difficulties

ARGS = {
    "M": M,
    "K": K,
    "q_0": q_0,
    "rng": rng,
    "difficulties": difficulties,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, scipy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 2, dtype family float, shape (5, 3) — shape (5, 3) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
generate_splits Ran
ran; our contract draft was wrong, not the code
jznmsl/siren/03_analysis_scripts/exp2_near_tie.py
code served (permissive licence) · get_code("39d80562b160e690")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Set up random generator for reproducibility
rng = np.random.Generator(np.random.PCG64(seed=42))

# Define arguments for generate_splits
M = 10  # Total number of samples
R = 3   # Number of splits to create
rho = 0.7  # Proportion for dev split (70% dev, 30% eval)

ARGS = {
    "M": M,
    "R": R,
    "rho": rho,
    "rng": rng
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list — a list output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_true_answer Ran
ran; our contract draft was wrong, not the code
jznmsl/siren/03_analysis_scripts/build_tensor.py
code served (permissive licence) · get_code("25605f30a28c4300")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

# Analyze the function: get_true_answer takes one parameter 'item'
# The function expects 'item' to be a dict-like object with:
# - 'answer' key (can be int or string)
# - 'answer_index' key (can be int)

# Create test items to verify the function works
item1 = {
    'answer': 5,  # integer answer
    'answer_index': None
}

item2 = {
    'answer': 'B',  # string answer
    'answer_index': None
}

item3 = {
    'answer': '  c  ',  # string with whitespace
    'answer_index': None
}

item4 = {
    'answer_index': 3  # uses answer_index
}

item5 = {
    'answer': 'invalid_answer',  # returns as-is if not valid
    'answer_index': None
}

ARGS = {
    'item': item1
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_multimodel Ran
ran; our contract draft was wrong, not the code
jznmsl/siren/03_analysis_scripts/compare_models.py
code served (permissive licence) · get_code("0a1998c1fc760741")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
from pathlib import Path

# Create a temporary directory to serve as 'd'
temp_dir = tempfile.mkdtemp()
d = Path(temp_dir)

# Create the study_e_results.json file with valid data
results_data = {
    'model': 'test_model_v1',
    'results': {
        'batch_1': {
            'ground_truth': [0, 1, 0, 1],
            'predictions': [0.1, 0.9, 0.2, 0.8]
        },
        'batch_2': {
            'ground_truth': [1, 0, 1],
            'predictions': [0.95, 0.1, 0.85]
        },
        'batch_3': {
            'predictions': [0.5, 0.5]
        }
    }
}

# Write the JSON file
results_file = d / 'study_e_results.json'
with open(results_file, 'w') as fh:
    json.dump(results_data, fh)

ARGS = {
    'd': d
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
make_qualities Ran
ran, honoured the contract we drafted
jznmsl/siren/03_analysis_scripts/exp1_coverage.py
code served (permissive licence) · get_code("529cdb87aecf6052")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (5,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

ARGS = {
    "K": 5,
    "q_base": 0.5,
    "delta_q": 1.0,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (5,) — shape (5,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
prompteval_point Ran
ran; our contract draft was wrong, not the code
jznmsl/siren/03_analysis_scripts/study_h_prompteval_law.py
code served (permissive licence) · get_code("7b48b187da83fc0a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Create a small synthetic Y matrix (I=5 prompts, J=4 items)
# Y contains binary ratings in [0, 1]
np.random.seed(42)
I, J = 5, 4
Y = np.random.binomial(1, 0.6, size=(I, J)).astype(np.float32)

# budget_cells: number of cells to observe in the mask
# Should be less than I*J = 20
budget_cells = 8

# seed: random seed for reproducibility
seed = 0

ARGS = {
    "Y": Y,
    "budget_cells": budget_cells,
    "seed": seed,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, scipy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
softmax Ran
ran, violated the contract we drafted
jznmsl/siren/03_analysis_scripts/compute_ground_truth.py
code served (permissive licence) · get_code("d3064adf95e8f16e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Construct valid arguments for softmax function
# The function takes x (array-like) and tau (temperature parameter)
# and computes softmax with temperature scaling

# x should be a numpy array or array-like
x = np.array([1.0, 2.0, 3.0, 4.0])

# tau should be a positive scalar (temperature parameter)
tau = 1.0

ARGS = {
    "x": x,
    "tau": tau
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
softmax Ran
ran, violated the contract we drafted
jznmsl/siren/03_analysis_scripts/exp1_coverage.py
code served (permissive licence) · get_code("8b1d8ac976df7a27")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (5,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Create a small 1-D numpy array with K=5 elements
x = np.array([1.0, 2.0, 3.0, 0.5, 1.5])

# Temperature parameter
tau = 0.5

# Construct the arguments dictionary
ARGS = {
    "x": x,
    "tau": tau,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (5,) — shape (5,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
softmax Ran
ran, violated the contract we drafted
jznmsl/siren/03_analysis_scripts/exp2_near_tie.py
code served (permissive licence) · get_code("f7b74dbfee2cac12")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Create a small numpy array for the softmax function
x = np.array([1.0, 2.0, 3.0, 0.5])

# Create a temperature parameter
tau = 1.0

# Construct the arguments dictionary
ARGS = {
    "x": x,
    "tau": tau
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
softmax Ran
ran, violated the contract we drafted
jznmsl/siren/03_analysis_scripts/exp3_optimism.py
code served (permissive licence) · get_code("72c3fa12241ebe8f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Construct arguments for the softmax function
# The function takes:
# - x: a numpy array that will be divided, have max subtracted, exponentiated
# - tau: a temperature parameter (scalar) for scaling

# Create a small test array
x = np.array([1.0, 2.0, 3.0, 4.0])
tau = 1.0

ARGS = {
    "x": x,
    "tau": tau,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
two_way_balanced_sample Ran
ran, violated the contract we drafted
jznmsl/siren/03_analysis_scripts/study_h_prompteval_law.py
code served (permissive licence) · get_code("c5507d6d47ba2f2c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 2, dtype family float, shape (5, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Set random seed for reproducibility
np.random.seed(0)

# Create a random number generator
rng = np.random.RandomState(42)

# Define parameters for two_way_balanced_sample
I = 5  # number of rows
J = 4  # number of columns
budget_cells = 12  # budget for number of True cells (less than I*J=20)

# Construct ARGS dictionary
ARGS = {
    "I": I,
    "J": J,
    "budget_cells": budget_cells,
    "rng": rng
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 2, dtype family float, shape (5, 4) — shape (5, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block

Also present in other papers' code

Of 24 samples lifted from this paper's linked repositories, 24 carry a function identity and 0 do not. Of the 21 distinct functions with an identity, 0 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 0 later) and 21 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Adaptive prompt and program search makes LLM evaluation selection-sensitive. Once benchmark items are reused inside tuning, the observed winner's score need not estimate the fresh-data performance of the full tune-then-deploy procedure. We study inference for this procedure-level target under explicit tuning budgets. We propose SIREN, a selection-aware repeated-split reporting protocol that freezes the post-search shortlist, separates splitwise selection from held-out evaluation, and uses an item-level Gaussian multiplier bootstrap for uncertainty quantification. In a fixed-shortlist regime with smooth stabilized selection, the estimator admits a first-order item-level representation, and the bootstrap yields valid simultaneous inference on a finite budget grid. This supports confidence intervals for procedureperformance curves and pre-specified equal-budget and cross-budget comparisons. Controlled simulations and MMLU-Pro tuning experiments show that winnerbased reporting can be optimistic and can change deployment conclusions, while SIREN remains close to the finite-sample reporting target. Codes are available at https://github.com/jznmsl/siren.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2605.05973")
get_code_for_paper("2605.05973")
have("2605.05973")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2605.05973.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2605.05973.svg)](https://syntology.ai/paper/2605.05973)

Connect an agent — have() is free.