SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2406.11939 · 2024

From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline

arXiv · PDF · Open in the Atlas

Code that ran

19 of 21 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 1 honoured the contract we drafted, 0 violated it and 1 has no contract verdict; 17 failures of our instrument; 2 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2406.11939.json · badge: /api/ran/2406.11939.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
lm-sys/arena-hard-auto canonical (classifier label)
classifier basis: census
4 ran · 0 no recorded run
lm-sys/arena-hard community (classifier label)
classifier basis: plan_row
15 ran · 2 no recorded run
FunctionStatusWhere it lives
calculate_score Ran
ran, honoured the contract we drafted
lm-sys/arena-hard/BenchBuilder/filter.py
code served (permissive licence) · get_code("06750b8d972efe27")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Analyzing the function:
# - It takes a conversation dict
# - Accesses conversation.get('category_tag', {}).get('criteria_v0.1', {})
# - Iterates over criteria.values() and counts truthy values
# - Returns an integer sum

# Build a valid conversation dict with the expected structure
conversation = {
    'category_tag': {
        'criteria_v0.1': {
            'criterion_1': True,
            'criterion_2': False,
            'criterion_3': True,
            'criterion_4': True,
            'criterion_5': False,
        }
    }
}

ARGS = {
    'conversation': conversation
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
count_markdown_elements Ran
ran; our contract draft was wrong, not the code
lm-sys/arena-hard/utils/add_markdown_info.py
code served (permissive licence) · get_code("df075cc813246dcd")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

# Construct valid arguments for count_markdown_elements
markdown_text = """# Header 1
## Header 2
### Header 3
#### Header 4
##### Header 5
###### Header 6

1. First item
2. Second item
3. Third item

- Bullet point
* Another bullet
+ Plus bullet

**bold text**
**more bold**
__underscored bold__
__another underscored__
"""

suffix = "_test"

ARGS = {
    "markdown_text": markdown_text,
    "suffix": suffix
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
create_llama3_body Ran
ran; our contract draft was wrong, not the code
lm-sys/arena-hard/utils/bedrock_utils.py
code served (permissive licence) · get_code("da15e93460148f70")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json

# Construct valid arguments for create_llama3_body
ARGS = {
    'messages': [
        {
            'content': ['Hello, how are you?', 'Tell me a story.']
        },
        {
            'content': ['I am doing well.']
        }
    ],
    'max_gen_len': 2048,
    'temperature': 0.7,
    'top_p': 0.9,
    'top_k': 50
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
create_nova_messages Ran
ran; our contract draft was wrong, not the code
lm-sys/arena-hard/utils/bedrock_utils.py
code served (permissive licence) · get_code("6743a5e05fc4ad6d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Define test messages following the expected structure
messages = [
    {
        'role': 'user',
        'content': 'Hello, how are you?'
    },
    {
        'role': 'assistant',
        'content': {
            'text': 'I am doing well, thank you for asking.'
        }
    },
    {
        'role': 'user',
        'content': [
            {
                'text': 'What is your name?'
            }
        ]
    },
    {
        'role': 'assistant',
        'content': [
            [
                {
                    'text': 'My name is Nova.'
                }
            ]
        ]
    }
]

ARGS = {
    'messages': messages
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_innermost_text Ran
ran; our contract draft was wrong, not the code
lm-sys/arena-hard/utils/bedrock_utils.py
code served (permissive licence) · get_code("759c3030a0ca985f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Test case 1: Simple string
content_1 = "hello world"

# Test case 2: Dict with text key containing a string
content_2 = {"text": "nested text"}

# Test case 3: Dict with text key containing another dict
content_3 = {"text": {"text": "deeply nested"}}

# Test case 4: List containing a string
content_4 = ["first item"]

# Test case 5: List containing a dict with text
content_5 = [{"text": "list with dict"}]

# Test case 6: Deeply nested structure with list and dict
content_6 = [{"text": [{"text": "very deep"}]}]

# Test case 7: Empty list (should return empty string)
content_7 = []

# Test case 8: Dict without text key (should return empty string)
content_8 = {"other": "value"}

# Test case 9: Empty string
content_9 = ""

# Test case 10: Complex nested structure
content_10 = [[[{"text": "complex"}]]]

ARGS = {
    "content": content_6
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
format_confidence_interval Ran
ran; our contract draft was wrong, not the code
lm-sys/arena-hard/show_result.py
code served (permissive licence) · get_code("bf60419bbfec3311")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd
import numpy as np

# Create sample DataFrames with model scores and confidence intervals
mean_scores = pd.DataFrame({
    'model': ['ModelA', 'ModelB', 'ModelC'],
    'scores': [0.85, 0.72, 0.91]
})

lower_scores = pd.DataFrame({
    'model': ['ModelA', 'ModelB', 'ModelC'],
    'lower': [0.82, 0.68, 0.88]
})

upper_scores = pd.DataFrame({
    'model': ['ModelA', 'ModelB', 'ModelC'],
    'upper': [0.88, 0.76, 0.94]
})

baseline = 'Baseline'

ARGS = {
    'mean_scores': mean_scores,
    'lower_scores': lower_scores,
    'upper_scores': upper_scores,
    'baseline': baseline
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_categories Ran
ran; our contract draft was wrong, not the code
lm-sys/arena-hard/qa_browser.py
code served (permissive licence) · get_code("31d3137625992d50")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import pandas as pd
import tempfile
import json

# Create temporary directory and test data
temp_dir = tempfile.mkdtemp()
data_dir = os.path.join(temp_dir, 'data')
os.makedirs(data_dir, exist_ok=True)

# Create a test benchmark directory
benchmark_name = 'test_benchmark'
benchmark_dir = os.path.join(data_dir, benchmark_name)
os.makedirs(benchmark_dir, exist_ok=True)

# Create a sample question.jsonl file with categories
question_file = os.path.join(benchmark_dir, 'question.jsonl')
questions_data = [
    {'id': 1, 'category': 'math', 'question': 'What is 2+2?'},
    {'id': 2, 'category': 'science', 'question': 'What is gravity?'},
    {'id': 3, 'category': 'math', 'question': 'What is 3*3?'},
    {'id': 4, 'category': 'history', 'question': 'When was WWII?'},
]

with open(question_file, 'w') as f:
    for item in questions_data:
        f.write(json.dumps(item) + '\n')

# Change to temp directory so relative path 'data/...' works
original_cwd = os.getcwd()
os.chdir(temp_dir)

ARGS = {
    'benchmark': 'test_benchmark'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_element_counts Ran
ran; our contract draft was wrong, not the code
lm-sys/arena-hard/utils/add_markdown_info.py
code served (permissive licence) · get_code("0a127509b23580f5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd
import re

# Create a minimal DataFrame with the expected structure
# The function expects df[column] to return a Series where each element
# is a list of message dicts with a specific structure

# Build sample messages matching the expected structure
message1 = {
    'content': {
        'answer': """# Header 1
## Header 2
### Header 3

This is **bold text** and __underlined__.

- Item 1
- Item 2
1. First
2. Second


code block

"""
    }
}

message2 = {
    'content': {
        'answer': """#### Header 4
##### Header 5
###### Header 6

**Another bold** and __more underline__.

* Bullet point
+ Plus point
1. Numbered item

javascript
var x = 1;

"""
    }
}

# Create DataFrame with the structure expected by get_element_counts
df = pd.DataFrame({
    'messages': [
        [message1],
        [message2]
    ]
})

ARGS = {
    'df': df,
    'column': 'messages'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_endpoint Ran
ran; our contract draft was wrong, not the code
lm-sys/arena-hard-auto/BenchBuilder/label.py
code served (permissive licence) · get_code("c8136157d71c093b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import random

# The function takes endpoint_list as a parameter
# It can be None or a list of dictionaries (api_dict objects)
# The function calls random.choices(endpoint_list)[0] to select one dict from the list

endpoint_list = [
    {"api_key": "key1", "url": "https://api1.example.com"},
    {"api_key": "key2", "url": "https://api2.example.com"},
    {"api_key": "key3", "url": "https://api3.example.com"}
]

ARGS = {
    "endpoint_list": endpoint_list
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_languages Ran
ran; our contract draft was wrong, not the code
lm-sys/arena-hard/qa_browser.py
code served (permissive licence) · get_code("7ca039993ebdcc43")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import pandas as pd
import tempfile
import json

# Create a temporary directory and mock data file
temp_dir = tempfile.mkdtemp()
benchmark_dir = os.path.join(temp_dir, 'data', 'test_benchmark')
os.makedirs(benchmark_dir, exist_ok=True)

# Create a mock question.jsonl file with language column
question_file = os.path.join(benchmark_dir, 'question.jsonl')
with open(question_file, 'w') as f:
    f.write(json.dumps({'language': 'English', 'question': 'What is 2+2?'}) + '\n')
    f.write(json.dumps({'language': 'Spanish', 'question': '¿Cuánto es 2+2?'}) + '\n')
    f.write(json.dumps({'language': 'French', 'question': 'Qu\'est-ce que 2+2?'}) + '\n')

# Update the data directory to point to our temp directory
original_cwd = os.getcwd()
os.chdir(temp_dir)

ARGS = {
    'benchmark': 'test_benchmark'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_model_style_metadata Ran
ran; our contract draft was wrong, not the code
lm-sys/arena-hard/show_result.py
code served (permissive licence) · get_code("b1fbdabfc009df1c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd
import os
import tempfile
import json
from pathlib import Path

# Create a temporary directory structure with test data
temp_dir = tempfile.mkdtemp()
benchmark_name = "test_benchmark"
benchmark_path = os.path.join(temp_dir, "data", benchmark_name, "model_answer")
os.makedirs(benchmark_path, exist_ok=True)

# Create sample JSONL files with model answers and metadata
sample_data_1 = [
    {
        "uid": "uid_1",
        "model": "model_a",
        "metadata": {"style": "formal", "length": "short"}
    },
    {
        "uid": "uid_2", 
        "model": "model_a",
        "metadata": {"style": "informal", "length": "long"}
    }
]

sample_data_2 = [
    {
        "uid": "uid_3",
        "model": "model_b",
        "metadata": {"style": "casual", "length": "medium"}
    },
    {
        "uid": "uid_4",
        "model": "model_b",
        "metadata": {"style": "technical", "length": "short"}
    }
]

# Write JSONL files
with open(os.path.join(benchmark_path, "model_a.jsonl"), "w") as f:
    for item in sample_data_1:
        f.write(json.dumps(item) + "\n")

with open(os.path.join(benchmark_path, "model_b.jsonl"), "w") as f:
    for item in sample_data_2:
        f.write(json.dumps(item) + "\n")

# Change to temp directory so relative paths work
original_cwd = os.getcwd()
os.chdir(temp_dir)

ARGS = {
    "benchmark": benchmark_name
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_json Ran
ran; our contract draft was wrong, not the code
lm-sys/arena-hard-auto/BenchBuilder/filter.py
code served (permissive licence) · get_code("6805dfa84dd06130")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import orjson
import tempfile
import os

# Create a temporary file with JSON content
temp_dir = tempfile.mkdtemp()
temp_file = os.path.join(temp_dir, "test.json")

# Write test data to the temporary file
test_data = [
    {"id": 1, "name": "test1", "value": 42},
    {"id": 2, "name": "test2", "value": 100}
]

with open(temp_file, 'wb') as f:
    f.write(orjson.dumps(test_data))

ARGS = {
    "file_path": temp_file
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
orjson — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_jsonl Ran
ran; our contract draft was wrong, not the code
lm-sys/arena-hard/qa_browser.py
code served (permissive licence) · get_code("d049c92b5f5adfa3")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import pandas as pd
import tempfile
import json

# Create a temporary JSONL file for testing
temp_dir = tempfile.mkdtemp()
temp_jsonl_file = os.path.join(temp_dir, "test_data.jsonl")

# Write some test data to the temporary JSONL file
with open(temp_jsonl_file, 'w') as f:
    f.write(json.dumps({"id": 1, "text": "hello"}) + "\n")
    f.write(json.dumps({"id": 2, "text": "world"}) + "\n")

# Define arguments for load_jsonl function
ARGS = {
    "file_path": temp_jsonl_file
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_jsonl Ran
ran on a synthesized input
lm-sys/arena-hard/BenchBuilder/filter.py
code served (permissive licence) · get_code("e094bc160bb7d823")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import tempfile
import orjson

# Create a temporary JSONL file with test data
temp_dir = tempfile.mkdtemp()
file_path = f"{temp_dir}/test.jsonl"

# Write sample JSONL data
test_data = [
    {"id": 1, "message": "Hello", "role": "user"},
    {"id": 2, "message": "Hi there", "role": "assistant"},
    {"id": 3, "message": "How are you?", "role": "user"},
]

with open(file_path, 'wb') as f:
    for item in test_data:
        f.write(orjson.dumps(item) + b'\n')

ARGS = {
    "file_path": file_path,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
orjson — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_questions Ran
ran; our contract draft was wrong, not the code
lm-sys/arena-hard-auto/utils/completion.py
code served (permissive licence) · get_code("47892c5631a199d4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
import os

# Create a temporary file with JSONL content
temp_file = tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.jsonl')
temp_filename = temp_file.name

# Write sample question data in JSONL format
questions_data = [
    {"id": 1, "question": "What is 2+2?", "answer": "4"},
    {"id": 2, "question": "What is the capital of France?", "answer": "Paris"},
    {"id": 3, "question": "What is 5*3?", "answer": "15"}
]

for q in questions_data:
    temp_file.write(json.dumps(q) + '\n')

temp_file.close()

ARGS = {
    "question_file": temp_filename
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
make_config Ran
ran; our contract draft was wrong, not the code
lm-sys/arena-hard-auto/BenchBuilder/label.py
pointer only (licence: Apache-2.0); the stored copy get_code serves, from wwn1233/sedareval, is pointer only (licence: NONE) · get_code("07e6f41cf857a76f")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (Apache-2.0), but get_code serves the stored copy, first harvested from wwn1233/sedareval, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

register_api Ran
ran; our contract draft was wrong, not the code
lm-sys/arena-hard/utils/completion.py
code served (permissive licence) · get_code("7a5ee66f6fe09b60")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned function. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# The register_api function takes one parameter: api_type
# It's a string that serves as a key in the registered_api_completion dict
# The function returns a decorator, so we need to show how it would be used

# Setup: api_type should be a string identifier
api_type = "completion_v1"

ARGS = {
    "api_type": api_type
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type function — a function output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a function output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
register_engine Ran
ran; our contract draft was wrong, not the code
lm-sys/arena-hard/utils/completion.py
code served (permissive licence) · get_code("0399a259308f77a0")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned function. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# The register_engine function takes one parameter: engine_type
# It's a string that serves as a key in the registered_engine_completion dict
# The function returns a decorator, so we need to provide a string value for engine_type

engine_type_value = "completion_engine_v1"

ARGS = {
    "engine_type": engine_type_value
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type function — a function output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a function output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
remove_pattern Ran
ran; our contract draft was wrong, not the code
lm-sys/arena-hard/utils/add_markdown_info.py
code served (permissive licence) · get_code("bae70f745e7b2d09")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

# The function uses:
# - answer: a string to process
# - pattern: an object with a findall(string) method that returns a list of strings
# This matches the re.Pattern type from the re module

answer = "The quick brown fox jumps over the lazy dog. The fox is clever."
pattern = re.compile(r"The \w+")

ARGS = {
    "answer": answer,
    "pattern": pattern,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
chat_completion_openai Not yet run
unverified (harvested, no recorded run)
lm-sys/arena-hard/BenchBuilder/label.py
code served (permissive licence) · get_code("0322093a9ad847a4")
How it ran

no invocation recorded.

load_judgments Not yet run
unverified (harvested, no recorded run)
lm-sys/arena-hard/show_result.py
code served (permissive licence) · get_code("ff96063cc3cfa7d6")
How it ran

no invocation recorded.

Also present in other papers' code

Of 21 samples lifted from this paper's linked repositories, 21 carry a function identity and 0 do not. Of the 21 distinct functions with an identity, 19 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 19 later) and 2 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
make_config 3 other papers
3 later
2410.14872 (2024-10)
How to Evaluate Reward Models for RLHF
2501.15595 (2025-01)
SedarEval: Automated Evaluation using Self-Adaptive Rubrics
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
get_endpoint 2 other papers
2 later
2409.15254 (2024-09)
An Architecture Search Framework for Inference-Time Techniques
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
load_questions 2 other papers
2 later
2409.15254 (2024-09)
An Architecture Search Framework for Inference-Time Techniques
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
calculate_score 1 other paper
1 later
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
chat_completion_openai 1 other paper
1 later
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
count_markdown_elements 1 other paper
1 later
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
create_llama3_body 1 other paper
1 later
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
create_nova_messages 1 other paper
1 later
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
extract_innermost_text 1 other paper
1 later
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
format_confidence_interval 1 other paper
1 later
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
get_categories 1 other paper
1 later
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
get_element_counts 1 other paper
1 later
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
get_languages 1 other paper
1 later
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
get_model_style_metadata 1 other paper
1 later
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
load_jsonl 1 other paper
1 later
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
load_judgments 1 other paper
1 later
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
register_api 1 other paper
1 later
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
register_engine 1 other paper
1 later
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output
remove_pattern 1 other paper
1 later
2606.10528 (2026-06)
Representation-Aware Advantage Estimation: Your Reward Model Provides More Than A Scalar Output

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

The rapid evolution of Large Language Models (LLMs) has outpaced the development of model evaluation, highlighting the need for continuous curation of new, challenging benchmarks. However, manual curation of high-quality, human-aligned benchmarks is expensive and time-consuming. To address this, we introduce BenchBuilder, an automated pipeline that leverages LLMs to curate high-quality, open-ended prompts from large, crowd-sourced datasets, enabling continuous benchmark updates without human in the loop. We apply BenchBuilder to datasets such as Chatbot Arena and WildChat-1M, extracting challenging prompts and utilizing LLM-as-a-Judge for automatic model evaluation. To validate benchmark quality, we propose new metrics to measure a benchmark's alignment with human preferences and ability to separate models. We release Arena-Hard-Auto, a benchmark consisting 500 challenging prompts curated by BenchBuilder. Arena-Hard-Auto provides 3x higher separation of model performances compared to MT-Bench and achieves 98.6% correlation with human preference rankings, all at a cost of $20. Our work sets a new framework for the scalable curation of automated benchmarks from extensive data.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2406.11939")
get_code_for_paper("2406.11939")
have("2406.11939")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2406.11939.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2406.11939.svg)](https://syntology.ai/paper/2406.11939)

Connect an agent — have() is free.