| aggregate_scores |
Ran ran on a synthesized input |
opendatalab/pm4bench/src/pm4bench/miqa.py code served (permissive licence) · get_code("cc5c4cade54886a7") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# Construct valid records for the aggregate_scores function
records = [
{
'scores': {
'Creativity': 8,
'Richness': 7,
'Visual Perception': 9,
'Logical Coherence': 8,
'Answer Accuracy': 9,
'Image Relationship Understanding': 7
}
},
{
'scores': {
'Creativity': 6,
'Richness': 8,
'Visual Perception': 7,
'Logical Coherence': 9,
'Answer Accuracy': 8,
'Image Relationship Understanding': 8
}
},
{
'scores': {
'Creativity': 9,
'Richness': 6,
'Visual Perception': 8,
'Logical Coherence': 7,
'Answer Accuracy': 7,
'Image Relationship Understanding': 9
}
}
]
ARGS = {
'records': records
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| compare_gt |
Ran ran on a synthesized input |
opendatalab/pm4bench/src/pm4bench/mgui/render.py code served (permissive licence) · get_code("40019c900986263c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
from pathlib import Path
# Create temporary directories for candidate and reference roots
candidate_root = Path(tempfile.mkdtemp())
reference_root = Path(tempfile.mkdtemp())
# Create sample JSON files for comparison
candidate_file = candidate_root / "test.json"
reference_file = reference_root / "test.json"
candidate_data = {
"template": "template_1",
"template_title": "Title 1",
"lang": "en",
"image": "image.png",
"width": 800,
"height": 600,
"all_qid_boxes": {
"Q1": {
"x": 10.0,
"y": 20.0,
"width": 50.0,
"height": 30.0,
"text": "<built-in method copy of dict object at 0x12345678>"
},
"Q2": {
"x": 100.0,
"y": 150.0,
"width": 60.0,
"height": 40.0,
"text": "Sample text"
}
},
"questions": [
{
"qid_in_template": "Q1",
"difficulty": "easy",
"target_qid": "Q1",
"question": "What is this?",
"target_text_sample": "Text sample 1"
},
{
"qid_in_template": "Q2",
"difficulty": "hard",
"target_qid": "Q2",
"question": "What is that?",
"target_text_sample": "Text sample 2"
}
]
}
reference_data = {
"template": "template_1",
"template_title": "Title 1",
"lang": "en",
"image": "image.png",
"width": 800,
"height": 600,
"all_qid_boxes": {
"Q1": {
"x": 10.5,
"y": 20.1,
"width": 50.0,
"height": 30.0,
"text": "<built-in method copy of dict object at 0x87654321>"
},
"Q2": {
"x": 100.0,
"y": 150.0,
"width": 60.0,
"height": 40.0,
"text": "Sample text"
}
},
"questions": [
{
"qid_in_template": "Q1",
"difficulty": "easy",
"target_qid": "Q1",
"question": "What is this?",
"target_text_sample": "Text sample 1"
},
{
"qid_in_template": "Q2",
"difficulty": "hard",
"target_qid": "Q2",
"question": "What is that?",
"target_text_sample": "Text sample 2"
}
]
}
# Write JSON files
with candidate_file.open('w', encoding='utf-8') as f:
json.dump(candidate_data, f)
with reference_file.open('w', encoding='utf-8') as f:
json.dump(reference_data, f)
ARGS = {
"candidate_root": candidate_root,
"reference_root": reference_root,
"tolerance": 0.2,
"compare_geometry": True
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_answer_letter |
Ran ran on a synthesized input |
opendatalab/pm4bench/src/pm4bench/metrics.py code served (permissive licence) · get_code("e929aabca68bb917") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def extract_answer_letter(value: str) -> str | None:
text = re.sub('<think>.*?</think>', '', value or '', flags=re.DOTALL)
explicit = re.findall('(?:answer|option)\\s*[::]?\\s*\\(?([A-J])\\)?', text, re.IGNORECASE)
if explicit:
return explicit[-1].upper()
boxed = re.findall('\\\\boxed\\{\\s*([A-J])\\s*\\}', text, re.IGNORECASE)
if boxed:
return boxed[-1].upper()
standalone = re.findall('(?<![A-Za-z])([A-J])(?![A-Za-z])', text, re.IGNORECASE)
return standalone[-1].upper() if standalone else None
# Test case 1: explicit answer format
test_value_1 = "The answer is: B"
# Test case 2: boxed format
test_value_2 = "The final answer is \\boxed{C}"
# Test case 3: with think tags
test_value_3 = "<think>working through this</think>Answer: D"
# Test case 4: standalone letter
test_value_4 = "The result is E"
# Test case 5: None/empty string
test_value_5 = None
ARGS = {
"value": test_value_1
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_scores |
Ran ran on a synthesized input |
opendatalab/pm4bench/src/pm4bench/miqa.py code served (permissive licence) · get_code("6ec5c1760b10f4d4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import ast
import re
DIMENSIONS = ('Creativity', 'Richness', 'Visual Perception', 'Logical Coherence', 'Answer Accuracy', 'Image Relationship Understanding')
def _normalize_scores(value: dict, ) -> dict | None:
canonical = {dimension.lower(): dimension for dimension in DIMENSIONS + ('Overall Score',)}
result = {}
for key, score in value.items():
name = canonical.get(str(key).strip().lower())
if not name:
continue
try:
number = int(score)
except (TypeError, ValueError):
continue
if 1 <= number <= 10:
result[name] = number
return result if all((dimension in result for dimension in DIMENSIONS)) else None
def extract_scores(text: str) -> dict | None:
if not text:
return None
candidates = ['{' + match + '}' for match in re.findall('\\{\\{(.*?)\\}\\}', text, re.DOTALL)]
candidates.extend(re.findall('\\{[^{}]*\\}', text, re.DOTALL))
for candidate in reversed(candidates):
try:
value = ast.literal_eval(candidate)
except (SyntaxError, ValueError):
continue
if isinstance(value, dict):
normalized = _normalize_scores(value)
if normalized:
return normalized
pairs = re.findall('[\'\\"]?([A-Za-z][A-Za-z ]+?)[\'\\"]?\\s*[::]\\s*(\\d{1,2})', text)
return _normalize_scores(dict(pairs))
# Test case 1: Text with dictionary in double braces
text1 = """
{{
'Creativity': 8,
'Richness': 7,
'Visual Perception': 9,
'Logical Coherence': 8,
'Answer Accuracy': 9,
'Image Relationship Understanding': 8
}}
"""
# Test case 2: Text with key-value pairs in colon format
text2 = """
Creativity: 7
Richness: 8
Visual Perception: 6
Logical Coherence: 9
Answer Accuracy: 8
Image Relationship Understanding: 7
"""
# Test case 3: Text with dictionary in single braces
text3 = """{
"Creativity": 5,
"Richness": 6,
"Visual Perception": 7,
"Logical Coherence": 8,
"Answer Accuracy": 9,
"Image Relationship Understanding": 10
}"""
# Test case 4: Empty string
text4 = ""
# Test case 5: Text with Chinese colon (:)
text5 = """
Creativity:9
Richness:8
Visual Perception:7
Logical Coherence:9
Answer Accuracy:8
Image Relationship Understanding:9
"""
ARGS = {
"text": text1
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| format_reward |
Ran ran on a synthesized input |
opendatalab/pm4bench/src/pm4bench/qgo/reward.py code served (permissive licence) · get_code("c17d2901ecfbf4cb") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import math
REPETITION_THRESHOLD = 0.6
REPETITION_PENALTY_WEIGHT = 0.4
LENGTH_PENALTY_WEIGHT = 0.8
LENGTH_REWARD_WEIGHT = 0.2
def format_reward(solution: str, minimum_length: int=1000, maximum_length: int=10000) -> dict[str, float]:
length = len(solution)
if '[Internal Processing Omitted]' in solution:
return {'combined': -1.2, 'length': 0.0, 'length_penalty': 0.0, 'repetition': 0.0}
check_text = solution[-3000:] if length > 5000 else solution
ngrams = [check_text[index:index + 4] for index in range(max(0, len(check_text) - 3))]
diversity = len(set(ngrams)) / len(ngrams) if ngrams else 1.0
repetition = max(0.0, 1.0 - diversity / REPETITION_THRESHOLD)
length_reward = float(minimum_length <= length <= maximum_length)
if length < minimum_length:
length_penalty = 1.0 - length / minimum_length
elif length > maximum_length:
length_penalty = min(1.0, math.log1p((length - maximum_length) / 1200.0))
else:
length_penalty = 0.0
combined = length_reward * LENGTH_REWARD_WEIGHT - length_penalty * LENGTH_PENALTY_WEIGHT - repetition * REPETITION_PENALTY_WEIGHT
return {'combined': combined, 'length': length_reward, 'length_penalty': length_penalty, 'repetition': repetition}
# Create a solution string within the valid length range
solution_text = "This is a test solution. " * 100
ARGS = {
'solution': solution_text,
'minimum_length': 1000,
'maximum_length': 10000
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_predictions |
Ran ran on a synthesized input |
opendatalab/pm4bench/src/pm4bench/io.py code served (permissive licence) · get_code("e6b5d81f514b3ebb") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import json
from pathlib import Path
# Create a temporary JSONL file with valid records
with tempfile.NamedTemporaryFile(mode='w', suffix='.jsonl', delete=False, encoding='utf-8') as f:
temp_path = f.name
# Write valid records with 'id' field
f.write(json.dumps({'id': '1', 'prediction': 'value1'}) + '\n')
f.write(json.dumps({'id': '2', 'prediction': 'value2'}) + '\n')
# Write a record with 'index' field
f.write(json.dumps({'index': 3, 'prediction': 'value3'}) + '\n')
# Include an empty line to test line skipping
f.write('\n')
f.write(json.dumps({'id': '4', 'prediction': 'value4'}) + '\n')
ARGS = {
'path': temp_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| mdur_correct |
Ran ran on a synthesized input |
opendatalab/pm4bench/src/pm4bench/metrics.py code served (permissive licence) · get_code("cd1307d13954b765") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def extract_answer_letter(value: str) -> str | None:
text = re.sub('<think>.*?</think>', '', value or '', flags=re.DOTALL)
explicit = re.findall('(?:answer|option)\\s*[::]?\\s*\\(?([A-J])\\)?', text, re.IGNORECASE)
if explicit:
return explicit[-1].upper()
boxed = re.findall('\\\\boxed\\{\\s*([A-J])\\s*\\}', text, re.IGNORECASE)
if boxed:
return boxed[-1].upper()
standalone = re.findall('(?<![A-Za-z])([A-J])(?![A-Za-z])', text, re.IGNORECASE)
return standalone[-1].upper() if standalone else None
def mdur_correct(response: str, answer_key: str) -> bool:
return extract_answer_letter(response) == answer_key.strip().upper()
ARGS = {
"response": "The answer is (B). Let me think about this problem.",
"answer_key": "b"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| normalize_historical_text |
Ran ran on a synthesized input |
opendatalab/pm4bench/src/pm4bench/mgui/render.py code served (permissive licence) · get_code("3ad8b96dec73d6f2") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
HISTORICAL_METHOD_ADDRESS = re.compile('(<built-in method (?:copy|update) of dict object at )0x[0-9a-fA-F]+(>)')
def normalize_historical_text(value: str) -> str:
"""Ignore only nondeterministic addresses emitted by two legacy templates."""
return HISTORICAL_METHOD_ADDRESS.sub('\\g<1>0x<address>\\2', value)
# Test input with a historical method address pattern
test_value = "<built-in method copy of dict object at 0x7f1a2b3c4d5e>"
ARGS = {
"value": test_value
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| normalize_text |
Ran ran on a synthesized input |
opendatalab/pm4bench/src/pm4bench/metrics.py code served (permissive licence) · get_code("abd00a80d4ff626d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
import unicodedata
def normalize_text(value: str) -> str:
return re.sub('\\s+', '', unicodedata.normalize('NFKC', value or ''))
# The function takes a single parameter: value (a string)
# It normalizes the string using NFKC normalization and removes all whitespace
# We need to provide a valid string argument
ARGS = {
'value': ' Hello World \n\t Test '
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| record_key |
Ran ran on a synthesized input |
opendatalab/pm4bench/src/pm4bench/io.py code served (permissive licence) · get_code("cbd1256114d18a72") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
# The function record_key takes a dict with optional 'id' or 'index' keys
# and returns a string representation of whichever is present (id takes priority)
# Create a valid record dict with an 'id' field
record_with_id = {
'id': 42,
'name': 'test_record',
'data': [1, 2, 3]
}
ARGS = {
'record': record_with_id
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| resolve_asset |
Ran ran on a synthesized input |
opendatalab/pm4bench/src/pm4bench/io.py code served (permissive licence) · get_code("791bca2fe997c4f0") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned PosixPath. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from pathlib import Path
import tempfile
import os
# Create a temporary directory to serve as the dataset root
temp_dir = tempfile.mkdtemp()
dataset_root = temp_dir
# Create a subdirectory structure for testing
subdir = os.path.join(temp_dir, "assets")
os.makedirs(subdir, exist_ok=True)
# Create a test file within the root
test_file_path = os.path.join(subdir, "test_asset.txt")
with open(test_file_path, "w") as f:
f.write("test")
# Define relative path that stays within the root
relative_path = "assets/test_asset.txt"
ARGS = {
"dataset_root": dataset_root,
"relative_path": relative_path,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type PosixPath — a PosixPath output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a PosixPath output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| render_mgui |
Not yet run unverified (harvested, no recorded run) |
opendatalab/pm4bench/src/pm4bench/mgui/render.py code served (permissive licence) · get_code("9a98d066a6e55e2e") How it ran |