| DocQARecord |
Ran ran on a synthesized input · constructed an object; no result was computed |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("58cd26d69bca691d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, DocQARecord: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional
torch.manual_seed(0)
@dataclass
class DocQARecord:
plan_id: str
doc_id: str
attempts: List[Dict[str, Any]] = field(default_factory=list)
ARGS = {
"plan_id": "plan_12345",
"doc_id": "doc_67890",
"attempts": [
{"query": "What is this?", "answer": "This is a test", "score": 0.95},
{"query": "Where is it?", "answer": "It is here", "score": 0.87}
]
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DocQARecord — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a DocQARecord output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| ExtractorRegistry |
Ran ran on a synthesized input |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("39eaaef56c224774") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned NoneType. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from typing import Any, Dict, List, Optional, Protocol
import importlib
from typing import Any, Callable, Dict, Optional
SingleDocExtractor = Callable[[str, str], str]
class ExtractorRegistry:
"""
Registry for pluggable single-document extraction backends.
Core contract:
- input: document_id (str), prompt (str)
- output: extracted answer text (str)
This makes `ask_bchatdoc.py` replaceable by any single-doc extractor system.
"""
_registry: Dict[str, SingleDocExtractor] = {}
@classmethod
def register(cls, name: str, func: SingleDocExtractor) -> None:
cls._registry[name] = func
@classmethod
def get(cls, name: str) -> SingleDocExtractor:
if name not in cls._registry:
raise KeyError(f'unknown extractor backend: {name}')
return cls._registry[name]
@classmethod
def load_from_entrypoint(cls, spec: str) -> SingleDocExtractor:
"""
`spec` format: module.submodule:function_name
Example: my_backend.chat:extract_single_doc
"""
if ':' not in spec:
raise ValueError('entrypoint must be module_path:function_name')
module_path, func_name = spec.split(':', 1)
module = importlib.import_module(module_path)
func = getattr(module, func_name)
if not callable(func):
raise TypeError(f'entrypoint {spec} is not callable')
return func
# Setup code
INIT_ARGS = {}
ARGS = {
'name': 'test_extractor',
'func': lambda doc_id, prompt: f'extracted from {doc_id}: {prompt[:20]}'
}
METHOD = 'register'
# Construct and call
registry = ExtractorRegistry(**INIT_ARGS)
registry.register(**ARGS)
# Verify registration
retrieved = registry.get('test_extractor')
result = retrieved('doc_123', 'What is the meaning of life?')
print(f"Result: {result}")
# Test get method
ARGS_GET = {'name': 'test_extractor'}
METHOD_GET = 'get'
extractor = registry.get(**ARGS_GET)
print(f"Retrieved extractor: {extractor}")
# Test load_from_entrypoint with a valid spec
ARGS_LOAD = {'spec': 'os.path:join'}
METHOD_LOAD = 'load_from_entrypoint'
loaded_func = registry.load_from_entrypoint(**ARGS_LOAD)
print(f"Loaded function: {loaded_func}")
- Setup block drafted by
- a language model, class construct-then-call prompt, model not recorded
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type NoneType — a NoneType output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a NoneType output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
|
| LLMResponse |
Ran ran on a synthesized input · constructed an object; no result was computed |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("814a75b8ec1ca439") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, LLMResponse: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
ARGS = {
"text": "This is a test response from the language model."
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type LLMResponse — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a LLMResponse output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| PromptTemplate |
Ran ran on a synthesized input · constructed an object; no result was computed |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("7083ff639bbbaa35") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, PromptTemplate: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
from typing import Any, Dict, List, Optional, Protocol
_PLACEHOLDER_RE = re.compile('\\{([a-zA-Z_][a-zA-Z0-9_]*)\\}')
def safe_template_format(template: str, values: Dict[str, Any]) -> str:
def _replace(match: re.Match) -> str:
key = match.group(1)
return str(values[key]) if key in values else match.group(0)
return _PLACEHOLDER_RE.sub(_replace, template)
class PromptTemplate:
def __init__(self, template: str):
self.template = template
def format(self, **kwargs: Any) -> str:
return safe_template_format(self.template, kwargs)
ARGS = {
"template": "Hello {name}, you have {count} messages and {score} points."
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type PromptTemplate — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a PromptTemplate output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| RunLog |
Ran ran on a synthesized input · constructed an object; no result was computed |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("8179670b62c5ef66") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, RunLog: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional
@dataclass
class DocQARecord:
plan_id: str
doc_id: str
attempts: List[Dict[str, Any]] = field(default_factory=list)
@dataclass
class RunLog:
run_id: str
qid: str
task: str
metadata_description: str
started_at: str
finished_at: Optional[str] = None
status: str = 'running'
error: Optional[str] = None
steps: List[Dict[str, Any]] = field(default_factory=list)
doc_interactions: Dict[str, Dict[str, DocQARecord]] = field(default_factory=dict)
artifacts: Dict[str, Any] = field(default_factory=dict)
torch.manual_seed(0)
# Construct sample DocQARecord instances
doc_qa_record_1 = DocQARecord(
plan_id="plan_001",
doc_id="doc_001",
attempts=[
{"attempt_num": 1, "result": "success", "score": 0.95},
{"attempt_num": 2, "result": "failure", "score": 0.42}
]
)
doc_qa_record_2 = DocQARecord(
plan_id="plan_002",
doc_id="doc_002",
attempts=[
{"attempt_num": 1, "result": "success", "score": 0.87}
]
)
# Construct doc_interactions structure
doc_interactions = {
"section_a": {
"qa_001": doc_qa_record_1,
"qa_002": doc_qa_record_2
},
"section_b": {
"qa_003": DocQARecord(
plan_id="plan_003",
doc_id="doc_003",
attempts=[]
)
}
}
# Construct steps list
steps = [
{"step_id": 1, "action": "retrieve", "timestamp": "2024-01-01T10:00:00"},
{"step_id": 2, "action": "process", "timestamp": "2024-01-01T10:00:05"},
{"step_id": 3, "action": "validate", "timestamp": "2024-01-01T10:00:10"}
]
# Construct artifacts dict
artifacts = {
"embedding_tensor": torch.randn(2, 4),
"metadata": {"version": "1.0", "model": "gpt-4"},
"scores": [0.91, 0.87, 0.73]
}
ARGS = {
"run_id": "run_20240101_001",
"qid": "question_123",
"task": "document_qa",
"metadata_description": "Question answering task on technical documentation",
"started_at": "2024-01-01T10:00:00Z",
"finished_at": "2024-01-01T10:05:32Z",
"status": "completed",
"error": None,
"steps": steps,
"doc_interactions": doc_interactions,
"artifacts": artifacts
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type RunLog — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a RunLog output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| StructuredLogger |
Ran ran on a synthesized input · constructed an object; no result was computed |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("dcf5d6dbc37366a4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, StructuredLogger: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import os
import tempfile
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional
# Create a temporary directory for the base_dir parameter
temp_dir = tempfile.mkdtemp()
ARGS = {
'qid': 'query_12345',
'task': 'document_qa_extraction',
'metadata_description': 'Extracting Q&A from financial documents',
'base_dir': temp_dir
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type StructuredLogger — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a StructuredLogger output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| _execute_python_code |
Ran ran; our contract draft was wrong, not the code |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("067c2b0c9b8a1172") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import subprocess
import sys
import os
# The function _execute_python_code takes:
# - code: str - Python code to execute
# - timeout_sec: int - timeout in seconds (default 120)
# We'll create a simple valid Python code string to execute
simple_code = """
x = 42
print(x)
"""
ARGS = {
'code': simple_code,
'timeout_sec': 120
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| _guess_doc_id |
Ran ran; our contract draft was wrong, not the code |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("14bc3048e73177d0") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from typing import Any, Dict, List, Optional
torch.manual_seed(0)
# Construct a valid meta dictionary for _guess_doc_id
# The function looks for keys: 'chatdoc_upload_id', 'doc_id', 'document_id', 'toBid', 'id'
# and returns the string value of the first one found that is not None
meta = {
'chatdoc_upload_id': 'upload_123',
'doc_id': 'doc_456',
'document_id': 'document_789',
'toBid': 'bid_000',
'id': 'id_111',
'other_field': 'some_value'
}
ARGS = {
'meta': meta
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| _match_restriction_value |
Ran ran, violated the contract we drafted |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("6422d17edb6287a6") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from typing import Any, Dict, List, Optional
torch.manual_seed(0)
# The function _match_restriction_value takes two parameters:
# 1. meta_value: Any - can be anything that can be converted to string
# 2. restriction_value: Any - can be a list/tuple/set or a single value
# Test case 1: meta_value as a string, restriction_value as a string
meta_value_1 = "test_mode"
restriction_value_1 = "test_mode"
# Test case 2: meta_value as an integer, restriction_value as a list
meta_value_2 = 42
restriction_value_2 = [42, 43, 44]
# Test case 3: meta_value as a float, restriction_value as a tuple
meta_value_3 = 3.14
restriction_value_3 = (2.71, 3.14, 1.41)
# Test case 4: meta_value as a tensor, restriction_value as a set
meta_value_4 = torch.tensor([1, 2, 3])
restriction_value_4 = {"tensor([1, 2, 3])", "other"}
# For the primary ARGS, use distinct test cases
# Using case 1: exact string match that should return True
ARGS = {
"meta_value": "test_mode",
"restriction_value": "test_mode"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| _normalize_for_hash |
Ran ran; our contract draft was wrong, not the code |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("6383424fe927625d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from typing import Any, Dict, List
torch.manual_seed(0)
# Test cases for _normalize_for_hash
# The function recursively normalizes dicts, lists, and other values for hashing
ARGS = {
"value": {
"z_key": [1, 2, 3],
"a_key": {"nested_z": 10, "nested_a": 20},
"m_key": [{"inner_z": 5}, {"inner_a": 15}]
}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| _redact |
Ran ran; our contract draft was wrong, not the code |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("aa9d78ea2f9ce79a") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
from typing import Any, Dict, List, Optional, Protocol
def _redact(x: Any) -> Any:
if isinstance(x, str):
x = re.sub('sk-[a-zA-Z0-9_-]{8,}', lambda m: m.group(0)[:6] + '***' + m.group(0)[-3:], x)
x = re.sub('ak--[a-zA-Z0-9_-]{8,}', lambda m: m.group(0)[:6] + '***' + m.group(0)[-3:], x)
return x
if isinstance(x, dict):
return {k: _redact(v) for k, v in x.items()}
if isinstance(x, list):
return [_redact(v) for v in x]
return x
# Test cases covering different input types
test_string_with_sk = "my api key is sk-abcdefghijklmnop"
test_string_with_ak = "my access key is ak--abcdefghijklmnop"
test_dict = {
"api_key": "sk-1234567890abcdef",
"nested": {
"access_key": "ak--zyxwvutsrqponml"
}
}
test_list = [
"sk-testkey1234567",
"ak--testaccess12345",
{"token": "sk-secrettoken123"}
]
test_integer = 42
test_float = 3.14
test_none = None
test_plain_string = "hello world"
ARGS = {
"x": test_dict
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| _to_posix |
Ran ran; our contract draft was wrong, not the code |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("c3dfe608b73c6159") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import os
def _to_posix(p: str) -> str:
return os.path.abspath(p).replace('\\', '/')
# The function takes a single string parameter `p` representing a file path
# It converts it to an absolute path and normalizes backslashes to forward slashes
# We need to provide a valid path string
ARGS = {
'p': '/tmp/test/path'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| build_question_id |
Ran ran; our contract draft was wrong, not the code |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("58682eacd4f958ef") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
from typing import Any, Dict, List, Optional
# Construct valid arguments for build_question_id
# The function takes:
# 1. question: Any - will be converted to string and stripped
# 2. metadata: List[Dict[str, Any]] - a list of dictionaries
ARGS = {
'question': 'What is the capital of France?',
'metadata': [
{'source': 'wikipedia', 'year': 2024, 'confidence': 0.95},
{'category': 'geography', 'difficulty': 'easy'},
{'tags': ['europe', 'capitals'], 'verified': True}
]
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| clean_code |
Ran ran; our contract draft was wrong, not the code |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("5b43b0d8ca375888") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import textwrap
# Analyze the function: clean_code takes a single parameter 'code' which is a string
# The function:
# 1. Checks if code is empty/falsy
# 2. Strips and dedents the code
# 3. Removes various code block prefixes (case-insensitive)
# 4. Returns the cleaned code string
# Create test cases for the 'code' parameter
code_with_prefix = "python\n def hello():\n pass"
code_with_spaces = " \n some code here\n "
code_simple = "x = 1 + 2"
code_empty = ""
code_with_py_prefix = "py\nprint('hello')"
code_with_code_prefix = "code: x = 5"
code_with_chinese = "执行:\n y = 10"
# The function uses the 'code' parameter as a string
# It calls: .strip(), .lower(), .startswith(), slicing, and .lstrip()
# All standard string operations
ARGS = {
"code": "python\n def example():\n return 42"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_tag_content |
Ran ran; our contract draft was wrong, not the code |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("8d211f361659e3f9") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
from typing import Any, Dict, List, Optional, Protocol
def extract_tag_content(text: str, tag: str) -> Optional[str]:
m = re.search(f'<{tag}>\\s*([\\s\\S]*?)\\s*</{tag}>', text)
if not m:
return None
return m.group(1).strip()
ARGS = {
"text": "<example> Hello World </example>",
"tag": "example"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| find_brace_json |
Ran ran; our contract draft was wrong, not the code |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("c6e31237e73bb6fe") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
# The function find_brace_json takes a single parameter: text (str)
# It searches for a JSON brace pair in the text and returns the substring
# or None if not found
ARGS = {
"text": '{"key": "value", "nested": {"inner": "data"}}'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| is_valid_python_code |
Ran ran, violated the contract we drafted |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("e2b550ccc397bb00") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
# The function is_valid_python_code takes a single string parameter 'code'
# and checks if it's valid Python code by attempting to compile it.
# We need to provide valid and invalid code strings as test cases.
ARGS = {
"code": "x = 1 + 2\nprint(x)"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| repair_json_like |
Ran ran; our contract draft was wrong, not the code |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("c06c9168b4af5105") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def repair_json_like(s: str) -> str:
if not isinstance(s, str):
return s
t = s
t = re.sub('\\bTrue\\b', 'true', t)
t = re.sub('\\bFalse\\b', 'false', t)
t = re.sub('\\bNone\\b', 'null', t)
t = re.sub("'([A-Za-z0-9_\\- ]+)'\\s*:", '"\\1":', t)
t = re.sub(":\\s*'([^']*)'", ': "\\1"', t)
t = re.sub(',\\s*([}\\]])', '\\1', t)
return t
# Test cases to demonstrate the function works
test_string = "{'key': 'value', 'flag': True, 'empty': None, 'another': 'test',}"
ARGS = {
's': test_string
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| resolve_question_id |
Ran ran; our contract draft was wrong, not the code |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("d38392764827dbe4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import json
from typing import Any, Dict, List
def _normalize_for_hash(value: Any) -> Any:
if isinstance(value, dict):
return {str(k): _normalize_for_hash(value[k]) for k in sorted(value.keys(), key=str)}
if isinstance(value, list):
return [_normalize_for_hash(item) for item in value]
return value
def build_question_id(question: Any, metadata: List[Dict[str, Any]]) -> str:
import hashlib
payload = {'question': str(question or '').strip(), 'metadata': _normalize_for_hash(metadata or [])}
raw = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(',', ':'))
return 'q_' + hashlib.sha256(raw.encode('utf-8')).hexdigest()[:16]
def resolve_question_id(item: Dict[str, Any]) -> str:
explicit = item.get('question_id')
if explicit not in (None, ''):
return str(explicit)
return build_question_id(item.get('question'), item.get('metadata') or [])
# Test case 1: item with explicit question_id
item_with_explicit_id = {
'question_id': 'q_explicit_12345',
'question': 'What is the answer?',
'metadata': [{'source': 'test', 'version': 1}]
}
# Test case 2: item without explicit question_id, should compute from question and metadata
item_without_explicit_id = {
'question': 'What is machine learning?',
'metadata': [
{'source': 'book', 'chapter': 3},
{'source': 'paper', 'year': 2023}
]
}
# Test case 3: item with empty question_id (should compute)
item_with_empty_id = {
'question_id': '',
'question': 'How does neural network work?',
'metadata': [{'type': 'educational'}]
}
# Test case 4: minimal item with only question
item_minimal = {
'question': 'Simple question'
}
ARGS = {
'item': item_with_explicit_id
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| robust_parse_json |
Ran ran; our contract draft was wrong, not the code |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("27f7f31226057a49") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import re
from typing import Any, Dict, List, Optional
# Construct test cases for robust_parse_json
ARGS = {
"text": '{"key": "value", "number": 42, "nested": {"inner": true}}',
"expected_type": None
}
# Test with different scenarios to ensure function works
test_cases = [
# Basic JSON string
('{"name": "test"}', None),
# JSON in code fence
('```json\n{"a": 1}\n```', None),
# JSON in plain code fence
('```\n{"b": 2}\n```', None),
# JSON with True/False/None (Python style)
('{\'key\': \'value\', \'flag\': True}', None),
# Plain brace JSON
('Some text before {"x": 1} some text after', None),
# Expected type as list
('[1, 2, 3]', 'list'),
# Expected type as dict
('{"data": [1,2,3]}', 'dict'),
# Empty/malformed input
('', None),
# Non-string (edge case - function checks isinstance)
('', None),
]
# Use the first test case as the primary ARGS
ARGS = {
"text": '{"name": "Alice", "age": 30, "active": true}',
"expected_type": 'dict'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| safe_template_format |
Ran ran; our contract draft was wrong, not the code |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("531accf8ba302721") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
from typing import Any, Dict, List, Optional, Protocol
_PLACEHOLDER_RE = re.compile('\\{([a-zA-Z_][a-zA-Z0-9_]*)\\}')
def safe_template_format(template: str, values: Dict[str, Any]) -> str:
def _replace(match: re.Match) -> str:
key = match.group(1)
return str(values[key]) if key in values else match.group(0)
return _PLACEHOLDER_RE.sub(_replace, template)
# Setup code
ARGS = {
"template": "Hello {name}, you have {count} messages and {score} points.",
"values": {
"name": "Alice",
"count": 42,
"score": 99.5
}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| AsyncMutiDocAgent |
Not yet run unverified (harvested, no recorded run) |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("bfeecb9b17f841a7") How it ran |
| OpenRouter |
Not yet run unverified (harvested, no recorded run) |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("3d73678810ab717a") How it ran |
| extract_json_from_code_fence |
Not yet run unverified (harvested, no recorded run) |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("b7d4a774bdb8e5b7") How it ran |
| extract_python_code |
Not yet run unverified (harvested, no recorded run) |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("41d7561067299007") How it ran |
| resolve_extractor |
Not yet run unverified (harvested, no recorded run) |
Zhanli-Li/MuDABench/agent/agent_runner.py code served (permissive licence) · get_code("74220594a73fb1a6") How it ran |