| extract_json |
Ran ran; our contract draft was wrong, not the code |
JingbiaoMei/ATM-Bench/memqa/qa_agent_baselines/A-Mem/memory_layer.py code served (permissive licence) · get_code("1ba9020de9ace83e") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
from typing import Any, Dict, List, Optional, Tuple, Union
def extract_json(payload: str) -> Optional[Dict[str, Any]]:
cleaned = payload.strip()
if cleaned.startswith('```'):
cleaned = cleaned.strip('`')
if cleaned.lower().startswith('json'):
cleaned = cleaned[4:]
start = cleaned.find('{')
end = cleaned.rfind('}')
if start == -1 or end == -1 or end <= start:
return None
fragment = cleaned[start:end + 1]
try:
return json.loads(fragment)
except Exception:
return None
# Setup code to construct valid arguments
ARGS = {
"payload": '```json\n{"key": "value", "number": 42}\n```'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_json_blob |
Ran ran; our contract draft was wrong, not the code |
JingbiaoMei/ATM-Bench/agent_systems/extract_answer.py code served (permissive licence) · get_code("b2d758e135419723") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
from typing import Any, Dict, Optional
def _find_first_json_object(text: str) -> Optional[str]:
in_string = False
escape = False
depth = 0
start_idx: Optional[int] = None
for idx, ch in enumerate(text):
if start_idx is None:
if ch == '{':
start_idx = idx
depth = 1
continue
if in_string:
if escape:
escape = False
continue
if ch == '\\':
escape = True
continue
if ch == '"':
in_string = False
continue
if ch == '"':
in_string = True
continue
if ch == '{':
depth += 1
continue
if ch == '}':
depth -= 1
if depth == 0:
return text[start_idx:idx + 1]
return None
def extract_json_blob(text: str) -> Dict[str, Any]:
text = text.strip()
if not text:
raise ValueError('empty text')
if text.startswith('```'):
lines = text.splitlines()
if len(lines) >= 2 and lines[-1].strip().startswith('```'):
text = '\n'.join(lines[1:-1]).strip()
try:
parsed = json.loads(text)
if isinstance(parsed, dict):
return parsed
except json.JSONDecodeError:
pass
candidate = _find_first_json_object(text)
if candidate:
parsed = json.loads(candidate)
if isinstance(parsed, dict):
return parsed
raise ValueError('could not parse JSON object from text')
# Test case 1: Simple JSON object
simple_json = '{"key": "value", "number": 42}'
# Test case 2: JSON wrapped in markdown code block
markdown_json = '''```json
{"name": "test", "count": 10}
'''
# Test case 3: JSON object embedded in text
embedded_json = 'Here is some text before {"embedded": true, "value": 123} and after'
# Test case 4: Nested JSON object
nested_json = '{"outer": {"inner": "value"}, "list": [1, 2, 3]}'
# Test case 5: JSON with escaped quotes
escaped_json = '{"escaped": "value with \\"quotes\\"", "other": "test"}'
ARGS = {
'text': simple_json
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_model_config |
Ran ran; our contract draft was wrong, not the code |
JingbiaoMei/ATM-Bench/memqa/global_config.py code served (permissive licence) · get_code("f9e247fbeeffa980") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import os
from pathlib import Path
from unittest.mock import patch, MagicMock
# Mock environment variables and file system to avoid actual file I/O
os.environ.pop('OPENAI_API_KEY', None)
os.environ.pop('VLLM_API_KEY', None)
os.environ.pop('ATMBENCH_STRICT_API_KEYS', None)
# Create minimal stand-in for the module to avoid actual file operations
import sys
from types import ModuleType
# We need to set up the context so the module loads correctly
# Mock the api_keys directory check
mock_api_keys_dir = Path('/tmp/api_keys_mock')
# Patch the file existence checks and reads
def mock_exists(self):
return False
def mock_open_file(self, *args, **kwargs):
raise FileNotFoundError()
# We'll patch Path.exists at import time by mocking the environment
with patch.dict(os.environ, {'ATMBENCH_STRICT_API_KEYS': '0'}):
# Now we can safely call the functions with empty API keys
ARGS = {
'provider': 'vllm',
'model_type': 'vision',
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_output_path |
Ran ran; our contract draft was wrong, not the code |
JingbiaoMei/ATM-Bench/memqa/global_config.py code served (permissive licence) · get_code("3b464087313f9c13") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned PosixPath. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from pathlib import Path
# The function get_output_path takes:
# - module: str (e.g., "image", "email", "event")
# - filename: str or None (optional filename to append)
# Reading the function body:
# 1. It creates a base_path = OUTPUT_DIR / module
# 2. If filename is provided, it appends it: base_path / filename
# 3. Returns a Path object
# The function only uses string concatenation with Path objects, so we just need valid strings.
ARGS = {
"module": "image",
"filename": "test_output.txt"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type PosixPath — a PosixPath output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a PosixPath output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| item_id |
Ran ran; our contract draft was wrong, not the code |
JingbiaoMei/ATM-Bench/agent_systems/memory_variants.py code served (permissive licence) · get_code("fba80e0d9cdecb65") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
from pathlib import Path
from typing import Any
torch.manual_seed(0)
ARGS = {
'item': {
'file_path': '/home/user/documents/example_file.txt',
},
'path_key': 'file_path',
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| last_token_pool |
Ran ran; our contract draft was wrong, not the code |
JingbiaoMei/ATM-Bench/memqa/retrieve/retrievers.py code served (permissive licence) · get_code("b15d6769e9f3531d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Create a batch of 2 sequences with hidden state dimension 4
batch_size = 2
seq_length = 3
hidden_dim = 4
# Create last_hidden_states: shape (batch_size, seq_length, hidden_dim)
last_hidden_states = torch.randn(batch_size, seq_length, hidden_dim)
# Create attention_mask: shape (batch_size, seq_length)
# For left padding case: all sequences have full attention at the end
attention_mask = torch.ones(batch_size, seq_length, dtype=torch.long)
ARGS = {
"last_hidden_states": last_hidden_states,
"attention_mask": attention_mask,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_json |
Ran ran; our contract draft was wrong, not the code |
JingbiaoMei/ATM-Bench/agent_systems/extract_answer.py code served (permissive licence) · get_code("e1a33f82f601f5fe") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
from pathlib import Path
import tempfile
# Create a temporary directory and a test JSON file
temp_dir = tempfile.mkdtemp()
test_file = Path(temp_dir) / "test.json"
# Write some test data to the JSON file
test_data = {
"key1": "value1",
"key2": [1, 2, 3],
"key3": {"nested": "object"}
}
with test_file.open('w', encoding='utf-8') as f:
json.dump(test_data, f)
# Define the arguments dictionary
ARGS = {
"path": test_file
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_json |
Ran ran; our contract draft was wrong, not the code |
JingbiaoMei/ATM-Bench/agent_systems/extract_usage.py code served (permissive licence) · get_code("d0d3faa7cc886a81") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
from pathlib import Path
import tempfile
import os
# Create a temporary directory and a temporary JSON file
temp_dir = tempfile.mkdtemp()
temp_json_path = Path(temp_dir) / "test.json"
# Write a valid JSON file
test_data = {
"key1": "value1",
"key2": [1, 2, 3],
"key3": {"nested": "object"}
}
with open(temp_json_path, 'w', encoding='utf-8') as f:
json.dump(test_data, f)
ARGS = {
"path": temp_json_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_json_list |
Ran ran; our contract draft was wrong, not the code |
JingbiaoMei/ATM-Bench/agent_systems/memory_variants.py code served (permissive licence) · get_code("97f74443ec5601e5") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
from pathlib import Path
import tempfile
# Create a temporary JSON file with valid list content
with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False, encoding='utf-8') as f:
json_data = [
{"key1": "value1", "key2": 42},
{"key1": "value2", "key2": 100},
{"nested": {"a": 1, "b": 2}}
]
json.dump(json_data, f)
temp_path = Path(f.name)
ARGS = {
"path": temp_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_json_with_recovery |
Ran ran; our contract draft was wrong, not the code |
JingbiaoMei/ATM-Bench/agent_systems/extract_answer.py code served (permissive licence) · get_code("288ca5fef01abdbc") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
from pathlib import Path
import tempfile
import os
def _find_first_json_object(text: str):
in_string = False
escape = False
depth = 0
start_idx = None
for idx, ch in enumerate(text):
if start_idx is None:
if ch == '{':
start_idx = idx
depth = 1
continue
if in_string:
if escape:
escape = False
continue
if ch == '\\':
escape = True
continue
if ch == '"':
in_string = False
continue
if ch == '"':
in_string = True
continue
if ch == '{':
depth += 1
continue
if ch == '}':
depth -= 1
if depth == 0:
return text[start_idx:idx + 1]
return None
def load_json_with_recovery(path: Path):
text = path.read_text(encoding='utf-8')
try:
return json.loads(text)
except json.JSONDecodeError:
candidate = _find_first_json_object(text)
if candidate:
return json.loads(candidate)
raise
# Create a temporary directory and file
temp_dir = tempfile.mkdtemp()
temp_file_path = Path(temp_dir) / "test.json"
# Write valid JSON to the file
valid_json_content = '{"key": "value", "number": 42, "nested": {"inner": true}}'
temp_file_path.write_text(valid_json_content, encoding='utf-8')
ARGS = {
'path': temp_file_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_question_ids |
Ran ran; our contract draft was wrong, not the code |
JingbiaoMei/ATM-Bench/agent_systems/collect_results.py code served (permissive licence) · get_code("00dedf0286b4a63c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from pathlib import Path
import tempfile
# Create a temporary directory with a question_ids.txt file
temp_dir = tempfile.TemporaryDirectory()
eval_root = Path(temp_dir.name)
# Create a sample question_ids.txt file
ids_file = eval_root / 'question_ids.txt'
ids_file.write_text('q_001\nq_002\nq_003\n\nq_004\n', encoding='utf-8')
ARGS = {
'eval_root': eval_root,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_questions |
Ran ran; our contract draft was wrong, not the code |
JingbiaoMei/ATM-Bench/agent_systems/prepare_sandbox.py code served (permissive licence) · get_code("e748b717c3a734f2") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
from pathlib import Path
# Create a temporary JSON file with sample ATM-hard questions
temp_dir = tempfile.TemporaryDirectory()
temp_path = Path(temp_dir.name) / "questions.json"
sample_data = [
{
"id": "q1",
"question": "What is the capital of France?",
"ground_truth": "Paris",
"other_field": "should_be_stripped"
},
{
"id": "q2",
"question": "What is 2 + 2?",
"ground_truth": "4",
"answer": "should_also_be_stripped"
}
]
with open(temp_path, 'w', encoding='utf-8') as f:
json.dump(sample_data, f)
ARGS = {
'path': temp_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| normalize_memory_mode |
Ran ran; our contract draft was wrong, not the code |
JingbiaoMei/ATM-Bench/agent_systems/memory_variants.py code served (permissive licence) · get_code("28eaaa08671641d6") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
# Setup code for normalize_memory_mode function
ARGS = {
'mode': 'sgm'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| resolve_repo_path |
Ran ran; our contract draft was wrong, not the code |
JingbiaoMei/ATM-Bench/agent_systems/config.py code served (permissive licence) · get_code("49b8dc6d41651408") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned PosixPath. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
from pathlib import Path
# Create a temporary directory structure to serve as REPO_ROOT
with tempfile.TemporaryDirectory() as tmpdir:
repo_root = Path(tmpdir)
parent_dir = repo_root.parent
# Create a subdirectory structure
subdir = repo_root / "subdir"
subdir.mkdir(exist_ok=True)
# Create some test files
(repo_root / "test.txt").touch()
(subdir / "nested.txt").touch()
# Test with a relative path
test_relative_path = "subdir/nested.txt"
# Test with an absolute path
test_absolute_path = str(repo_root / "test.txt")
# For this test, we'll use a relative path since the function
# handles both absolute and relative paths
ARGS = {
"path": "subdir/nested.txt"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type PosixPath — a PosixPath output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a PosixPath output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| runtime_eval_value |
Ran ran; our contract draft was wrong, not the code |
JingbiaoMei/ATM-Bench/agent_systems/collect_results.py code served (permissive licence) · get_code("088fcf6bbfb120b4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# Analyze the function:
# - value: str parameter, can be empty string, a string value, or in legacy_values
# - fallback: str parameter, returned when value is empty or in legacy_values
# - legacy_values: tuple of strings, optional parameter with default ()
# The function checks:
# 1. if not value (empty string check)
# 2. if value in legacy_values (membership test)
# 3. returns fallback if either condition is true
# 4. otherwise returns value
ARGS = {
"value": "current_value",
"fallback": "default_value",
"legacy_values": ("old_value_1", "old_value_2", "deprecated")
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_api_key |
Not yet run unverified (harvested, no recorded run) |
JingbiaoMei/ATM-Bench/memqa/global_config.py code served (permissive licence) · get_code("159e7f0ea5b4e0b2") How it ran |
| load_qwen3_vl_embedder |
Not yet run unverified (harvested, no recorded run) |
JingbiaoMei/ATM-Bench/memqa/retrieve/retrievers.py code served (permissive licence) · get_code("99f295e69c490e0a") How it ran |
| load_qwen3_vl_reranker |
Not yet run unverified (harvested, no recorded run) |
JingbiaoMei/ATM-Bench/memqa/retrieve/rerankers.py code served (permissive licence) · get_code("e34f5854798ea44f") How it ran |