| average |
Ran ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT) |
viviable/mopd_code/analysis/compute_teacher_signal_metrics.py code served (permissive licence) · get_code("bd3548f55a4d47e5") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
# The function takes a list of floats and computes their average
# We need to construct a valid list of float values
values_list = [1.5, 2.5, 3.5, 4.5, 5.5]
ARGS = {
"values": values_list
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| build_student_targets |
Ran ran; our contract draft was wrong, not the code |
viviable/mopd_code/analysis/build_target_responses.py code served (permissive licence) · get_code("d7aa73af385d98f7") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import hashlib
from collections import defaultdict
from typing import Any
def stable_target_id(target_type: str, prompt_id: str, source_response_id: str | None, response_text: str) -> str:
raw = f"{target_type}|{prompt_id}|{source_response_id or 'none'}|{response_text}"
return hashlib.sha1(raw.encode('utf-8')).hexdigest()[:20]
def build_student_targets(candidates: list[dict[str, Any]], student_success_per_prompt: int, student_failure_per_prompt: int) -> list[dict[str, Any]]:
by_prompt: dict[str, list[dict[str, Any]]] = defaultdict(list)
for row in candidates:
by_prompt[row['prompt_id']].append(row)
targets: list[dict[str, Any]] = []
for prompt_id, rows in by_prompt.items():
rows = sorted(rows, key=lambda x: x['response_id'])
successes = [row for row in rows if row['success']][:student_success_per_prompt]
failures = [row for row in rows if not row['success']][:student_failure_per_prompt]
for row in successes:
targets.append({'target_id': stable_target_id('student_success', prompt_id, row['response_id'], row['response_text']), 'target_type': 'student_success', 'prompt_id': prompt_id, 'source_response_id': row['response_id'], 'response_text': row['response_text'], 'success': True, 'reward': row['reward'], 'source_model': 'student_rollout', 'difficulty_bucket': row['difficulty_bucket'], 'metadata': {'rollout_step': row.get('rollout_step'), 'pred': row.get('pred'), 'ground_truth': row.get('ground_truth')}})
for row in failures:
targets.append({'target_id': stable_target_id('student_failure', prompt_id, row['response_id'], row['response_text']), 'target_type': 'student_failure', 'prompt_id': prompt_id, 'source_response_id': row['response_id'], 'response_text': row['response_text'], 'success': False, 'reward': row['reward'], 'source_model': 'student_rollout', 'difficulty_bucket': row['difficulty_bucket'], 'metadata': {'rollout_step': row.get('rollout_step'), 'pred': row.get('pred'), 'ground_truth': row.get('ground_truth')}})
return targets
candidates = [
{
'prompt_id': 'prompt_1',
'response_id': 'resp_1',
'response_text': 'This is a correct answer.',
'success': True,
'reward': 0.95,
'difficulty_bucket': 'easy',
'rollout_step': 1,
'pred': 'correct_output',
'ground_truth': 'correct_output'
},
{
'prompt_id': 'prompt_1',
'response_id': 'resp_2',
'response_text': 'This is an incorrect answer.',
'success': False,
'reward': 0.1,
'difficulty_bucket': 'easy',
'rollout_step': 1,
'pred': 'wrong_output',
'ground_truth': 'correct_output'
},
{
'prompt_id': 'prompt_1',
'response_id': 'resp_3',
'response_text': 'Another correct response.',
'success': True,
'reward': 0.92,
'difficulty_bucket': 'medium',
'rollout_step': 2,
'pred': 'correct_output',
'ground_truth': 'correct_output'
},
{
'prompt_id': 'prompt_2',
'response_id': 'resp_4',
'response_text': 'A successful response.',
'success': True,
'reward': 0.88,
'difficulty_bucket': 'hard',
'rollout_step': 1,
'pred': 'result_a',
'ground_truth': 'result_a'
},
]
ARGS = {
'candidates': candidates,
'student_success_per_prompt': 2,
'student_failure_per_prompt': 1,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| check_correctness |
Ran ran; our contract draft was wrong, not the code |
viviable/mopd_code/verl/utils/reward_score/prime_code/utils.py code served (permissive licence) · get_code("2d94e42b971ca757") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import multiprocessing
import sys
# Mock run_test function that the code calls
def run_test(in_outs, test, debug, timeout):
"""Mock implementation of run_test"""
# Returns (result_list, metadata_dict)
num_inputs = len(in_outs['inputs'])
result = [1 for _ in range(num_inputs)] # Success indicators
metadata = {'test_name': 'sample_test', 'passed': True}
return result, metadata
# Create mock in_outs dict with inputs
in_outs = {
'inputs': [
{'arg1': 10, 'arg2': 20},
{'arg1': 30, 'arg2': 40},
{'arg1': 50, 'arg2': 60}
]
}
# Mock generation string (code to test)
generation = """
def add(a, b):
return a + b
result = add(10, 20)
"""
ARGS = {
'in_outs': in_outs,
'generation': generation,
'timeout': 10,
'debug': True
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| ci_yerr |
Ran ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT) |
viviable/mopd_code/analysis/plot_teacher_metrics.py code served (permissive licence) · get_code("5ef8c6dd23fae513") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 2, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Build the metrics_view dict structure that ci_yerr expects
metrics_view = {
'conditions': {
'condition_A': {
'accuracy_ci': (0.05, 0.15),
'loss_ci': (0.02, 0.08)
},
'condition_B': {
'accuracy_ci': (0.03, 0.12),
'loss_ci': (0.01, 0.06)
},
'condition_C': {
'accuracy_ci': (0.04, 0.14),
'loss_ci': (0.025, 0.075)
}
}
}
# Build the arguments
ARGS = {
'metrics_view': metrics_view,
'conditions': ['condition_A', 'condition_B', 'condition_C'],
'metric': 'accuracy',
'values': [0.85, 0.90, 0.88]
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 2, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| condition_colors |
Ran ran; our contract draft was wrong, not the code |
viviable/mopd_code/analysis/plot_teacher_metrics.py code served (permissive licence) · get_code("288f33b5eca65cc7") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
ARGS = {
'n': 5
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| fetch_history |
Ran ran; our contract draft was wrong, not the code |
viviable/mopd_code/analysis/draw_acc.py code served (permissive licence) · get_code("caa5d8692e5bdb76") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import wandb
from unittest.mock import Mock, MagicMock
# Create a mock API and run object to avoid actual network calls
mock_api = Mock()
mock_run = Mock()
# Mock the scan_history method to return test data
def mock_scan_history(keys=None):
return [
{'_step': 0, 'loss': 0.5},
{'_step': 1, 'loss': 0.4},
{'_step': 2, 'loss': None}, # This row will be skipped
{'_step': 3, 'loss': 0.3},
]
mock_run.scan_history = mock_scan_history
mock_api.run.return_value = mock_run
# Patch wandb.Api to return our mock
wandb.Api = Mock(return_value=mock_api)
ARGS = {
'entity': 'test_entity',
'project': 'test_project',
'run_id': 'test_run_123',
'metric': 'loss'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- wandb — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_metrics |
Ran ran; our contract draft was wrong, not the code |
viviable/mopd_code/analysis/plot_teacher_metrics.py code served (permissive licence) · get_code("aebe25129f6e22f7") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
from pathlib import Path
# Create a temporary metrics file
with tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False) as f:
metrics_data = {
"accuracy": 0.95,
"loss": 0.123,
"epochs": 10,
"model_name": "test_model"
}
json.dump(metrics_data, f)
temp_path = Path(f.name)
ARGS = {
"path": temp_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| make_client |
Ran ran; our contract draft was wrong, not the code |
viviable/mopd_code/analysis/generate_strong_success.py code served (permissive licence) · get_code("b97c9a97c01f9eea") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned OpenAI. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
# The function make_client requires:
# - api_key: str
# - base_url: str | None
# - default_headers: dict[str, str] | None (optional, defaults to None)
# Mock the OpenAI class since we don't want to actually import it
class OpenAI:
def __init__(self, api_key: str, base_url: str | None = None, default_headers: dict | None = None):
self.api_key = api_key
self.base_url = base_url
self.default_headers = default_headers
# Set OpenAI in the global namespace so the function can access it
import builtins
builtins.OpenAI = OpenAI
ARGS = {
'api_key': 'sk-test-key-12345',
'base_url': 'https://api.openai.com/v1',
'default_headers': {'Authorization': 'Bearer token', 'Content-Type': 'application/json'}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type OpenAI — a OpenAI output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a OpenAI output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| only_int_check |
Ran ran, violated the contract we drafted |
viviable/mopd_code/verl/utils/reward_score/prime_code/testing_util.py code served (permissive licence) · get_code("6339c418c950b9c8") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# The function only_int_check takes a single parameter 'val' and checks if it's an integer
# We need to provide a valid integer value for this parameter
ARGS = {
"val": 42
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| parse_run_path |
Ran ran; our contract draft was wrong, not the code |
viviable/mopd_code/analysis/draw_acc.py code served (permissive licence) · get_code("0898e734fc60f2e9") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from urllib.parse import urlparse
def parse_run_path(run_value: str) -> tuple[str, str, str]:
if run_value.startswith('http://') or run_value.startswith('https://'):
parsed = urlparse(run_value)
parts = [part for part in parsed.path.split('/') if part]
if len(parts) >= 4 and parts[2] == 'runs':
return (parts[0], parts[1], parts[3])
raise ValueError(f'Unsupported W&B run URL format: {run_value}')
parts = [part for part in run_value.split('/') if part]
if len(parts) == 3:
return (parts[0], parts[1], parts[2])
raise ValueError('Run must be a W&B URL or entity/project/run_id.')
# Test case 1: URL format with https
run_value_1 = "https://wandb.ai/entity/project/runs/run_id_123"
# Test case 2: Simple entity/project/run_id format
run_value_2 = "my_entity/my_project/run_abc"
ARGS = {
"run_value": run_value_1
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| rankdata |
Ran ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT) |
viviable/mopd_code/analysis/compute_teacher_signal_metrics.py code served (permissive licence) · get_code("414365b67878b9e4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# The function rankdata takes a list of floats and returns their ranks
# with average ranking for tied values
values = [3.5, 1.2, 3.5, 2.1, 1.2, 4.0]
ARGS = {
"values": values
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| read_jsonl |
Ran ran; our contract draft was wrong, not the code |
viviable/mopd_code/analysis/score_teacher_contexts.py code served (permissive licence) · get_code("903f3f3a478c7f63") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
from pathlib import Path
import tempfile
import os
# Create a temporary JSONL file with test data
temp_dir = tempfile.mkdtemp()
temp_file = Path(temp_dir) / "test.jsonl"
# Write some test data to the JSONL file
test_data = [
{"name": "Alice", "age": 30, "city": "New York"},
{"name": "Bob", "age": 25, "city": "Los Angeles"},
{"name": "Charlie", "age": 35, "city": "Chicago"},
]
with temp_file.open("w") as f:
for row in test_data:
f.write(json.dumps(row) + "\n")
ARGS = {
"path": temp_file,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| read_jsonl |
Ran ran; our contract draft was wrong, not the code |
viviable/mopd_code/analysis/compute_teacher_signal_metrics.py code served (permissive licence) · get_code("2b6d83de04b5e4ad") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
from pathlib import Path
from typing import Any
# Create a temporary JSONL file with test data
temp_dir = tempfile.TemporaryDirectory()
temp_path = Path(temp_dir.name) / "test.jsonl"
# Write some test JSONL data
with temp_path.open("w") as f:
f.write(json.dumps({"key1": "value1", "key2": 42}) + "\n")
f.write(json.dumps({"key1": "value2", "key2": 100}) + "\n")
f.write("\n") # Test empty line handling
f.write(json.dumps({"key1": "value3", "key2": 200}) + "\n")
ARGS = {
"path": temp_path,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| read_variant_config |
Ran ran; our contract draft was wrong, not the code |
viviable/mopd_code/analysis/build_context_variants.py code served (permissive licence) · get_code("759afe3c4dcbc678") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
from pathlib import Path
from tempfile import NamedTemporaryFile
# Create a temporary JSON file with a valid variant config
config_data = {
'global': {
'dont_reprompt_on_self_success': True,
'feedback_only_without_solution': False,
'summary_k': 2,
'summary_sampling_policy': 'first_k'
},
'variants': [
{'name': 'base'},
{'name': 'base_raw'},
{'name': 'solution', 'include_solution': True}
]
}
# Write to a temporary file
temp_file = NamedTemporaryFile(mode='w', suffix='.json', delete=False)
json.dump(config_data, temp_file)
temp_file.close()
ARGS = {
'path': temp_file.name
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| resolve_api_config |
Ran ran; our contract draft was wrong, not the code |
viviable/mopd_code/analysis/generate_strong_success.py code served (permissive licence) · get_code("d9156c6e78fc7a12") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import argparse
ARGS = {
'args': argparse.Namespace(
backend='openrouter',
api_key_env='OPENAI_API_KEY',
base_url='https://custom.example.com',
http_referer='https://myapp.example.com',
x_title='My Application'
)
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| resolve_dtype |
Ran ran; our contract draft was wrong, not the code |
viviable/mopd_code/analysis/score_teacher_contexts.py code served (permissive licence) · get_code("2d63c8da40e7f663") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dtype. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# The function takes a single parameter: dtype_name (str)
# It maps string names to torch dtypes or returns None for 'auto'
# We need to test with a valid dtype name string
ARGS = {
'dtype_name': 'float32'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dtype — a dtype output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dtype output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| resolve_run |
Ran ran; our contract draft was wrong, not the code |
viviable/mopd_code/analysis/fetch_wandb_metric_compare.py code served (permissive licence) · get_code("53fb0fcdb63f7f14") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Mock. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import wandb
from unittest.mock import Mock, MagicMock
# Create a mock wandb.Api instance
mock_api = Mock(spec=wandb.Api)
# Create a mock run object with necessary attributes
mock_run = Mock()
mock_run.display_name = "test_run"
mock_run.name = "test_run_id"
# Mock the api.runs() method to return a list with one run
mock_api.runs = Mock(return_value=[mock_run])
ARGS = {
'api': mock_api,
'entity': 'test_entity',
'project': 'test_project',
'run_name': 'test_run'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- wandb — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Mock — a Mock output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| sanitize |
Ran ran; our contract draft was wrong, not the code |
viviable/mopd_code/analysis/fetch_wandb_metric_compare.py code served (permissive licence) · get_code("6f3c534aa4ba4ee2") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
def sanitize(text: str) -> str:
return text.replace('/', '_').replace('@', 'at').replace(' ', '_').replace(':', '_')
ARGS = {
'text': 'hello/world@example.com:path with spaces'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| sanitize_metric_name |
Ran ran; our contract draft was wrong, not the code |
viviable/mopd_code/analysis/draw_acc.py code served (permissive licence) · get_code("f5674775fedebe04") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
ARGS = {
'metric': 'loss/train@epoch_1'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| stable_target_id |
Ran ran; our contract draft was wrong, not the code |
viviable/mopd_code/analysis/build_target_responses.py code served (permissive licence) · get_code("2d00b81e871f020d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import hashlib
ARGS = {
"target_type": "response",
"prompt_id": "prompt_123",
"source_response_id": "source_456",
"response_text": "This is a test response for stable target id generation."
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| stable_variant_id |
Ran ran; our contract draft was wrong, not the code |
viviable/mopd_code/analysis/build_context_variants.py code served (permissive licence) · get_code("3d4ee5bf4a4b62a8") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import hashlib
def stable_variant_id(condition: str, prompt_id: str, response_id: str) -> str:
raw = f'{condition}|{prompt_id}|{response_id}'
return hashlib.sha1(raw.encode('utf-8')).hexdigest()[:20]
ARGS = {
'condition': 'control_group_a',
'prompt_id': 'prompt_001',
'response_id': 'response_001'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| string_int_check |
Ran ran, violated the contract we drafted |
viviable/mopd_code/verl/utils/reward_score/prime_code/testing_util.py code served (permissive licence) · get_code("0b462a89ce04228a") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
ARGS = {
"val": "12345"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| truncatefn |
Ran ran; our contract draft was wrong, not the code |
viviable/mopd_code/verl/utils/reward_score/prime_code/testing_util.py code served (permissive licence) · get_code("22849627aa42fa47") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# The function truncatefn takes a string and optional length parameter
# It checks if the string is a string type, and if it's longer than length,
# it truncates it by keeping the first half and last half with '...(truncated) ...' in between
# Test case 1: Short string that doesn't need truncation
short_string = "Hello, World!"
# Test case 2: Long string that will be truncated
long_string = "The quick brown fox jumps over the lazy dog. " * 20 # Create a long string
ARGS = {
"s": long_string,
"length": 300
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_model_and_tokenizer |
Not yet run unverified (harvested, no recorded run) |
viviable/mopd_code/analysis/score_teacher_contexts.py code served (permissive licence) · get_code("1a52c3aa3027b3fe") How it ran |
| load_spec |
Not yet run unverified (harvested, no recorded run) |
viviable/mopd_code/analysis/plot_heatmap_grid.py code served (permissive licence) · get_code("984827baf66eba8d") How it ran |
| resolve_run_path |
Not yet run unverified (harvested, no recorded run) |
viviable/mopd_code/analysis/fetch_wandb_metric_compare.py code served (permissive licence) · get_code("b29f33cf4c2111e6") How it ran |
| run_test_code |
Not yet run unverified (harvested, no recorded run) |
viviable/mopd_code/verl/utils/reward_score/feedback/code.py code served (permissive licence) · get_code("45691de9cab1a113") How it ran |
| run_test_func |
Not yet run unverified (harvested, no recorded run) |
viviable/mopd_code/verl/utils/reward_score/feedback/code.py code served (permissive licence) · get_code("4a71e4f42656119d") How it ran |
| run_test_std |
Not yet run unverified (harvested, no recorded run) |
viviable/mopd_code/verl/utils/reward_score/feedback/code.py code served (permissive licence) · get_code("f1b61b9749cac892") How it ran |