| add_boxed_match |
Ran ran; our contract draft was wrong, not the code |
monasch/llm-monitor/python_scripts/add_final_label.py code served (permissive licence) · get_code("64717da780036e1f") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
from pathlib import Path
import tempfile
import os
# Create a temporary directory for test files
temp_dir = tempfile.mkdtemp()
# Create a sample critique file with REFERENCE SOLUTION and steps
sample_content = r"""
REFERENCE SOLUTION:
Let me solve this step by step.
The answer is \boxed{\frac{1}{2}}
GENERATOR:
--- Step 1: Initial setup ---
We start with the problem.
--- Step 2: Calculation ---
Working through the math.
--- Step 3: Final answer ---
\boxed{\frac{1}{2}}
RAW_CRITIC: This is a test file.
"""
# Write the sample file
test_filepath = os.path.join(temp_dir, "test_critique.txt")
with open(test_filepath, 'w', encoding='utf-8') as f:
f.write(sample_content)
ARGS = {
'filepath': test_filepath
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| add_judge_probability_series |
Ran ran; our contract draft was wrong, not the code |
monasch/llm-monitor/src/evaluator/utils.py code served (permissive licence) · get_code("c3dc58a9c1c0ea0c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import pandas as pd
import numpy as np
# Create a small sample DataFrame with the required columns
df = pd.DataFrame({
'uq_problem_idx': [1, 1, 1, 2, 2, 2],
'num_steps': [1, 2, 3, 1, 2, 3],
'judge_probability': [0.5, 0.6, 0.7, 0.4, 0.5, 0.6]
})
ARGS = {
'df': df,
'prob_col': 'judge_probability',
'problem_col': 'uq_problem_idx',
'step_col': 'num_steps'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, pandas, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| build_guard_prompt |
Ran ran; our contract draft was wrong, not the code |
monasch/llm-monitor/python_scripts/llama_guard.py code served (permissive licence) · get_code("db227e21d54e1612") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from unittest.mock import MagicMock
torch.manual_seed(0)
# Create a minimal tokenizer mock that supports apply_chat_template
class MockTokenizer:
def apply_chat_template(self, turns, tokenize=False, add_generation_prompt=False):
# Simulate what a real tokenizer would do for last_turn mode
# Return a formatted string with chat template applied
convo_lines = []
for t in turns:
role_label = 'user' if t['role'] == 'user' else 'assistant'
convo_lines.append(f"<|start_header_id|>{role_label}<|end_header_id|>\n\n{t['content']}<|eot_id|>")
result = '<|begin_of_text|>' + ''.join(convo_lines)
if add_generation_prompt:
result += '<|start_header_id|>assistant<|end_header_id|>\n\n'
return result
# Create test data for different modes
tokenizer = MockTokenizer()
# Test case 1: last_turn mode - strictly alternating turns ending with assistant
turns_last_turn = [
{'role': 'user', 'content': 'Hello, how are you?'},
{'role': 'assistant', 'content': 'I am doing well, thank you for asking.'}
]
# Test case 2: full_context mode - multiple turns
turns_full_context = [
{'role': 'user', 'content': 'Can you help me?'},
{'role': 'assistant', 'content': 'Of course, I can help.'},
{'role': 'user', 'content': 'What is the capital of France?'},
{'role': 'assistant', 'content': 'The capital of France is Paris.'}
]
# Test case 3: per_turn_iso mode - single turn only
turns_per_turn = [
{'role': 'user', 'content': 'Is this message safe?'}
]
# Prepare ARGS for each mode
ARGS_last_turn = {
'tokenizer': tokenizer,
'turns': turns_last_turn,
'mode': 'last_turn'
}
ARGS_full_context = {
'tokenizer': tokenizer,
'turns': turns_full_context,
'mode': 'full_context'
}
ARGS_per_turn_iso = {
'tokenizer': tokenizer,
'turns': turns_per_turn,
'mode': 'per_turn_iso'
}
# Use last_turn as the primary ARGS
ARGS = ARGS_last_turn
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| csv_filename |
Ran ran; our contract draft was wrong, not the code |
monasch/llm-monitor/python_scripts/merge_signal_label.py code served (permissive licence) · get_code("393ee2e04384683a") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
ARGS = {
"gen_model": "gpt2",
"dataset": "wikitext",
"prm_model": "bert_base"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_boxed |
Ran ran; our contract draft was wrong, not the code |
monasch/llm-monitor/python_scripts/add_final_label.py code served (permissive licence) · get_code("5a8fb7969db533be") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def extract_boxed(text):
"""Extract the content of the last \\boxed{} in a string."""
matches = []
for m in re.finditer('\\\\boxed\\{', text):
start, depth, i = (m.end(), 1, m.end())
while i < len(text) and depth > 0:
if text[i] == '{':
depth += 1
elif text[i] == '}':
depth -= 1
i += 1
if depth == 0:
matches.append(text[start:i - 1])
return matches[-1].strip() if matches else None
# Setup code
text_with_boxed = r"The answer is \boxed{42}"
text_with_nested_boxed = r"Calculation: \boxed{x^2 + y = \{a, b\}}"
text_without_boxed = "No boxed content here"
text_with_multiple_boxed = r"First \boxed{answer1} and then \boxed{final_answer}"
ARGS = {
"text": text_with_multiple_boxed
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_last_boxed_balanced |
Ran ran; our contract draft was wrong, not the code |
monasch/llm-monitor/python_scripts/get_signal_prm.py code served (permissive licence) · get_code("0d406cfe40641377") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# The function extract_last_boxed_balanced takes a single parameter: text (str)
# It searches for \boxed{ patterns in the text and extracts the last one found
# The function processes the text to find balanced braces and normalize content
ARGS = {
"text": r"Some math problem with \boxed{x^2 + 1} and another \boxed{42} at the end."
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_ground_truth |
Ran ran; our contract draft was wrong, not the code |
monasch/llm-monitor/python_scripts/get_signal_prm.py code served (permissive licence) · get_code("35a984834e874c85") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from unittest.mock import Mock, patch
torch.manual_seed(0)
# Mock dataset that behaves like a HuggingFace dataset
class MockDataset:
def __init__(self, size=10):
self.size = size
self.data = [
{'solution': f'solution_{i}', 'answer': f'answer_{i}', 'output': f'output_{i}'}
for i in range(size)
]
def __len__(self):
return self.size
def __getitem__(self, idx):
return self.data[idx]
# Mock the load_dataset function to return our MockDataset
def mock_load_dataset(dataset_name, dataset_config=None, split=None):
return MockDataset(size=20)
ARGS = {
'dataset_name': 'math_dataset',
'dataset_config': 'algebra_2',
'dataset_split': 'train',
'num_problems': 5,
'sample_start': 0,
}
# Patch load_dataset before the function is called
import sys
sys.modules['datasets'] = Mock()
sys.modules['datasets'].load_dataset = mock_load_dataset
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- datasets, numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_prm_csvs |
Ran ran; our contract draft was wrong, not the code |
monasch/llm-monitor/python_scripts/merge_signal_label.py code served (permissive licence) · get_code("4261673306c32175") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import pandas as pd
import tempfile
from pathlib import Path
# Create temporary directory with mock CSV files
tmpdir = tempfile.mkdtemp()
base_path = Path(tmpdir)
# Create mock CSV files for a subset of datasets
mock_data_algebra = pd.DataFrame({
'uq_problem_idx': [1, 2, 3],
'score': [0.8, 0.9, 0.7],
'problem': ['prob1', 'prob2', 'prob3']
})
mock_data_geometry = pd.DataFrame({
'uq_problem_idx': [4, 5],
'score': [0.85, 0.75],
'problem': ['prob4', 'prob5']
})
# Write mock CSVs
csv_path_algebra = base_path / 'scored_gpt4_algebra_prm_v1.csv'
csv_path_geometry = base_path / 'scored_gpt4_geometry_prm_v1.csv'
mock_data_algebra.to_csv(csv_path_algebra, index=False)
mock_data_geometry.to_csv(csv_path_geometry, index=False)
ARGS = {
'prm_path': str(base_path),
'gen_model': 'gpt4',
'prm_model': 'prm_v1',
'dataconfig': 'all'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_trace |
Ran ran; our contract draft was wrong, not the code |
monasch/llm-monitor/python_scripts/generate_reasoning.py code served (permissive licence) · get_code("145fa20a8b0e8826") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import os
# Create a temporary file with the expected format for load_trace
temp_dir = tempfile.mkdtemp()
temp_file = os.path.join(temp_dir, "trace.txt")
# Write a valid trace file with all the expected sections
trace_content = """PROBLEM:
What is 2 + 2?
GENERATED REASONING:
We need to add two numbers together.
The first number is 2.
The second number is 2.
Adding them gives us 4.
GENERATOR: gpt-4
DATASET: math_dataset
PROBLEM_IDX: 42
"""
with open(temp_file, 'w', encoding='utf-8') as f:
f.write(trace_content)
ARGS = {
'path': temp_file
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_trace |
Ran ran; our contract draft was wrong, not the code |
monasch/llm-monitor/python_scripts/get_signal_prm.py code served (permissive licence) · get_code("2abcced0fca476c4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import os
# Create a temporary trace file with the expected format
temp_dir = tempfile.mkdtemp()
trace_path = os.path.join(temp_dir, "trace.txt")
trace_content = """PROBLEM:
Solve the quadratic equation x^2 + 2x + 1 = 0
GENERATED REASONING:
We can factor this as (x + 1)^2 = 0, so x = -1 is the solution.
GENERATOR: gpt-3.5-turbo
DATASET: math_problems
PROBLEM_IDX: 42
"""
with open(trace_path, 'w', encoding='utf-8') as f:
f.write(trace_content)
ARGS = {
'path': trace_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_trace |
Ran ran; our contract draft was wrong, not the code |
monasch/llm-monitor/python_scripts/label_step.py code served (permissive licence) · get_code("631222e85b81ab62") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import os
# Create a temporary trace file with the expected format
temp_dir = tempfile.mkdtemp()
trace_file_path = os.path.join(temp_dir, "test_trace.txt")
trace_content = """PROBLEM_IDX: 42
DATASET: test_dataset
GENERATOR: test_generator
PROBLEM:
This is a test problem statement that describes a mathematical or logical challenge.
GENERATED REASONING:
This is the generated reasoning that explains the solution step by step.
GENERATOR: custom_generator
DATASET: custom_dataset
"""
with open(trace_file_path, 'w', encoding='utf-8') as f:
f.write(trace_content)
ARGS = {
'path': trace_file_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| normalise |
Ran ran; our contract draft was wrong, not the code |
monasch/llm-monitor/python_scripts/add_final_label.py code served (permissive licence) · get_code("b680b953c3c12edc") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def normalise(expr: str) -> str:
"""Normalise a LaTeX expression for loose equality comparison."""
if expr is None:
return expr
expr = re.sub('\\s+', '', expr)
expr = expr.replace('\\dfrac', '\\frac')
return expr
ARGS = {
"expr": "\\dfrac{x}{y} + \\frac{a}{b}"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| parse_critique_file |
Ran ran; our contract draft was wrong, not the code |
monasch/llm-monitor/python_scripts/merge_signal_label.py code served (permissive licence) · get_code("31622f594024a084") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from pathlib import Path
import tempfile
# Create a temporary directory and a test critique file
temp_dir = tempfile.TemporaryDirectory()
temp_path = Path(temp_dir.name)
# Create a sample critique file with the expected format
critique_content = """GENERATOR: gpt-4
DATASET: math_dataset
NUM_STEPS: 3
BOXED_MATCH: 1
Some problem content here...
RAW_CRITIC_RESPONSE:
This is the critic's response.
The answer is \\boxed{[1, 0, 1]}
"""
critique_file = temp_path / "critique_problem_42.txt"
critique_file.write_text(critique_content, encoding='utf-8')
ARGS = {
'path': critique_file
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| sanitise_model_name |
Ran ran; our contract draft was wrong, not the code |
monasch/llm-monitor/python_scripts/generate_reasoning.py code served (permissive licence) · get_code("b979825109ded6db") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def sanitise_model_name(name: str) -> str:
short = name.split('/')[-1]
return re.sub('[^\\w\\-.]', '_', short)
ARGS = {
'name': 'huggingface/bert-base-uncased'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| sanitise_turns |
Ran ran; our contract draft was wrong, not the code |
monasch/llm-monitor/python_scripts/llama_guard.py code served (permissive licence) · get_code("29ed975da6f27a27") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from tqdm import tqdm
HUMAN_ROLE = 'user'
# Construct test arguments for sanitise_turns
turns_basic = [
{'role': 'user', 'content': 'Hello'},
{'role': 'assistant', 'content': 'Hi there'},
{'role': 'user', 'content': 'How are you?'},
]
turns_with_leading_assistant = [
{'role': 'assistant', 'content': 'Some intro'},
{'role': 'assistant', 'content': 'More intro'},
{'role': 'user', 'content': 'Hello'},
{'role': 'assistant', 'content': 'Response'},
]
turns_with_consecutive_same_role = [
{'role': 'user', 'content': 'First question'},
{'role': 'user', 'content': 'Second question'},
{'role': 'assistant', 'content': 'Answer'},
{'role': 'assistant', 'content': 'Follow-up'},
]
turns_empty = []
turns_only_assistant = [
{'role': 'assistant', 'content': 'Just assistant'},
]
ARGS = {
'turns': turns_with_consecutive_same_role,
'conv_id': 42,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| parse_conversation |
Not yet run unverified (harvested, no recorded run) |
monasch/llm-monitor/python_scripts/llama_guard.py code served (permissive licence) · get_code("309d7502de75dc46") How it ran |