| align_two_jsonl |
Ran ran; our contract draft was wrong, not the code |
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/utils_pairwise.py code served (permissive licence) · get_code("f2553208ec910158") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
import os
from typing import List, Dict, Any
# Create temporary JSONL files with test data
temp_dir = tempfile.mkdtemp()
# Create file_a with some instructions and responses
file_a = os.path.join(temp_dir, "file_a.jsonl")
with open(file_a, 'w', encoding='utf-8') as f:
f.write(json.dumps({"instruction": "What is 2+2?", "response": "The answer is 4"}) + "\n")
f.write(json.dumps({"instruction": "Hello world", "response": "Hi there"}) + "\n")
f.write(json.dumps({"instruction": "Test question", "response": "Test response A"}) + "\n")
# Create file_b with overlapping and different instructions
file_b = os.path.join(temp_dir, "file_b.jsonl")
with open(file_b, 'w', encoding='utf-8') as f:
f.write(json.dumps({"instruction": "What is 2+2?", "response": "It equals four"}) + "\n")
f.write(json.dumps({"instruction": "Hello world", "response": "Hello there"}) + "\n")
f.write(json.dumps({"instruction": "Different question", "response": "Different response"}) + "\n")
ARGS = {
"file_a": file_a,
"file_b": file_b,
"instruction_key": "instruction",
"response_key": "response"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| collect_error_examples |
Ran ran; our contract draft was wrong, not the code |
ZDCSlab/Rubrics-as-an-Attack-Surface/rubrics_search/search/optimize.py code served (permissive licence) · get_code("e7004c97af9d006f") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import pandas as pd
import random
from typing import List, Dict, Any
# Create a small target dataframe
target_df = pd.DataFrame({
'ground_truth': ['A', 'B', 'A'],
'prompt': ['What is 2+2?', 'What is the capital of France?', 'Who wrote Hamlet?'],
'response_A': ['4', 'Paris', 'Shakespeare'],
'response_B': ['5', 'London', 'Marlowe'],
})
target_df.index = [0, 1, 2]
# Create a small benchmark dataframe
bench_df = pd.DataFrame({
'ground_truth': ['B', 'A'],
'prompt': ['What is Python?', 'What is 10/2?'],
'response_A': ['A language', '4'],
'response_B': ['A snake', '5'],
})
bench_df.index = [0, 1]
# Create target rows with some errors (decision != ground_truth for target errors)
target_rows = [
{
'data_index': 0,
'decision': 'B', # Error: gold is 'A', decision is 'B'
'feedback': 'Response B seemed better.'
},
{
'data_index': 1,
'decision': 'A', # Error: gold is 'B', decision is 'A'
'feedback': 'Response A had merit.'
},
{
'data_index': 2,
'decision': 'A', # Correct: gold is 'A', decision is 'A'
'feedback': 'Response A is correct.'
},
]
# Create benchmark rows with errors (decision == ground_truth for bench errors to be skipped, or != for real errors)
bench_rows = [
{
'data_index': 0,
'decision': 'A', # Error: gold is 'B', decision is 'A'
'feedback': 'Response A seemed reasonable.'
},
{
'data_index': 1,
'decision': 'B', # Error: gold is 'A', decision is 'B'
'feedback': 'Response B looked good.'
},
]
ARGS = {
'step': 1,
'bench_df': bench_df,
'target_df': target_df,
'bench_rows': bench_rows,
'target_rows': target_rows,
'prompt_col': 'prompt',
'a_col': 'response_A',
'b_col': 'response_B',
'target_num': 3,
'bench_num': 2,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| compute_stats |
Ran ran; our contract draft was wrong, not the code |
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/analyze.py code served (permissive licence) · get_code("1db28a5ee819cd01") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
# Define the function
from typing import Any, Dict, List, Tuple
def compute_stats(winners: List[str]) -> Dict[str, Any]:
a_wins = sum((1 for w in winners if w == 'A'))
b_wins = sum((1 for w in winners if w == 'B'))
ties = sum((1 for w in winners if w == 'TIE'))
n = len(winners)
denom = a_wins + b_wins
return {'n': n, 'a_wins': a_wins, 'b_wins': b_wins, 'ties': ties, 'a_winrate_excluding_ties': a_wins / denom if denom > 0 else 0.0, 'a_winrate_with_ties_half': (a_wins + 0.5 * ties) / n if n > 0 else 0.0}
# Setup valid arguments
ARGS = {
'winners': ['A', 'B', 'A', 'TIE', 'B', 'A']
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| decide_winner |
Ran ran; our contract draft was wrong, not the code |
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/analyze.py code served (permissive licence) · get_code("ccb26c1a58a2eb7c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# The function takes:
# - score_a: float
# - score_b: float
# - direction: str
# - eps: float (optional, defaults to 1e-08)
# Test case 1: higher direction with A winning
ARGS = {
'score_a': 0.75,
'score_b': 0.5,
'direction': 'higher',
'eps': 1e-08
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| eval_on_test |
Ran ran; our contract draft was wrong, not the code |
ZDCSlab/Rubrics-as-an-Attack-Surface/rubrics_selection/selection.py code served (permissive licence) · get_code("156a7a83c700039e") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import pandas as pd
from typing import Optional, Dict, Tuple
# Create sample DataFrames with the required structure
bench_test = pd.DataFrame({
'prompt_id': ['seed', 'prompt_1', 'prompt_2'],
'acc': [0.85, 0.90, 0.75]
})
target_test = pd.DataFrame({
'prompt_id': ['seed', 'prompt_1', 'prompt_2'],
'acc': [0.80, 0.88, 0.70]
})
ARGS = {
'bench_test': bench_test,
'target_test': target_test,
'prompt_id': 'prompt_1',
'seed_id': 'seed'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| format_beaver_conversation |
Ran ran; our contract draft was wrong, not the code |
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/scores.py code served (permissive licence) · get_code("d05a5f62b77b232a") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
ARGS = {
'inst': 'What is machine learning?',
'resp': 'Machine learning is a subset of artificial intelligence that enables systems to learn and improve from experience.'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| format_error_examples_for_prompt |
Ran ran; our contract draft was wrong, not the code |
ZDCSlab/Rubrics-as-an-Attack-Surface/rubrics_search/search/optimize.py code served (permissive licence) · get_code("1974826b28a222b3") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
from dataclasses import dataclass
from typing import List
torch.manual_seed(0)
@dataclass
class ErrorExample:
"""Container for a single error case used to refine the rubric."""
dataset: str
error_type: str
prompt: str
response_A: str
response_B: str
decision: str
feedback: str
desired_behavior: str
# Create sample error examples
example_1 = ErrorExample(
dataset="math_eval",
error_type="calculation_error",
prompt="What is 2 + 2?",
response_A="The answer is 4.",
response_B="The answer is 5.",
decision="A",
feedback="Response A correctly identifies the sum as 4.",
desired_behavior="Judge should prefer mathematically correct answers."
)
example_2 = ErrorExample(
dataset="code_review",
error_type="style_violation",
prompt="Write a Python function to sort a list.",
response_A="def sort_list(lst):\n return sorted(lst)",
response_B="def sort_list(lst):\n # This function sorts\n x=sorted(lst)\n return x",
decision="A",
feedback="Response A is more concise and follows PEP 8 conventions.",
desired_behavior="Judge should prefer clean, Pythonic code over verbose implementations."
)
example_3 = ErrorExample(
dataset="translation",
error_type="semantic_drift",
prompt="Translate 'Je suis heureux' to English.",
response_A="I am happy.",
response_B="I am sad.",
decision="A",
feedback="Response A correctly translates the French phrase.",
desired_behavior="Judge must ensure translations preserve meaning accurately."
)
examples = [example_1, example_2, example_3]
ARGS = {
"examples": examples
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_rubric_id |
Ran ran; our contract draft was wrong, not the code |
ZDCSlab/Rubrics-as-an-Attack-Surface/rubrics_search/search/select_final.py code served (permissive licence) · get_code("40244ed50a2b0259") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
# The function get_rubric_id takes a dict 'r' and uses .get() method
# to retrieve values by keys: 'prompt_id', 'rubric_index', or 'id'
# It returns the first non-None value found in that order
ARGS = {
'r': {
'prompt_id': 'test_prompt_123',
'rubric_index': 42,
'id': 'fallback_id'
}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_gen_file_bon |
Ran ran; our contract draft was wrong, not the code |
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/scores.py code served (permissive licence) · get_code("9b9ceb2f80a80ac0") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import json
import os
# Create a temporary JSONL file with test data
temp_file = tempfile.NamedTemporaryFile(mode='w', suffix='.jsonl', delete=False, encoding='utf-8')
temp_path = temp_file.name
# Write test data to the temporary file
test_data = [
{
'instruction': 'What is 2+2?',
'responses': ['4', 'The answer is 4', 'Two plus two equals four']
},
{
'instruction': 'What is the capital of France?',
'responses': ['Paris']
},
{
'instruction': 'Tell me a joke',
'response': 'Why did the chicken cross the road?'
}
]
for item in test_data:
temp_file.write(json.dumps(item) + '\n')
temp_file.close()
ARGS = {
'path': temp_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_jsonl |
Ran ran; our contract draft was wrong, not the code |
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/utils_pairwise.py code served (permissive licence) · get_code("400eeff90e1ba381") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
import os
# Create a temporary JSONL file with sample data
temp_dir = tempfile.mkdtemp()
jsonl_path = os.path.join(temp_dir, "test_data.jsonl")
# Write sample JSONL data
sample_data = [
{"id": 1, "text": "hello world", "value": 42},
{"id": 2, "text": "foo bar", "value": 99},
{"id": 3, "text": "test data", "value": 123}
]
with open(jsonl_path, 'w', encoding='utf-8') as f:
for record in sample_data:
f.write(json.dumps(record) + '\n')
ARGS = {
"path": jsonl_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_rubrics_jsonl |
Ran ran; our contract draft was wrong, not the code |
ZDCSlab/Rubrics-as-an-Attack-Surface/rubrics_search/search/select_final.py code served (permissive licence) · get_code("555478893737fb22") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import json
import os
# Create a temporary directory and file for testing
tmpdir = tempfile.mkdtemp()
rubrics_path = os.path.join(tmpdir, "rubrics.jsonl")
# Write valid JSONL data with rubrics
rubrics_data = [
{"id": 1, "name": "rubric_1", "criteria": "test criteria 1"},
{"id": 2, "name": "rubric_2", "criteria": "test criteria 2"},
{"id": 3, "name": "rubric_3", "criteria": "test criteria 3"}
]
with open(rubrics_path, 'w', encoding='utf-8') as f:
for rubric in rubrics_data:
f.write(json.dumps(rubric) + '\n')
ARGS = {
"path": rubrics_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| parse_verdict |
Ran ran; our contract draft was wrong, not the code |
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/utils_pairwise.py code served (permissive licence) · get_code("f57b8c707f9f56df") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
_VERDICT_RE = re.compile('\\[\\[\\s*([abAB])\\s*\\]\\]')
def parse_verdict(text: str) -> str:
"""Return 'A'/'B'/'TIE' from model output."""
if not text:
return 'TIE'
m = _VERDICT_RE.search(text)
if m:
return m.group(1).upper()
t = text.upper()
if '[[A]]' in t:
return 'A'
if '[[B]]' in t:
return 'B'
return 'TIE'
ARGS = {
'text': '[[A]]'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| pick_best_in_subset |
Ran ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT) |
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/select_best.py code served (permissive licence) · get_code("23262afede4a7d29") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# Construct arguments for pick_best_in_subset
scores = [0.1, 0.9, 0.5, 0.3, 0.7]
idxs = [0, 1, 2, 3, 4]
direction = "higher"
ARGS = {
"scores": scores,
"idxs": idxs,
"direction": direction
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| read_jsonl |
Ran ran; our contract draft was wrong, not the code |
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/scores.py code served (permissive licence) · get_code("d97bc4cdc990894c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
import os
# Create a temporary JSONL file with valid JSON lines
temp_dir = tempfile.mkdtemp()
temp_file = os.path.join(temp_dir, "test.jsonl")
with open(temp_file, 'w', encoding='utf-8') as f:
f.write('{"name": "Alice", "age": 30}\n')
f.write('{"name": "Bob", "age": 25}\n')
f.write('\n') # empty line to test that it's skipped
f.write('{"name": "Charlie", "age": 35}\n')
ARGS = {
'path': temp_file
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| read_jsonl |
Ran ran; our contract draft was wrong, not the code |
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/analyze.py code served (permissive licence) · get_code("ffbf5293f2483abe") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
import os
# Create a temporary JSONL file with test data
temp_dir = tempfile.mkdtemp()
test_jsonl_path = os.path.join(temp_dir, 'test.jsonl')
# Write test JSONL data
test_data = [
{'id': 1, 'text': 'hello world'},
{'id': 2, 'text': 'foo bar'},
{'id': 3, 'value': 42}
]
with open(test_jsonl_path, 'w', encoding='utf-8') as f:
for item in test_data:
f.write(json.dumps(item) + '\n')
# Define ARGS mapping the parameter to its value
ARGS = {
'path': test_jsonl_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| read_jsonl |
Ran ran; our contract draft was wrong, not the code |
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/generate.py code served (permissive licence) · get_code("f07107c7894990c1") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import json
import os
# Create a temporary JSONL file with valid JSON lines
temp_dir = tempfile.mkdtemp()
temp_jsonl_path = os.path.join(temp_dir, "test.jsonl")
# Write some valid JSONL data
test_data = [
{"id": 1, "text": "hello"},
{"id": 2, "text": "world"},
{"id": 3, "value": 42}
]
with open(temp_jsonl_path, 'w', encoding='utf-8') as f:
for item in test_data:
f.write(json.dumps(item) + '\n')
ARGS = {
"path": temp_jsonl_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| read_jsonl |
Ran ran; our contract draft was wrong, not the code |
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/select_best.py code served (permissive licence) · get_code("6a94611fb33432fd") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
import os
# Create a temporary JSONL file with valid JSON lines
temp_dir = tempfile.mkdtemp()
temp_file = os.path.join(temp_dir, "test.jsonl")
with open(temp_file, 'w', encoding='utf-8') as f:
f.write('{"key1": "value1", "number": 42}\n')
f.write('{"key2": "value2", "nested": {"inner": "data"}}\n')
f.write('\n') # Empty line to test skipping
f.write('{"key3": [1, 2, 3]}\n')
ARGS = {
'path': temp_file
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| sample_instructions_from_dataset |
Ran ran; our contract draft was wrong, not the code |
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/generate.py code served (permissive licence) · get_code("3e064fc5b4504d36") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import random
from transformers import AutoTokenizer
# Create a mock tokenizer that implements the encode method
class MockTokenizer:
def encode(self, text, add_special_tokens=False):
# Simple mock: split by spaces and return token indices
tokens = text.split()
return list(range(len(tokens)))
# Create dataset rows with instructions
dataset_rows = [
{
'instruction': 'What is the capital of France?',
'output': 'Paris'
},
{
'instruction': 'How do you make coffee?',
'output': 'Brew with hot water'
},
{
'instruction': 'Explain quantum computing in one sentence.',
'output': 'A complex topic'
},
{
'instruction': 'Tell me a joke.',
'output': 'Why did the chicken cross the road?'
},
{
'instruction': 'What is 2+2?',
'output': '4'
},
{
'not_instruction': 'This has no instruction field'
},
{
'instruction': 123 # Not a string, should be skipped
}
]
tokenizer_for_len = MockTokenizer()
ARGS = {
'dataset_rows': dataset_rows,
'tokenizer_for_len': tokenizer_for_len,
'max_inst_tokens': 10,
'sample_size': 3,
'seed': 42
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- transformers — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| select_rubric |
Ran ran; our contract draft was wrong, not the code |
ZDCSlab/Rubrics-as-an-Attack-Surface/rubrics_selection/selection.py code served (permissive licence) · get_code("6ee262d896a4c9ff") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import pandas as pd
# Create valid DataFrames for bench_val and target_val
bench_val = pd.DataFrame({
'prompt_id': ['seed', 'prompt_1', 'prompt_2', 'prompt_3'],
'acc': [0.75, 0.80, 0.70, 0.85]
})
target_val = pd.DataFrame({
'prompt_id': ['seed', 'prompt_1', 'prompt_2', 'prompt_3'],
'acc': [0.75, 0.65, 0.60, 0.78]
})
ARGS = {
'bench_val': bench_val,
'target_val': target_val,
'seed_id': 'seed',
'mode': 'min_target_acc'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| subset_indices |
Ran ran, honoured the contract we drafted |
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/select_best.py code served (permissive licence) · get_code("de5d416266f706a6") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import random
# Create a random number generator
rng = random.Random(42)
# Define arguments for subset_indices
ARGS = {
'total': 10,
'k': 3,
'mode': 'random',
'rng': rng
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family int — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_token_len |
Not yet run unverified (harvested, no recorded run) |
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/generate.py code served (permissive licence) · get_code("26bdfbc372c5a3a9") How it ran |