SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2602.13576 · 2026

Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges

Zhun Deng, He Sun, Yizhong Wang, Zhiwei Steven Wu, Ruomeng Ding, Yifei Pang

arXiv · PDF · Open in the Atlas

Code that ran

20 of 21 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 1 honoured the contract we drafted, 0 violated it and 0 have no contract verdict; 19 failures of our instrument; 1 no recorded run), measured 2026-11-22; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2602.13576.json · badge: /api/ran/2602.13576.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
ZDCSlab/Rubrics-as-an-Attack-Surface canonical (classifier label)
classifier basis: census
20 ran · 1 no recorded run
FunctionStatusWhere it lives
align_two_jsonl Ran
ran; our contract draft was wrong, not the code
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/utils_pairwise.py
code served (permissive licence) · get_code("f2553208ec910158")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
import os
from typing import List, Dict, Any

# Create temporary JSONL files with test data
temp_dir = tempfile.mkdtemp()

# Create file_a with some instructions and responses
file_a = os.path.join(temp_dir, "file_a.jsonl")
with open(file_a, 'w', encoding='utf-8') as f:
    f.write(json.dumps({"instruction": "What is 2+2?", "response": "The answer is 4"}) + "\n")
    f.write(json.dumps({"instruction": "Hello world", "response": "Hi there"}) + "\n")
    f.write(json.dumps({"instruction": "Test question", "response": "Test response A"}) + "\n")

# Create file_b with overlapping and different instructions
file_b = os.path.join(temp_dir, "file_b.jsonl")
with open(file_b, 'w', encoding='utf-8') as f:
    f.write(json.dumps({"instruction": "What is 2+2?", "response": "It equals four"}) + "\n")
    f.write(json.dumps({"instruction": "Hello world", "response": "Hello there"}) + "\n")
    f.write(json.dumps({"instruction": "Different question", "response": "Different response"}) + "\n")

ARGS = {
    "file_a": file_a,
    "file_b": file_b,
    "instruction_key": "instruction",
    "response_key": "response"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
collect_error_examples Ran
ran; our contract draft was wrong, not the code
ZDCSlab/Rubrics-as-an-Attack-Surface/rubrics_search/search/optimize.py
code served (permissive licence) · get_code("e7004c97af9d006f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd
import random
from typing import List, Dict, Any

# Create a small target dataframe
target_df = pd.DataFrame({
    'ground_truth': ['A', 'B', 'A'],
    'prompt': ['What is 2+2?', 'What is the capital of France?', 'Who wrote Hamlet?'],
    'response_A': ['4', 'Paris', 'Shakespeare'],
    'response_B': ['5', 'London', 'Marlowe'],
})
target_df.index = [0, 1, 2]

# Create a small benchmark dataframe
bench_df = pd.DataFrame({
    'ground_truth': ['B', 'A'],
    'prompt': ['What is Python?', 'What is 10/2?'],
    'response_A': ['A language', '4'],
    'response_B': ['A snake', '5'],
})
bench_df.index = [0, 1]

# Create target rows with some errors (decision != ground_truth for target errors)
target_rows = [
    {
        'data_index': 0,
        'decision': 'B',  # Error: gold is 'A', decision is 'B'
        'feedback': 'Response B seemed better.'
    },
    {
        'data_index': 1,
        'decision': 'A',  # Error: gold is 'B', decision is 'A'
        'feedback': 'Response A had merit.'
    },
    {
        'data_index': 2,
        'decision': 'A',  # Correct: gold is 'A', decision is 'A'
        'feedback': 'Response A is correct.'
    },
]

# Create benchmark rows with errors (decision == ground_truth for bench errors to be skipped, or != for real errors)
bench_rows = [
    {
        'data_index': 0,
        'decision': 'A',  # Error: gold is 'B', decision is 'A'
        'feedback': 'Response A seemed reasonable.'
    },
    {
        'data_index': 1,
        'decision': 'B',  # Error: gold is 'A', decision is 'B'
        'feedback': 'Response B looked good.'
    },
]

ARGS = {
    'step': 1,
    'bench_df': bench_df,
    'target_df': target_df,
    'bench_rows': bench_rows,
    'target_rows': target_rows,
    'prompt_col': 'prompt',
    'a_col': 'response_A',
    'b_col': 'response_B',
    'target_num': 3,
    'bench_num': 2,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
compute_stats Ran
ran; our contract draft was wrong, not the code
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/analyze.py
code served (permissive licence) · get_code("1db28a5ee819cd01")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

# Define the function
from typing import Any, Dict, List, Tuple

def compute_stats(winners: List[str]) -> Dict[str, Any]:
    a_wins = sum((1 for w in winners if w == 'A'))
    b_wins = sum((1 for w in winners if w == 'B'))
    ties = sum((1 for w in winners if w == 'TIE'))
    n = len(winners)
    denom = a_wins + b_wins
    return {'n': n, 'a_wins': a_wins, 'b_wins': b_wins, 'ties': ties, 'a_winrate_excluding_ties': a_wins / denom if denom > 0 else 0.0, 'a_winrate_with_ties_half': (a_wins + 0.5 * ties) / n if n > 0 else 0.0}

# Setup valid arguments
ARGS = {
    'winners': ['A', 'B', 'A', 'TIE', 'B', 'A']
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
decide_winner Ran
ran; our contract draft was wrong, not the code
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/analyze.py
code served (permissive licence) · get_code("ccb26c1a58a2eb7c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function takes:
# - score_a: float
# - score_b: float
# - direction: str
# - eps: float (optional, defaults to 1e-08)

# Test case 1: higher direction with A winning
ARGS = {
    'score_a': 0.75,
    'score_b': 0.5,
    'direction': 'higher',
    'eps': 1e-08
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
eval_on_test Ran
ran; our contract draft was wrong, not the code
ZDCSlab/Rubrics-as-an-Attack-Surface/rubrics_selection/selection.py
code served (permissive licence) · get_code("156a7a83c700039e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd
from typing import Optional, Dict, Tuple

# Create sample DataFrames with the required structure
bench_test = pd.DataFrame({
    'prompt_id': ['seed', 'prompt_1', 'prompt_2'],
    'acc': [0.85, 0.90, 0.75]
})

target_test = pd.DataFrame({
    'prompt_id': ['seed', 'prompt_1', 'prompt_2'],
    'acc': [0.80, 0.88, 0.70]
})

ARGS = {
    'bench_test': bench_test,
    'target_test': target_test,
    'prompt_id': 'prompt_1',
    'seed_id': 'seed'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
format_beaver_conversation Ran
ran; our contract draft was wrong, not the code
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/scores.py
code served (permissive licence) · get_code("d05a5f62b77b232a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

ARGS = {
    'inst': 'What is machine learning?',
    'resp': 'Machine learning is a subset of artificial intelligence that enables systems to learn and improve from experience.'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
format_error_examples_for_prompt Ran
ran; our contract draft was wrong, not the code
ZDCSlab/Rubrics-as-an-Attack-Surface/rubrics_search/search/optimize.py
code served (permissive licence) · get_code("1974826b28a222b3")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from dataclasses import dataclass
from typing import List

torch.manual_seed(0)

@dataclass
class ErrorExample:
    """Container for a single error case used to refine the rubric."""
    dataset: str
    error_type: str
    prompt: str
    response_A: str
    response_B: str
    decision: str
    feedback: str
    desired_behavior: str

# Create sample error examples
example_1 = ErrorExample(
    dataset="math_eval",
    error_type="calculation_error",
    prompt="What is 2 + 2?",
    response_A="The answer is 4.",
    response_B="The answer is 5.",
    decision="A",
    feedback="Response A correctly identifies the sum as 4.",
    desired_behavior="Judge should prefer mathematically correct answers."
)

example_2 = ErrorExample(
    dataset="code_review",
    error_type="style_violation",
    prompt="Write a Python function to sort a list.",
    response_A="def sort_list(lst):\n    return sorted(lst)",
    response_B="def sort_list(lst):\n    # This function sorts\n    x=sorted(lst)\n    return x",
    decision="A",
    feedback="Response A is more concise and follows PEP 8 conventions.",
    desired_behavior="Judge should prefer clean, Pythonic code over verbose implementations."
)

example_3 = ErrorExample(
    dataset="translation",
    error_type="semantic_drift",
    prompt="Translate 'Je suis heureux' to English.",
    response_A="I am happy.",
    response_B="I am sad.",
    decision="A",
    feedback="Response A correctly translates the French phrase.",
    desired_behavior="Judge must ensure translations preserve meaning accurately."
)

examples = [example_1, example_2, example_3]

ARGS = {
    "examples": examples
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_rubric_id Ran
ran; our contract draft was wrong, not the code
ZDCSlab/Rubrics-as-an-Attack-Surface/rubrics_search/search/select_final.py
code served (permissive licence) · get_code("40244ed50a2b0259")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# The function get_rubric_id takes a dict 'r' and uses .get() method
# to retrieve values by keys: 'prompt_id', 'rubric_index', or 'id'
# It returns the first non-None value found in that order

ARGS = {
    'r': {
        'prompt_id': 'test_prompt_123',
        'rubric_index': 42,
        'id': 'fallback_id'
    }
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_gen_file_bon Ran
ran; our contract draft was wrong, not the code
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/scores.py
code served (permissive licence) · get_code("9b9ceb2f80a80ac0")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import tempfile
import json
import os

# Create a temporary JSONL file with test data
temp_file = tempfile.NamedTemporaryFile(mode='w', suffix='.jsonl', delete=False, encoding='utf-8')
temp_path = temp_file.name

# Write test data to the temporary file
test_data = [
    {
        'instruction': 'What is 2+2?',
        'responses': ['4', 'The answer is 4', 'Two plus two equals four']
    },
    {
        'instruction': 'What is the capital of France?',
        'responses': ['Paris']
    },
    {
        'instruction': 'Tell me a joke',
        'response': 'Why did the chicken cross the road?'
    }
]

for item in test_data:
    temp_file.write(json.dumps(item) + '\n')

temp_file.close()

ARGS = {
    'path': temp_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_jsonl Ran
ran; our contract draft was wrong, not the code
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/utils_pairwise.py
code served (permissive licence) · get_code("400eeff90e1ba381")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
import os

# Create a temporary JSONL file with sample data
temp_dir = tempfile.mkdtemp()
jsonl_path = os.path.join(temp_dir, "test_data.jsonl")

# Write sample JSONL data
sample_data = [
    {"id": 1, "text": "hello world", "value": 42},
    {"id": 2, "text": "foo bar", "value": 99},
    {"id": 3, "text": "test data", "value": 123}
]

with open(jsonl_path, 'w', encoding='utf-8') as f:
    for record in sample_data:
        f.write(json.dumps(record) + '\n')

ARGS = {
    "path": jsonl_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_rubrics_jsonl Ran
ran; our contract draft was wrong, not the code
ZDCSlab/Rubrics-as-an-Attack-Surface/rubrics_search/search/select_final.py
code served (permissive licence) · get_code("555478893737fb22")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import tempfile
import json
import os

# Create a temporary directory and file for testing
tmpdir = tempfile.mkdtemp()
rubrics_path = os.path.join(tmpdir, "rubrics.jsonl")

# Write valid JSONL data with rubrics
rubrics_data = [
    {"id": 1, "name": "rubric_1", "criteria": "test criteria 1"},
    {"id": 2, "name": "rubric_2", "criteria": "test criteria 2"},
    {"id": 3, "name": "rubric_3", "criteria": "test criteria 3"}
]

with open(rubrics_path, 'w', encoding='utf-8') as f:
    for rubric in rubrics_data:
        f.write(json.dumps(rubric) + '\n')

ARGS = {
    "path": rubrics_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
parse_verdict Ran
ran; our contract draft was wrong, not the code
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/utils_pairwise.py
code served (permissive licence) · get_code("f57b8c707f9f56df")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

_VERDICT_RE = re.compile('\\[\\[\\s*([abAB])\\s*\\]\\]')

def parse_verdict(text: str) -> str:
    """Return 'A'/'B'/'TIE' from model output."""
    if not text:
        return 'TIE'
    m = _VERDICT_RE.search(text)
    if m:
        return m.group(1).upper()
    t = text.upper()
    if '[[A]]' in t:
        return 'A'
    if '[[B]]' in t:
        return 'B'
    return 'TIE'


ARGS = {
    'text': '[[A]]'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
pick_best_in_subset Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/select_best.py
code served (permissive licence) · get_code("23262afede4a7d29")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Construct arguments for pick_best_in_subset
scores = [0.1, 0.9, 0.5, 0.3, 0.7]
idxs = [0, 1, 2, 3, 4]
direction = "higher"

ARGS = {
    "scores": scores,
    "idxs": idxs,
    "direction": direction
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
read_jsonl Ran
ran; our contract draft was wrong, not the code
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/scores.py
code served (permissive licence) · get_code("d97bc4cdc990894c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
import os

# Create a temporary JSONL file with valid JSON lines
temp_dir = tempfile.mkdtemp()
temp_file = os.path.join(temp_dir, "test.jsonl")

with open(temp_file, 'w', encoding='utf-8') as f:
    f.write('{"name": "Alice", "age": 30}\n')
    f.write('{"name": "Bob", "age": 25}\n')
    f.write('\n')  # empty line to test that it's skipped
    f.write('{"name": "Charlie", "age": 35}\n')

ARGS = {
    'path': temp_file
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
read_jsonl Ran
ran; our contract draft was wrong, not the code
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/analyze.py
code served (permissive licence) · get_code("ffbf5293f2483abe")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
import os

# Create a temporary JSONL file with test data
temp_dir = tempfile.mkdtemp()
test_jsonl_path = os.path.join(temp_dir, 'test.jsonl')

# Write test JSONL data
test_data = [
    {'id': 1, 'text': 'hello world'},
    {'id': 2, 'text': 'foo bar'},
    {'id': 3, 'value': 42}
]

with open(test_jsonl_path, 'w', encoding='utf-8') as f:
    for item in test_data:
        f.write(json.dumps(item) + '\n')

# Define ARGS mapping the parameter to its value
ARGS = {
    'path': test_jsonl_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
read_jsonl Ran
ran; our contract draft was wrong, not the code
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/generate.py
code served (permissive licence) · get_code("f07107c7894990c1")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import tempfile
import json
import os

# Create a temporary JSONL file with valid JSON lines
temp_dir = tempfile.mkdtemp()
temp_jsonl_path = os.path.join(temp_dir, "test.jsonl")

# Write some valid JSONL data
test_data = [
    {"id": 1, "text": "hello"},
    {"id": 2, "text": "world"},
    {"id": 3, "value": 42}
]

with open(temp_jsonl_path, 'w', encoding='utf-8') as f:
    for item in test_data:
        f.write(json.dumps(item) + '\n')

ARGS = {
    "path": temp_jsonl_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
read_jsonl Ran
ran; our contract draft was wrong, not the code
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/select_best.py
code served (permissive licence) · get_code("6a94611fb33432fd")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
import os

# Create a temporary JSONL file with valid JSON lines
temp_dir = tempfile.mkdtemp()
temp_file = os.path.join(temp_dir, "test.jsonl")

with open(temp_file, 'w', encoding='utf-8') as f:
    f.write('{"key1": "value1", "number": 42}\n')
    f.write('{"key2": "value2", "nested": {"inner": "data"}}\n')
    f.write('\n')  # Empty line to test skipping
    f.write('{"key3": [1, 2, 3]}\n')

ARGS = {
    'path': temp_file
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
sample_instructions_from_dataset Ran
ran; our contract draft was wrong, not the code
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/generate.py
code served (permissive licence) · get_code("3e064fc5b4504d36")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import random
from transformers import AutoTokenizer

# Create a mock tokenizer that implements the encode method
class MockTokenizer:
    def encode(self, text, add_special_tokens=False):
        # Simple mock: split by spaces and return token indices
        tokens = text.split()
        return list(range(len(tokens)))

# Create dataset rows with instructions
dataset_rows = [
    {
        'instruction': 'What is the capital of France?',
        'output': 'Paris'
    },
    {
        'instruction': 'How do you make coffee?',
        'output': 'Brew with hot water'
    },
    {
        'instruction': 'Explain quantum computing in one sentence.',
        'output': 'A complex topic'
    },
    {
        'instruction': 'Tell me a joke.',
        'output': 'Why did the chicken cross the road?'
    },
    {
        'instruction': 'What is 2+2?',
        'output': '4'
    },
    {
        'not_instruction': 'This has no instruction field'
    },
    {
        'instruction': 123  # Not a string, should be skipped
    }
]

tokenizer_for_len = MockTokenizer()

ARGS = {
    'dataset_rows': dataset_rows,
    'tokenizer_for_len': tokenizer_for_len,
    'max_inst_tokens': 10,
    'sample_size': 3,
    'seed': 42
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
transformers — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
select_rubric Ran
ran; our contract draft was wrong, not the code
ZDCSlab/Rubrics-as-an-Attack-Surface/rubrics_selection/selection.py
code served (permissive licence) · get_code("6ee262d896a4c9ff")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd

# Create valid DataFrames for bench_val and target_val
bench_val = pd.DataFrame({
    'prompt_id': ['seed', 'prompt_1', 'prompt_2', 'prompt_3'],
    'acc': [0.75, 0.80, 0.70, 0.85]
})

target_val = pd.DataFrame({
    'prompt_id': ['seed', 'prompt_1', 'prompt_2', 'prompt_3'],
    'acc': [0.75, 0.65, 0.60, 0.78]
})

ARGS = {
    'bench_val': bench_val,
    'target_val': target_val,
    'seed_id': 'seed',
    'mode': 'min_target_acc'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
subset_indices Ran
ran, honoured the contract we drafted
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/select_best.py
code served (permissive licence) · get_code("de5d416266f706a6")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import random

# Create a random number generator
rng = random.Random(42)

# Define arguments for subset_indices
ARGS = {
    'total': 10,
    'k': 3,
    'mode': 'random',
    'rng': rng
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family int — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_token_len Not yet run
unverified (harvested, no recorded run)
ZDCSlab/Rubrics-as-an-Attack-Surface/downstream_eval/eval/generate.py
code served (permissive licence) · get_code("26bdfbc372c5a3a9")
How it ran

no invocation recorded.

Also present in other papers' code

Of 21 samples lifted from this paper's linked repositories, 21 carry a function identity and 0 do not. Of the 19 distinct functions with an identity, 0 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 0 later) and 19 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Evaluation and alignment pipelines for large language models increasingly rely on LLM-based judges, whose behavior is guided by natural-language rubrics and validated on benchmarks. We identify a previously under-recognized vulnerability in this workflow, which we term Rubric-Induced Preference Drift (RIPD). Even when rubric edits pass benchmark validation, they can still produce systematic and directional shifts in a judge's preferences on target domains. Because rubrics serve as a high-level decision interface, such drift can emerge from seemingly natural, criterion-preserving edits and remain difficult to detect through aggregate benchmark metrics or limited spot-checking. We further show this vulnerability can be exploited through rubric-based preference attacks, in which benchmark-compliant rubric edits steer judgments away from a fixed human or trusted reference on target domains, systematically inducing RIPD and reducing target-domain accuracy up to 9.5% (helpfulness) and 27.9% (harmlessness). When these judgments are used to generate preference labels for downstream post-training, the induced bias propagates through alignment pipelines and becomes internalized in trained policies. This leads to persistent and systematic drift in model behavior. Overall, our findings highlight evaluation rubrics as a sensitive and manipulable control interface, revealing a system-level alignment risk that extends beyond evaluator reliability alone. The code is available at: https://github.com/ZDCSlab/Rubrics-as-an-Attack-Surface. Warning: Certain sections may contain potentially harmful content that may not be appropriate for all readers.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2602.13576")
get_code_for_paper("2602.13576")
have("2602.13576")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2602.13576.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2602.13576.svg)](https://syntology.ai/paper/2602.13576)

Connect an agent — have() is free.