| build_answer_retry_messages |
Ran ran; our contract draft was wrong, not the code |
kaons-research/drift-bench/src/prompts.py code served (permissive licence) · get_code("6b3409ba60d6f4b6") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from typing import Any
# Build setup for build_answer_retry_messages function
# Define test data
prior_assistant_response = "Alice sits in seat 1, Bob in seat 2, Charlie in seat 3."
domain = "seating"
entities = ["Alice", "Bob", "Charlie"]
context = {
"categories": {
"color": ["red", "blue", "green"],
"animal": ["cat", "dog", "bird"],
"job": ["doctor", "lawyer", "engineer"]
}
}
ARGS = {
"prior_assistant_response": prior_assistant_response,
"domain": domain,
"entities": entities,
"context": context
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| build_truncation_retry_messages |
Ran ran; our contract draft was wrong, not the code |
kaons-research/drift-bench/src/prompts.py code served (permissive licence) · get_code("a9dc69f03fd54a01") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
# Build base_messages: a list of dicts with 'role' and 'content' keys
base_messages = [
{'role': 'user', 'content': 'What is the capital of France?'},
{'role': 'assistant', 'content': 'The capital of France is Paris.'},
{'role': 'user', 'content': 'Tell me more about it.'}
]
# Domain parameter: choose a valid domain
domain = 'logic_grid'
# Entities parameter: list of strings representing entities
entities = ['Alice', 'Bob', 'Charlie', 'Diana']
# Context parameter: dict with arbitrary string keys and values
context = {
'categories': {
'color': ['red', 'blue', 'green'],
'age': ['young', 'old'],
'job': ['teacher', 'doctor']
},
'clues': ['Alice is not red', 'Bob likes blue']
}
ARGS = {
'base_messages': base_messages,
'domain': domain,
'entities': entities,
'context': context
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| build_turn_messages |
Ran ran; our contract draft was wrong, not the code |
kaons-research/drift-bench/src/prompts.py code served (permissive licence) · get_code("e08375efdbb3fce2") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
ARGS = {
'method': 'ledger_only',
'conversation_history': [
{'role': 'user', 'content': 'Alice must sit in seat 1.'},
{'role': 'assistant', 'content': '{"solution": {"Alice": 1, "Bob": 2, "Charlie": 3}}'}
],
'new_user_message': 'Bob cannot sit adjacent to Alice.',
'domain': 'seating',
'entities': ['Alice', 'Bob', 'Charlie'],
'context': {
'table_shape': 'round',
'num_entities': 3
},
'ledger': [
{'entity': 'Alice', 'constraint': 'position', 'value': 1}
],
'repair_signal': None
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| build_user_message |
Ran ran; our contract draft was wrong, not the code |
kaons-research/drift-bench/src/generate_problems.py code served (permissive licence) · get_code("87aa5907f0cb48ee") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from typing import Any
# Construct a list of constraint dictionaries
# Each constraint dict needs an 'nl' key with a string value
constraints = [
{
'nl': 'The output should be valid JSON'
},
{
'nl': 'The response must be under 100 tokens'
},
{
'nl': 'All numbers should be positive integers.'
}
]
ARGS = {
'constraints': constraints
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| canonicalize_constraint |
Ran ran; our contract draft was wrong, not the code |
kaons-research/drift-bench/src/z3_checker.py code served (permissive licence) · get_code("421bced5ee970b0e") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
# Construct a valid constraint dictionary argument
constraint = {
'type': 'adjacent',
'args': ['entity_1', 'entity_2', 'extra_arg']
}
ARGS = {
'constraint': constraint
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| estimate_tokens_from_text |
Ran ran, honoured the contract we drafted |
kaons-research/drift-bench/src/run_experiment.py code served (permissive licence) · get_code("a23de49f4a4e2ca2") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
# The function only uses len() and basic arithmetic on a string
# So we just need to provide a text string
text = "This is a sample text to estimate tokens from."
ARGS = {
"text": text
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_json_block |
Ran ran; our contract draft was wrong, not the code |
kaons-research/drift-bench/src/extraction.py code served (permissive licence) · get_code("5727f4a3df316a5a") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import re
from typing import Any
ARGS = {
"text": """
<think>
Let me analyze this JSON extraction problem.
</think>
{
"name": "test",
"values": [1, 2, 3],
"nested": {
"key": "value"
}
}
"""
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| ledger_token_count |
Ran ran, honoured the contract we drafted |
kaons-research/drift-bench/src/run_experiment.py code served (permissive licence) · get_code("fa850eb8d5f90663") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
from typing import Any
def estimate_tokens_from_text(text: str) -> int:
return max(1, len(text) // 4)
def ledger_token_count(ledger: list[dict[str, Any]]) -> int:
if not ledger:
return 0
payload = json.dumps({'constraints': ledger}, ensure_ascii=True)
return estimate_tokens_from_text(payload)
# Construct a valid ledger argument
# The function uses ledger as a list of dicts that gets JSON serialized
ledger_value = [
{
"constraint_id": 1,
"type": "budget",
"value": 100
},
{
"constraint_id": 2,
"type": "deadline",
"value": "2024-12-31"
},
{
"constraint_id": 3,
"parameters": [1, 2, 3],
"metadata": {"source": "test"}
}
]
ARGS = {
"ledger": ledger_value
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_problem_gold |
Ran ran; our contract draft was wrong, not the code |
kaons-research/drift-bench/src/analyze.py code served (permissive licence) · get_code("e1dd1a7c6320d923") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
from pathlib import Path
# Create a temporary directory with test problem files
temp_dir = tempfile.TemporaryDirectory()
problems_dir = Path(temp_dir.name)
# Create first problem file
problem1 = {
"problem_id": "prob_001",
"split": "train",
"turns": [
{
"turn_number": "1",
"new_constraints": [
{"constraint_type": "linear", "value": 5}
]
},
{
"turn_number": "2",
"new_constraints": [
{"constraint_type": "nonlinear", "value": 10}
]
}
]
}
# Create second problem file
problem2 = {
"problem_id": "prob_002",
"split": "test",
"turns": [
{
"turn_number": "1",
"new_constraints": []
}
]
}
# Write problem files
with open(problems_dir / "problem_001.json", "w") as f:
json.dump(problem1, f)
with open(problems_dir / "problem_002.json", "w") as f:
json.dump(problem2, f)
ARGS = {
"problems_dir": problems_dir
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_problems |
Ran ran; our contract draft was wrong, not the code |
kaons-research/drift-bench/src/run_experiment.py code served (permissive licence) · get_code("4737d24ad6135c21") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
from pathlib import Path
from typing import Any
# Create temporary directory structure with test JSON files
temp_dir = tempfile.TemporaryDirectory()
problems_dir = Path(temp_dir.name)
# Create subdirectory for the split
split_name = "train"
split_dir = problems_dir / split_name
split_dir.mkdir(parents=True, exist_ok=True)
# Create test JSON files
test_problem_1 = {
"split": "train",
"id": "problem_1",
"data": "test data 1"
}
test_problem_2 = {
"split": "train",
"id": "problem_2",
"data": "test data 2"
}
test_problem_wrong_split = {
"split": "test",
"id": "problem_3",
"data": "wrong split"
}
# Write JSON files to the split directory
with open(split_dir / "problem_1.json", "w", encoding="utf-8") as f:
json.dump(test_problem_1, f)
with open(split_dir / "problem_2.json", "w", encoding="utf-8") as f:
json.dump(test_problem_2, f)
with open(split_dir / "problem_3.json", "w", encoding="utf-8") as f:
json.dump(test_problem_wrong_split, f)
ARGS = {
"problems_dir": problems_dir,
"split": "train"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_rows |
Ran ran; our contract draft was wrong, not the code |
kaons-research/drift-bench/src/analyze.py code served (permissive licence) · get_code("9f1ee38270e3cbda") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import sqlite3
import tempfile
# Create a temporary in-memory SQLite database
conn = sqlite3.connect(':memory:')
# Create the results table with the required columns
conn.execute('''
CREATE TABLE results (
model TEXT,
method TEXT,
domain TEXT,
problem_id TEXT,
turn_number INTEGER,
answer_correct INTEGER,
extraction_null INTEGER,
ledger_json TEXT,
constraint_extraction_success INTEGER,
z3_sat INTEGER,
mus_size INTEGER,
mus_json TEXT,
repair_attempted INTEGER,
repair_trigger TEXT,
repair_trigger_count INTEGER,
repair_reason_json TEXT
)
''')
# Insert some sample rows
conn.execute('''
INSERT INTO results VALUES
('model1', 'method1', 'domain1', 'prob1', 1, 1, 0, '{}', 1, 1, 5, '[]', 1, 'trigger1', 2, '{}'),
('model1', 'method1', 'domain1', 'prob2', 1, 0, 1, '{}', 0, 0, 0, '[]', 0, NULL, NULL, NULL),
('model2', 'method2', 'domain2', 'prob3', 2, 1, 0, '{}', 1, 1, 3, '[]', 1, 'trigger2', 1, '{}')
''')
conn.commit()
# Create a problems_split_map
problems_split_map = {
'prob1': 'train',
'prob2': 'val',
'prob3': 'train'
}
# Define the ARGS dictionary with all required parameters
ARGS = {
'conn': conn,
'split': 'all',
'problems_split_map': problems_split_map
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| ordered_unique |
Ran ran; our contract draft was wrong, not the code |
kaons-research/drift-bench/src/repair_policy.py code served (permissive licence) · get_code("8bead903b69ffed0") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from typing import List
# The function takes a list of strings and returns a list of strings
# It filters out empty strings and duplicates, preserving order
ARGS = {
"values": ["apple", "banana", "apple", "", "cherry", "banana", "date"]
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| pick_distinct_pair |
Ran ran; our contract draft was wrong, not the code |
kaons-research/drift-bench/src/generate_problems.py code served (permissive licence) · get_code("2added5f4145a1d2") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import random
ARGS = {
"rng": random.Random(42),
"items": ["apple", "banana", "cherry", "date", "elderberry"]
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| policy_uses_sat_triggers |
Ran ran, violated the contract we drafted |
kaons-research/drift-bench/src/repair_policy.py code served (permissive licence) · get_code("1a41a1bb65acebd1") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
ARGS = {
"policy": "expanded"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| sample_seating_constraint |
Ran ran; our contract draft was wrong, not the code |
kaons-research/drift-bench/src/generate_problems.py code served (permissive licence) · get_code("d33b30864c53fb64") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import random
def pick_distinct_pair(rng: random.Random, items: list[str]) -> tuple[str, str]:
a, b = rng.sample(items, 2)
return (a, b)
def sample_seating_constraint(rng: random.Random, entities: list[str], num_entities: int, table_shape: str) -> dict:
kinds = ['adjacent', 'not_adjacent', 'at_position', 'left_of', 'separated_by']
if table_shape == 'rectangular':
kinds.extend(['same_side', 'opposite_side'])
ctype = rng.choice(kinds)
if ctype == 'adjacent':
a, b = pick_distinct_pair(rng, entities)
return {'type': ctype, 'args': [a, b], 'nl': f'{a} must sit next to {b}'}
if ctype == 'not_adjacent':
a, b = pick_distinct_pair(rng, entities)
return {'type': ctype, 'args': [a, b], 'nl': f'{a} cannot sit next to {b}'}
if ctype == 'at_position':
a = rng.choice(entities)
pos = rng.randint(1, num_entities)
return {'type': ctype, 'args': [a, pos], 'nl': f'{a} must sit at position {pos}'}
if ctype == 'left_of':
a, b = pick_distinct_pair(rng, entities)
return {'type': ctype, 'args': [a, b], 'nl': f'{a} sits immediately left of {b}'}
if ctype == 'separated_by':
a, b = pick_distinct_pair(rng, entities)
n_sep = rng.randint(1, max(1, num_entities // 2 - 1))
return {'type': ctype, 'args': [a, b, n_sep], 'nl': f'{a} and {b} must have at least {n_sep} seats between them'}
if ctype == 'same_side':
a, b = pick_distinct_pair(rng, entities)
return {'type': ctype, 'args': [a, b], 'nl': f'{a} and {b} must sit on the same side of the table'}
if ctype == 'opposite_side':
a, b = pick_distinct_pair(rng, entities)
return {'type': ctype, 'args': [a, b], 'nl': f'{a} and {b} must sit on opposite sides of the table'}
raise ValueError(f'Unsupported seating constraint type: {ctype}')
ARGS = {
'rng': random.Random(42),
'entities': ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'],
'num_entities': 5,
'table_shape': 'rectangular'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| table_columns |
Ran ran; our contract draft was wrong, not the code |
kaons-research/drift-bench/src/analyze.py code served (permissive licence) · get_code("82a316cc22bc1015") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned set. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import sqlite3
import tempfile
import os
# Create a temporary database file
temp_db = tempfile.NamedTemporaryFile(delete=False, suffix='.db')
temp_db_path = temp_db.name
temp_db.close()
try:
# Create a connection and a test table
conn = sqlite3.connect(temp_db_path)
conn.execute('''
CREATE TABLE test_table (
id INTEGER PRIMARY KEY,
name TEXT,
age INTEGER,
email TEXT
)
''')
conn.commit()
# Define the arguments
ARGS = {
'conn': conn,
'table': 'test_table'
}
finally:
# Clean up will happen after function execution
pass
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type set — a set output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a set output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| unique_trigger_codes |
Ran ran; our contract draft was wrong, not the code |
kaons-research/drift-bench/src/repair_policy.py code served (permissive licence) · get_code("26f1b08bdc9578f4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from typing import Any
# Construct arguments for unique_trigger_codes function
# The function expects: reasons: list[dict[str, Any]]
ARGS = {
"reasons": [
{"code": "ERROR_001"},
{"code": " WARNING_002 "},
{"code": ""},
{"code": "ERROR_001"}, # duplicate, should be filtered
{"other_field": "value"}, # missing 'code' key
{"code": None},
{"code": "INFO_003"},
]
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_answer |
Not yet run unverified (harvested, no recorded run) |
kaons-research/drift-bench/src/extraction.py code served (permissive licence) · get_code("2859b0d4274869b4") How it ran |
| extract_answer_with_retry |
Not yet run unverified (harvested, no recorded run) |
kaons-research/drift-bench/src/extraction.py code served (permissive licence) · get_code("94d9078851c10b86") How it ran |