| format_evaluation_requirements |
Ran ran on a synthesized input |
zhenyuli801/XIFBench/D2_validate_translations.py code served (permissive licence) · get_code("3a43b74a35f656eb") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
ARGS = {
"evaluation_requirements": [
{
"requirement_category": "Accuracy",
"requirement_direction": "maximize",
"requirement_specification": "Translation should preserve the original meaning"
},
{
"requirement_category": "Fluency",
"requirement_direction": "maximize",
"requirement_specification": "Output should read naturally in the target language"
},
{
"requirement_category": "Latency",
"requirement_direction": "minimize",
"requirement_specification": "Translation should complete within 100ms"
}
]
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| format_sampled_constraints |
Ran ran on a synthesized input |
zhenyuli801/XIFBench/B2_combine_constraints.py code served (permissive licence) · get_code("1cfb172de71361fd") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Create sample constraint dictionaries
sampled_constraints = [
{
"constraint_category": "Safety",
"constraint_direction": "maximize",
"constraint_specification": "The response should not contain harmful content"
},
{
"constraint_category": "Relevance",
"constraint_direction": "minimize",
"constraint_specification": "Avoid off-topic information in the answer"
},
{
"constraint_category": "Clarity",
"constraint_direction": "maximize",
"constraint_specification": "The explanation should be easy to understand"
}
]
ARGS = {
"sampled_constraints": sampled_constraints
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| generate_reasoning_and_responses |
Ran ran on a synthesized input |
zhenyuli801/XIFBench/G1_api_inference.py code served (permissive licence) · get_code("635301a89612a3d4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 2, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import time
from tqdm import tqdm
from loguru import logger
from openai import OpenAI
from unittest.mock import Mock, patch, MagicMock
# Create mock response objects that mimic OpenAI API responses
mock_message = Mock()
mock_message.reasoning_content = "This is the reasoning for the response."
mock_message.content = "This is the generated response."
mock_choice = Mock()
mock_choice.message = mock_message
mock_response = Mock()
mock_response.choices = [mock_choice]
# Patch the OpenAI client to return our mock response
def mock_api_call(*args, **kwargs):
return mock_response
# Create sample prompts
sample_prompts = [
"What is the capital of France?",
"Explain quantum computing in simple terms.",
]
# Define the arguments dictionary
ARGS = {
'prompts': sample_prompts,
'api_key': 'sk-test-key-12345',
'base_url': 'https://api.openai.com/v1',
'model': 'gpt-4-turbo',
'max_tokens': 1024,
'temperature': 0.0,
'max_api_retry': 5,
'base_wait_time': 1,
'debug_mode': False,
}
# Apply the mock to make the function executable without actual API calls
with patch('openai.OpenAI') as mock_openai_class:
mock_client = Mock()
mock_client.chat.completions.create = mock_api_call
mock_openai_class.return_value = mock_client
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- loguru, openai, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 2, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_alpacaeval |
Ran ran on a synthesized input |
zhenyuli801/XIFBench/A1_sample_instructions.py code served (permissive licence) · get_code("1aac49f7aa943d77") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
import os
# Create a temporary JSON file with AlpacaEval-like data
temp_dir = tempfile.mkdtemp()
test_file = os.path.join(temp_dir, "alpacaeval_test.json")
# Create sample AlpacaEval data
sample_data = [
{
"instruction": "What is the capital of France?",
"input": "",
"output": "The capital of France is Paris."
},
{
"instruction": "Write a short poem about nature.",
"input": "",
"output": "Leaves dance in the wind,\nBirds sing their sweet song,\nNature's beauty endless."
}
]
# Write to temporary file
with open(test_file, 'w', encoding='utf-8') as f:
json.dump(sample_data, f)
ARGS = {
"file_path": test_file
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_json |
Ran ran; our contract draft was wrong, not the code |
zhenyuli801/XIFBench/A1_sample_instructions.py code served (permissive licence) · get_code("cf254803acaaf54c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
import os
# Create a temporary JSON file with sample data
temp_dir = tempfile.mkdtemp()
temp_file = os.path.join(temp_dir, "test.json")
sample_data = {
"key1": "value1",
"key2": [1, 2, 3],
"key3": {"nested": "data"}
}
with open(temp_file, 'w', encoding='utf-8') as f:
json.dump(sample_data, f)
ARGS = {
"file_path": temp_file
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_jsonl |
Ran ran on a synthesized input |
zhenyuli801/XIFBench/A1_sample_instructions.py code served (permissive licence) · get_code("d4c04971c3f6dc69") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
import os
# Create a temporary JSONL file for testing
temp_dir = tempfile.mkdtemp()
temp_file = os.path.join(temp_dir, "test.jsonl")
# Write sample JSONL data
sample_data = [
{"id": 1, "text": "hello world", "value": 42},
{"id": 2, "text": "foo bar", "value": 3.14},
{"id": 3, "text": "test data", "value": True}
]
with open(temp_file, 'w', encoding='utf-8') as f:
for item in sample_data:
f.write(json.dumps(item) + '\n')
ARGS = {
"file_path": temp_file
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| parse_constraint_brainstorming_response |
Ran ran on a synthesized input |
zhenyuli801/XIFBench/B1_brainstorm_constraints.py code served (permissive licence) · get_code("c104e11d10734d8e") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
from loguru import logger
ARGS = {
"response": """# Content constraints
- **clarity**: The constraint should be easy to understand and interpret
- **specificity**: The constraint must be precise and well-defined
# Format constraints
- **conciseness**: Keep constraint descriptions brief and to the point
- **structure**: Use consistent formatting across all constraints
# Relevance constraints
- **applicability**: Ensure constraints are relevant to the problem domain"""
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- loguru — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| parse_constraint_combination_response |
Ran ran on a synthesized input |
zhenyuli801/XIFBench/B2_combine_constraints.py code served (permissive licence) · get_code("e02d2d2f5e843460") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
ARGS = {
"response": " This is a combined instruction from constraint combination. "
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| parse_requirement_categorization_response |
Ran ran on a synthesized input |
zhenyuli801/XIFBench/C2_categorize_requirements.py code served (permissive licence) · get_code("505083d5b2dec3e6") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
from loguru import logger
VALID_REQUIREMENT_CATEGORIES = {'content', 'style', 'situation', 'format', 'numerical'}
INVALID_CATEGORY_TAG = '[INVALID_CATEGORY]'
def parse_requirement_categorization_response(response: str) -> list[dict[str, str]]:
""" Parse the response from Requirement Categorization and extract the requirement categories. """
parsed_data = []
CATEGORY_DIRECTION_PATTERN = '^\\d+\\.\\s*(.+?)\\s*-\\s*(.+)'
try:
lines = response.strip().splitlines()
for line in lines:
line = line.strip()
if not line:
continue
category_match = re.match(CATEGORY_DIRECTION_PATTERN, line)
if category_match:
category = category_match.group(1).lower()
direction = category_match.group(2).lower()
category = category.strip('*')
direction = direction.strip('*')
category = category.split(' ')[0]
if category in VALID_REQUIREMENT_CATEGORIES:
parsed_data.append({'requirement_category': category, 'requirement_direction': direction})
else:
logger.warning(f'Unexpected category encountered: {category}')
parsed_data.append({'requirement_category': f'{INVALID_CATEGORY_TAG} - {category}', 'requirement_direction': f'{INVALID_CATEGORY_TAG} - {direction}'})
return parsed_data
except Exception as e:
logger.error(f'Failed to extract requirement categories from response. Error: {e}')
return None
# Setup code
response_valid = """1. content - increase clarity and precision
2. style - use active voice
3. format - organize in bullet points
4. numerical - provide exact metrics"""
response_with_invalid = """1. content - increase clarity
2. invalid_cat - some direction
3. style - use passive voice"""
response_empty = ""
response_malformed = """This is not formatted correctly
No numbered items here"""
ARGS = {
'response': response_valid
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- loguru — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| parse_requirement_decomposition_response |
Ran ran on a synthesized input |
zhenyuli801/XIFBench/C1_decompose_requirements.py code served (permissive licence) · get_code("d8bad0118c48538d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
from loguru import logger
def parse_requirement_decomposition_response(response: str) -> list[str]:
""" Parse the response from Requirement Decomposition and extract the evaluation requirements. """
requirements = []
REQUIREMENT_PATTERN = '^\\d+\\.\\s*(.+)'
try:
lines = response.strip().splitlines()
for line in lines:
line = line.strip()
if not line:
continue
requirement_match = re.match(REQUIREMENT_PATTERN, line)
if requirement_match:
requirement = requirement_match.group(1).strip()
requirements.append(requirement)
return requirements
except Exception as e:
logger.error(f'Failed to extract requirements from response. Error: {e}')
return None
# Setup code to construct valid arguments
response_text = """1. The system shall validate user input before processing
2. The system shall return results within 2 seconds
3. The system shall handle edge cases gracefully
4. Additional requirement with proper formatting"""
ARGS = {
"response": response_text
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- loguru — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| sample_constraints |
Ran ran on a synthesized input |
zhenyuli801/XIFBench/B2_combine_constraints.py code served (permissive licence) · get_code("33730eed64c4e455") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Create brainstormed constraints with the expected structure
brainstormed_constraints = [
{
'constraints_category': 'grammar',
'constraints_content': [
{'rule': 'subject-verb agreement', 'description': 'Ensure subject and verb agree in number'},
{'rule': 'tense consistency', 'description': 'Maintain consistent verb tense throughout'},
]
},
{
'constraints_category': 'style',
'constraints_content': [
{'rule': 'formal tone', 'description': 'Use formal language'},
{'rule': 'conciseness', 'description': 'Keep sentences short and clear'},
]
},
{
'constraints_category': 'semantic',
'constraints_content': [
{'rule': 'factual accuracy', 'description': 'Ensure all statements are factually correct'},
{'rule': 'logical flow', 'description': 'Maintain logical progression of ideas'},
]
},
]
max_constraint_count = 6
constraint_sample_times = 3
random_seed = 42
ARGS = {
'brainstormed_constraints': brainstormed_constraints,
'max_constraint_count': max_constraint_count,
'constraint_sample_times': constraint_sample_times,
'random_seed': random_seed,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list — a list output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_openai_response |
Not yet run unverified (harvested, no recorded run) |
zhenyuli801/XIFBench/B1_brainstorm_constraints.py code served (permissive licence) · get_code("d52e4d9cbd07b9c1") How it ran |