| extract_lean_code |
Ran ran; our contract draft was wrong, not the code |
koreankiwi99/formalization-gaming/src/utils/lean_utils.py code served (permissive licence) · get_code("ea7eba31cb0ee7bf") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def extract_lean_code(llm_response):
"""Extract Lean code from LLM response.
Looks for code wrapped in <lean></lean> tags or ```lean``` markdown blocks.
Args:
llm_response: The LLM's text response
Returns:
str or None: Extracted Lean code, or None if not found
"""
match = re.search('<lean>(.*?)</lean>', llm_response, re.DOTALL | re.IGNORECASE)
if match:
return match.group(1).strip()
code_blocks = re.findall('```lean\\s*(.*?)```', llm_response, re.DOTALL | re.IGNORECASE)
if code_blocks:
return '\n\n'.join((block.strip() for block in code_blocks))
return None
# Test case 1: LLM response with <lean></lean> tags
llm_response_with_tags = """
The solution is:
<lean>
def hello : String := "world"
</lean>
"""
# Test case 2: LLM response with markdown code blocks
llm_response_with_markdown = """
Here's the Lean code:
lean
theorem example : 1 + 1 = 2 := by
rfl
"""
# Test case 3: LLM response with no Lean code
llm_response_no_code = "This is just plain text without any code."
# Test case 4: LLM response with multiple markdown blocks
llm_response_multiple = """
First block:
lean
def add (a b : Nat) : Nat := a + b
Second block:
lean
theorem add_comm : ∀ a b, add a b = add b a := by
sorry
"""
ARGS = {
"llm_response": llm_response_with_tags
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| format_premises |
Ran ran; our contract draft was wrong, not the code |
koreankiwi99/formalization-gaming/src/analysis/evaluate_faithfulness.py code served (permissive licence) · get_code("a6aac36936d3c4f2") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Test case 1: premises as a list
premises_list = [
"All humans are mortal",
"Socrates is human",
"Therefore, Socrates is mortal"
]
# Test case 2: premises as a string
premises_string = "All dogs are animals"
# Test case 3: premises as None (falsy value)
premises_none = None
# We'll use the list version as the primary test case
ARGS = {
"premises": premises_list
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| format_system_prompt |
Ran ran; our contract draft was wrong, not the code |
koreankiwi99/formalization-gaming/src/utils/prompts.py code served (permissive licence) · get_code("46c489c0d1c6a73c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
# Construct arguments for format_system_prompt
template = "Please answer the question with one of: {answer_format}. Answer true as '{answer_true}' and false as '{answer_false}'."
dataset = "folio"
ARGS = {
"template": template,
"dataset": dataset,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_false_negatives |
Ran ran; our contract draft was wrong, not the code |
koreankiwi99/formalization-gaming/src/analysis/analyze_errors.py code served (permissive licence) · get_code("39e31a2e1463691a") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# Construct valid arguments for get_false_negatives
# The function expects a list of results
# Each result is either None or a dict with:
# - 'lean_verification': dict with 'success' key
# - 'correct': boolean key
results = [
# False negative case: lean_verification.success=True AND correct=False
{
'lean_verification': {'success': True},
'correct': False,
},
# False negative case: lean_verification.success=True AND correct=False
{
'lean_verification': {'success': True, 'other_field': 'data'},
'correct': False,
},
# Not a false negative: lean_verification.success=False
{
'lean_verification': {'success': False},
'correct': False,
},
# Not a false negative: correct=True (default)
{
'lean_verification': {'success': True},
'correct': True,
},
# Not a false negative: lean_verification is None/falsy
{
'lean_verification': None,
'correct': False,
},
# None entry (skipped)
None,
# Not a false negative: correct defaults to True when missing
{
'lean_verification': {'success': True},
},
]
ARGS = {
'results': results,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_lean_code |
Ran ran; our contract draft was wrong, not the code |
koreankiwi99/formalization-gaming/src/analysis/evaluate_faithfulness.py code served (permissive licence) · get_code("044e45410848bef5") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Case 1: Baseline case with lean_code
case_baseline = {
'lean_code': 'theorem example : 1 + 1 = 2 := by rfl'
}
# Case 2: Two-stage case with stage2_code
case_two_stage = {
'stage2_code': 'theorem two_stage : ∀ n, n + 0 = n := by intro n; simp'
}
# Case 3: Divergent case with true_lean_code and false_lean_code
case_divergent = {
'true_lean_code': 'theorem true_branch : True := trivial',
'false_lean_code': 'theorem false_branch : False := by contradiction'
}
# Use the divergent case as the primary test case
ARGS = {
'case': case_divergent
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_provider |
Ran ran; our contract draft was wrong, not the code |
koreankiwi99/formalization-gaming/src/utils/api_client.py code served (permissive licence) · get_code("64e253007e4a5ed0") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
# The function get_provider takes a single parameter: model (a string)
# It checks if the model is in MODEL_PROVIDERS dict and returns the provider,
# otherwise returns 'openai'
# Test with a known model
model_known = 'gpt-4o'
# Test with an unknown model
model_unknown = 'some-unknown-model'
ARGS = {
'model': model_known
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_folio |
Ran ran; our contract draft was wrong, not the code |
koreankiwi99/formalization-gaming/src/utils/datasets.py code served (permissive licence) · get_code("60cbd940377d6d13") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
import os
# Create a temporary directory and file with sample FOLIO data
temp_dir = tempfile.mkdtemp()
folio_data = [
{
'story_id': 'story_1',
'example_id': 'ex_1',
'premises': ['Premise 1', 'Premise 2'],
'conclusion': 'Conclusion 1',
'label': 'Entailment'
},
{
'story_id': 'story_2',
'example_id': 'ex_2',
'premises': ['Premise 3', 'Premise 4'],
'conclusion': 'Conclusion 2',
'label': 'Neutral'
}
]
folio_path = os.path.join(temp_dir, 'folio-validation.json')
with open(folio_path, 'w') as f:
json.dump(folio_data, f)
ARGS = {
'folio_path': folio_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_multilogieval |
Ran ran, honoured the contract we drafted |
koreankiwi99/formalization-gaming/src/utils/datasets.py code served (permissive licence) · get_code("150640dc130c1d59") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
from pathlib import Path
# Create temporary directory structure that mimics the expected dataset layout
temp_dir = tempfile.mkdtemp()
data_dir = Path(temp_dir) / 'data' / 'multilogieval' / 'original' / 'data'
data_dir.mkdir(parents=True, exist_ok=True)
# Create sample dataset structure
depths_to_create = [2, 3]
logic_types_to_create = ['propositional', 'first_order']
for depth in depths_to_create:
depth_dir = data_dir / f'{depth}_Data'
depth_dir.mkdir(exist_ok=True)
for logic_type in logic_types_to_create:
logic_dir = depth_dir / logic_type
logic_dir.mkdir(exist_ok=True)
# Create sample JSON files
sample_data = {
'rule': f'rule_{logic_type}_{depth}',
'samples': [
{
'id': f'sample_1_{logic_type}_{depth}',
'context': f'Context for {logic_type} at depth {depth}',
'question': f'Question for {logic_type}?',
'answer': 'TRUE'
},
{
'id': f'sample_2_{logic_type}_{depth}',
'context': f'Another context for {logic_type}',
'question': f'Another question?',
'answer': 'FALSE'
}
]
}
json_file = logic_dir / f'test_{logic_type}.json'
with open(json_file, 'w', encoding='utf-8') as f:
json.dump(sample_data, f)
# Patch the function to use our temporary directory
import sys
from unittest.mock import patch
original_load = None
def load_multilogieval_patched(depths, logic_types):
cases = []
for depth in depths:
depth_dir = data_dir / f'{depth}_Data'
if not depth_dir.exists():
print(f'Warning: {depth_dir} does not exist')
continue
for logic_type in logic_types:
logic_dir = depth_dir / logic_type
if not logic_dir.exists():
continue
for json_file in logic_dir.glob('*.json'):
try:
with open(json_file, encoding='utf-8') as f:
data = json.load(f)
except UnicodeDecodeError:
with open(json_file, encoding='latin-1') as f:
data = json.load(f)
rule = data.get('rule', json_file.stem)
samples = data.get('samples', [])
for sample in samples:
cases.append({'id': sample.get('id'), 'context': sample.get('context', ''), 'question': sample.get('question', ''), 'ground_truth': sample.get('answer', '').lower(), 'logic_type': logic_type, 'depth': depth, 'rule': rule})
return cases
ARGS = {
'depths': [2, 3],
'logic_types': ['propositional', 'first_order']
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_multilogieval_sampled |
Ran ran; our contract draft was wrong, not the code |
koreankiwi99/formalization-gaming/src/utils/datasets.py code served (permissive licence) · get_code("77490f433de992ed") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
import os
# Create a temporary JSON file with sample MultiLogiEval data
sample_data = [
{
'idx': 0,
'id': 'sample_0',
'context': 'All birds can fly. Tweety is a bird.',
'question': 'Can Tweety fly?',
'answer': 'Yes',
'logic': 'fol',
'depth': '2',
'rule': 'modus_ponens'
},
{
'idx': 1,
'id': 'sample_1',
'context': 'Some mammals are carnivores. Lions are mammals.',
'question': 'Are lions carnivores?',
'answer': 'Possibly',
'logic': 'fol',
'depth': '1',
'rule': 'universal_instantiation'
},
{
'idx': 2,
'id': 'sample_2',
'context': 'If it rains, the ground is wet.',
'question': 'Is the ground wet?',
'answer': 'Unknown',
'logic': 'pl',
'depth': '1',
'rule': 'conditional'
}
]
# Create temporary file
temp_file = tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False)
json.dump(sample_data, temp_file)
temp_file.close()
ARGS = {
'data_file': temp_file.name
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_prompt |
Ran ran; our contract draft was wrong, not the code |
koreankiwi99/formalization-gaming/src/utils/prompts.py code served (permissive licence) · get_code("872d7c12ef1aca3f") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import os
# Create a temporary file with some prompt content
temp_dir = tempfile.mkdtemp()
prompt_file = os.path.join(temp_dir, "test_prompt.txt")
with open(prompt_file, 'w') as f:
f.write("This is a test prompt with some content.\n")
ARGS = {
"path": prompt_file
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_prompt_template |
Ran ran; our contract draft was wrong, not the code |
koreankiwi99/formalization-gaming/src/analysis/analyze_errors.py code served (permissive licence) · get_code("0f1c89036844e212") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import os
# Create a temporary prompt template file
temp_dir = tempfile.mkdtemp()
prompt_file = os.path.join(temp_dir, "prompt_template.txt")
with open(prompt_file, 'w') as f:
f.write("You are a helpful assistant.\nRespond to the following query: {query}\n")
ARGS = {
"prompt_path": prompt_file
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| normalize_answer |
Ran ran; our contract draft was wrong, not the code |
koreankiwi99/formalization-gaming/src/utils/answer_parsing.py code served (permissive licence) · get_code("651b0f7470e1a3e9") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
# Setup for normalize_answer function
ARGS = {
'answer': 'yes',
'answer_format': 'yes_no'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| parse_answer |
Ran ran; our contract draft was wrong, not the code |
koreankiwi99/formalization-gaming/src/utils/answer_parsing.py code served (permissive licence) · get_code("5200c424eaeecdfc") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def normalize_answer(answer, answer_format='yes_no'):
"""Normalize answer to standard format.
Args:
answer: The answer to normalize
answer_format: "yes_no", "true_false", "bidir_true", or "bidir_false"
Returns:
str: Normalized answer
"""
if not answer:
return 'Failure' if answer_format.startswith('bidir') else 'Uncertain'
low = answer.lower().strip()
if answer_format == 'yes_no':
if low in ['yes', 'y', 'true', 't', '1']:
return 'Yes'
elif low in ['no', 'n', 'false', 'f', '0']:
return 'No'
elif low in ['unknown', 'uncertain', 'u']:
return 'Uncertain'
elif answer_format == 'true_false':
if low in ['true', 't', 'yes', 'y']:
return 'True'
elif low in ['false', 'f', 'no', 'n']:
return 'False'
elif low in ['unknown', 'uncertain', 'u']:
return 'Uncertain'
elif answer_format == 'bidir_true':
if low in ['yes', 'y']:
return 'Yes'
elif low in ['true', 't', 'success']:
return 'True'
elif low in ['failure', 'failed', 'fail', 'f']:
return 'Failure'
elif answer_format == 'bidir_false':
if low in ['no', 'n']:
return 'No'
elif low in ['false', 'f', 'success']:
return 'False'
elif low in ['failure', 'failed', 'fail']:
return 'Failure'
return 'Failure' if answer_format.startswith('bidir') else 'Uncertain'
def parse_answer(response, answer_format='true_false'):
"""Extract answer from model response.
Args:
response: The model's text response
answer_format: "true_false", "yes_no", "bidir_true", or "bidir_false"
Returns:
tuple: (answer, parse_status)
Status: SUCCESS, FALLBACK, EMPTY, PARSE_FAILED
"""
if not response or not response.strip():
return (None, 'EMPTY')
if answer_format == 'bidir_true':
pattern = 'ANSWER:\\s*(True|Yes|Failure|Failed)'
fallback_pattern = '\\b(True|Yes|Failure|Failed)\\b'
elif answer_format == 'bidir_false':
pattern = 'ANSWER:\\s*(False|No|Failure|Failed)'
fallback_pattern = '\\b(False|No|Failure|Failed)\\b'
elif answer_format == 'true_false':
pattern = 'ANSWER:\\s*(True|False|Unknown|Uncertain)'
fallback_pattern = '\\b(True|False|Unknown|Uncertain)\\b'
else:
pattern = 'ANSWER:\\s*(Yes|No|True|False|Unknown|Uncertain)'
fallback_pattern = '\\b(Yes|No|True|False|Unknown|Uncertain)\\b'
answer_match = re.search(pattern, response, re.IGNORECASE)
if answer_match:
answer = normalize_answer(answer_match.group(1), answer_format)
return (answer, 'SUCCESS')
all_answers = re.findall(fallback_pattern, response, re.IGNORECASE)
if all_answers:
answer = normalize_answer(all_answers[-1], answer_format)
return (answer, 'FALLBACK')
return (None, 'PARSE_FAILED')
ARGS = {
'response': 'The model says: ANSWER: True with confidence.',
'answer_format': 'true_false',
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| parse_folio_answer |
Ran ran; our contract draft was wrong, not the code |
koreankiwi99/formalization-gaming/src/utils/answer_parsing.py code served (permissive licence) · get_code("75f5e519aa2c36de") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def normalize_answer(answer, answer_format='yes_no'):
"""Normalize answer to standard format.
Args:
answer: The answer to normalize
answer_format: "yes_no", "true_false", "bidir_true", or "bidir_false"
Returns:
str: Normalized answer
"""
if not answer:
return 'Failure' if answer_format.startswith('bidir') else 'Uncertain'
low = answer.lower().strip()
if answer_format == 'yes_no':
if low in ['yes', 'y', 'true', 't', '1']:
return 'Yes'
elif low in ['no', 'n', 'false', 'f', '0']:
return 'No'
elif low in ['unknown', 'uncertain', 'u']:
return 'Uncertain'
elif answer_format == 'true_false':
if low in ['true', 't', 'yes', 'y']:
return 'True'
elif low in ['false', 'f', 'no', 'n']:
return 'False'
elif low in ['unknown', 'uncertain', 'u']:
return 'Uncertain'
elif answer_format == 'bidir_true':
if low in ['yes', 'y']:
return 'Yes'
elif low in ['true', 't', 'success']:
return 'True'
elif low in ['failure', 'failed', 'fail', 'f']:
return 'Failure'
elif answer_format == 'bidir_false':
if low in ['no', 'n']:
return 'No'
elif low in ['false', 'f', 'success']:
return 'False'
elif low in ['failure', 'failed', 'fail']:
return 'Failure'
return 'Failure' if answer_format.startswith('bidir') else 'Uncertain'
def parse_answer(response, answer_format='true_false'):
"""Extract answer from model response.
Args:
response: The model's text response
answer_format: "true_false", "yes_no", "bidir_true", or "bidir_false"
Returns:
tuple: (answer, parse_status)
Status: SUCCESS, FALLBACK, EMPTY, PARSE_FAILED
"""
if not response or not response.strip():
return (None, 'EMPTY')
if answer_format == 'bidir_true':
pattern = 'ANSWER:\\s*(True|Yes|Failure|Failed)'
fallback_pattern = '\\b(True|Yes|Failure|Failed)\\b'
elif answer_format == 'bidir_false':
pattern = 'ANSWER:\\s*(False|No|Failure|Failed)'
fallback_pattern = '\\b(False|No|Failure|Failed)\\b'
elif answer_format == 'true_false':
pattern = 'ANSWER:\\s*(True|False|Unknown|Uncertain)'
fallback_pattern = '\\b(True|False|Unknown|Uncertain)\\b'
else:
pattern = 'ANSWER:\\s*(Yes|No|True|False|Unknown|Uncertain)'
fallback_pattern = '\\b(Yes|No|True|False|Unknown|Uncertain)\\b'
answer_match = re.search(pattern, response, re.IGNORECASE)
if answer_match:
answer = normalize_answer(answer_match.group(1), answer_format)
return (answer, 'SUCCESS')
all_answers = re.findall(fallback_pattern, response, re.IGNORECASE)
if all_answers:
answer = normalize_answer(all_answers[-1], answer_format)
return (answer, 'FALLBACK')
return (None, 'PARSE_FAILED')
def parse_folio_answer(response, return_status=False):
"""Extract True/False/Unknown answer from FOLIO response."""
answer, status = parse_answer(response, answer_format='true_false')
if return_status:
return (answer, status)
return answer
ARGS = {
'response': 'The statement is true. ANSWER: True',
'return_status': False
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| resolve_model_name |
Ran ran; our contract draft was wrong, not the code |
koreankiwi99/formalization-gaming/src/utils/api_client.py code served (permissive licence) · get_code("a8c0ee30b68f960e") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
# Setup code for resolve_model_name function
ARGS = {
'model': 'llama-70b'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| create_client |
Not yet run unverified (harvested, no recorded run) |
koreankiwi99/formalization-gaming/src/utils/api_client.py code served (permissive licence) · get_code("88e267ab5c120ee7") How it ran |
| format_user_prompt |
Not yet run unverified (harvested, no recorded run) |
koreankiwi99/formalization-gaming/src/utils/prompts.py code served (permissive licence) · get_code("6dbbca04c916feb9") How it ran |