SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2604.19459 · 2026

Do LLMs Game Formalization? Evaluating Faithfulness in Logical Reasoning

Antoine Bosselut, Kyuhee Kim, Auguste Poiroux

arXiv · PDF · Open in the Atlas

Code that ran

15 of 17 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 1 honoured the contract we drafted, 0 violated it and 0 have no contract verdict; 14 failures of our instrument; 2 no recorded run), measured 2026-11-22; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2604.19459.json · badge: /api/ran/2604.19459.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
koreankiwi99/formalization-gaming canonical (classifier label)
classifier basis: census
15 ran · 2 no recorded run
FunctionStatusWhere it lives
extract_lean_code Ran
ran; our contract draft was wrong, not the code
koreankiwi99/formalization-gaming/src/utils/lean_utils.py
code served (permissive licence) · get_code("ea7eba31cb0ee7bf")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def extract_lean_code(llm_response):
    """Extract Lean code from LLM response.

    Looks for code wrapped in <lean></lean> tags or ```lean``` markdown blocks.

    Args:
        llm_response: The LLM's text response

    Returns:
        str or None: Extracted Lean code, or None if not found
    """
    match = re.search('<lean>(.*?)</lean>', llm_response, re.DOTALL | re.IGNORECASE)
    if match:
        return match.group(1).strip()
    code_blocks = re.findall('```lean\\s*(.*?)```', llm_response, re.DOTALL | re.IGNORECASE)
    if code_blocks:
        return '\n\n'.join((block.strip() for block in code_blocks))
    return None


# Test case 1: LLM response with <lean></lean> tags
llm_response_with_tags = """
The solution is:
<lean>
def hello : String := "world"
</lean>
"""

# Test case 2: LLM response with markdown code blocks
llm_response_with_markdown = """
Here's the Lean code:
lean
theorem example : 1 + 1 = 2 := by
  rfl

"""

# Test case 3: LLM response with no Lean code
llm_response_no_code = "This is just plain text without any code."

# Test case 4: LLM response with multiple markdown blocks
llm_response_multiple = """
First block:
lean
def add (a b : Nat) : Nat := a + b


Second block:
lean
theorem add_comm : ∀ a b, add a b = add b a := by
  sorry

"""

ARGS = {
    "llm_response": llm_response_with_tags
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
format_premises Ran
ran; our contract draft was wrong, not the code
koreankiwi99/formalization-gaming/src/analysis/evaluate_faithfulness.py
code served (permissive licence) · get_code("a6aac36936d3c4f2")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Test case 1: premises as a list
premises_list = [
    "All humans are mortal",
    "Socrates is human",
    "Therefore, Socrates is mortal"
]

# Test case 2: premises as a string
premises_string = "All dogs are animals"

# Test case 3: premises as None (falsy value)
premises_none = None

# We'll use the list version as the primary test case
ARGS = {
    "premises": premises_list
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
format_system_prompt Ran
ran; our contract draft was wrong, not the code
koreankiwi99/formalization-gaming/src/utils/prompts.py
code served (permissive licence) · get_code("46c489c0d1c6a73c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

# Construct arguments for format_system_prompt
template = "Please answer the question with one of: {answer_format}. Answer true as '{answer_true}' and false as '{answer_false}'."
dataset = "folio"

ARGS = {
    "template": template,
    "dataset": dataset,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_false_negatives Ran
ran; our contract draft was wrong, not the code
koreankiwi99/formalization-gaming/src/analysis/analyze_errors.py
code served (permissive licence) · get_code("39e31a2e1463691a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Construct valid arguments for get_false_negatives
# The function expects a list of results
# Each result is either None or a dict with:
# - 'lean_verification': dict with 'success' key
# - 'correct': boolean key

results = [
    # False negative case: lean_verification.success=True AND correct=False
    {
        'lean_verification': {'success': True},
        'correct': False,
    },
    # False negative case: lean_verification.success=True AND correct=False
    {
        'lean_verification': {'success': True, 'other_field': 'data'},
        'correct': False,
    },
    # Not a false negative: lean_verification.success=False
    {
        'lean_verification': {'success': False},
        'correct': False,
    },
    # Not a false negative: correct=True (default)
    {
        'lean_verification': {'success': True},
        'correct': True,
    },
    # Not a false negative: lean_verification is None/falsy
    {
        'lean_verification': None,
        'correct': False,
    },
    # None entry (skipped)
    None,
    # Not a false negative: correct defaults to True when missing
    {
        'lean_verification': {'success': True},
    },
]

ARGS = {
    'results': results,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_lean_code Ran
ran; our contract draft was wrong, not the code
koreankiwi99/formalization-gaming/src/analysis/evaluate_faithfulness.py
code served (permissive licence) · get_code("044e45410848bef5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Case 1: Baseline case with lean_code
case_baseline = {
    'lean_code': 'theorem example : 1 + 1 = 2 := by rfl'
}

# Case 2: Two-stage case with stage2_code
case_two_stage = {
    'stage2_code': 'theorem two_stage : ∀ n, n + 0 = n := by intro n; simp'
}

# Case 3: Divergent case with true_lean_code and false_lean_code
case_divergent = {
    'true_lean_code': 'theorem true_branch : True := trivial',
    'false_lean_code': 'theorem false_branch : False := by contradiction'
}

# Use the divergent case as the primary test case
ARGS = {
    'case': case_divergent
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_provider Ran
ran; our contract draft was wrong, not the code
koreankiwi99/formalization-gaming/src/utils/api_client.py
code served (permissive licence) · get_code("64e253007e4a5ed0")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

# The function get_provider takes a single parameter: model (a string)
# It checks if the model is in MODEL_PROVIDERS dict and returns the provider,
# otherwise returns 'openai'

# Test with a known model
model_known = 'gpt-4o'

# Test with an unknown model
model_unknown = 'some-unknown-model'

ARGS = {
    'model': model_known
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_folio Ran
ran; our contract draft was wrong, not the code
koreankiwi99/formalization-gaming/src/utils/datasets.py
code served (permissive licence) · get_code("60cbd940377d6d13")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
import os

# Create a temporary directory and file with sample FOLIO data
temp_dir = tempfile.mkdtemp()
folio_data = [
    {
        'story_id': 'story_1',
        'example_id': 'ex_1',
        'premises': ['Premise 1', 'Premise 2'],
        'conclusion': 'Conclusion 1',
        'label': 'Entailment'
    },
    {
        'story_id': 'story_2',
        'example_id': 'ex_2',
        'premises': ['Premise 3', 'Premise 4'],
        'conclusion': 'Conclusion 2',
        'label': 'Neutral'
    }
]

folio_path = os.path.join(temp_dir, 'folio-validation.json')
with open(folio_path, 'w') as f:
    json.dump(folio_data, f)

ARGS = {
    'folio_path': folio_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_multilogieval Ran
ran, honoured the contract we drafted
koreankiwi99/formalization-gaming/src/utils/datasets.py
code served (permissive licence) · get_code("150640dc130c1d59")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
from pathlib import Path

# Create temporary directory structure that mimics the expected dataset layout
temp_dir = tempfile.mkdtemp()
data_dir = Path(temp_dir) / 'data' / 'multilogieval' / 'original' / 'data'
data_dir.mkdir(parents=True, exist_ok=True)

# Create sample dataset structure
depths_to_create = [2, 3]
logic_types_to_create = ['propositional', 'first_order']

for depth in depths_to_create:
    depth_dir = data_dir / f'{depth}_Data'
    depth_dir.mkdir(exist_ok=True)
    
    for logic_type in logic_types_to_create:
        logic_dir = depth_dir / logic_type
        logic_dir.mkdir(exist_ok=True)
        
        # Create sample JSON files
        sample_data = {
            'rule': f'rule_{logic_type}_{depth}',
            'samples': [
                {
                    'id': f'sample_1_{logic_type}_{depth}',
                    'context': f'Context for {logic_type} at depth {depth}',
                    'question': f'Question for {logic_type}?',
                    'answer': 'TRUE'
                },
                {
                    'id': f'sample_2_{logic_type}_{depth}',
                    'context': f'Another context for {logic_type}',
                    'question': f'Another question?',
                    'answer': 'FALSE'
                }
            ]
        }
        
        json_file = logic_dir / f'test_{logic_type}.json'
        with open(json_file, 'w', encoding='utf-8') as f:
            json.dump(sample_data, f)

# Patch the function to use our temporary directory
import sys
from unittest.mock import patch

original_load = None

def load_multilogieval_patched(depths, logic_types):
    cases = []
    for depth in depths:
        depth_dir = data_dir / f'{depth}_Data'
        if not depth_dir.exists():
            print(f'Warning: {depth_dir} does not exist')
            continue
        for logic_type in logic_types:
            logic_dir = depth_dir / logic_type
            if not logic_dir.exists():
                continue
            for json_file in logic_dir.glob('*.json'):
                try:
                    with open(json_file, encoding='utf-8') as f:
                        data = json.load(f)
                except UnicodeDecodeError:
                    with open(json_file, encoding='latin-1') as f:
                        data = json.load(f)
                rule = data.get('rule', json_file.stem)
                samples = data.get('samples', [])
                for sample in samples:
                    cases.append({'id': sample.get('id'), 'context': sample.get('context', ''), 'question': sample.get('question', ''), 'ground_truth': sample.get('answer', '').lower(), 'logic_type': logic_type, 'depth': depth, 'rule': rule})
    return cases

ARGS = {
    'depths': [2, 3],
    'logic_types': ['propositional', 'first_order']
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_multilogieval_sampled Ran
ran; our contract draft was wrong, not the code
koreankiwi99/formalization-gaming/src/utils/datasets.py
code served (permissive licence) · get_code("77490f433de992ed")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
import os

# Create a temporary JSON file with sample MultiLogiEval data
sample_data = [
    {
        'idx': 0,
        'id': 'sample_0',
        'context': 'All birds can fly. Tweety is a bird.',
        'question': 'Can Tweety fly?',
        'answer': 'Yes',
        'logic': 'fol',
        'depth': '2',
        'rule': 'modus_ponens'
    },
    {
        'idx': 1,
        'id': 'sample_1',
        'context': 'Some mammals are carnivores. Lions are mammals.',
        'question': 'Are lions carnivores?',
        'answer': 'Possibly',
        'logic': 'fol',
        'depth': '1',
        'rule': 'universal_instantiation'
    },
    {
        'idx': 2,
        'id': 'sample_2',
        'context': 'If it rains, the ground is wet.',
        'question': 'Is the ground wet?',
        'answer': 'Unknown',
        'logic': 'pl',
        'depth': '1',
        'rule': 'conditional'
    }
]

# Create temporary file
temp_file = tempfile.NamedTemporaryFile(mode='w', suffix='.json', delete=False)
json.dump(sample_data, temp_file)
temp_file.close()

ARGS = {
    'data_file': temp_file.name
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_prompt Ran
ran; our contract draft was wrong, not the code
koreankiwi99/formalization-gaming/src/utils/prompts.py
code served (permissive licence) · get_code("872d7c12ef1aca3f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import tempfile
import os

# Create a temporary file with some prompt content
temp_dir = tempfile.mkdtemp()
prompt_file = os.path.join(temp_dir, "test_prompt.txt")
with open(prompt_file, 'w') as f:
    f.write("This is a test prompt with some content.\n")

ARGS = {
    "path": prompt_file
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_prompt_template Ran
ran; our contract draft was wrong, not the code
koreankiwi99/formalization-gaming/src/analysis/analyze_errors.py
code served (permissive licence) · get_code("0f1c89036844e212")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import tempfile
import os

# Create a temporary prompt template file
temp_dir = tempfile.mkdtemp()
prompt_file = os.path.join(temp_dir, "prompt_template.txt")

with open(prompt_file, 'w') as f:
    f.write("You are a helpful assistant.\nRespond to the following query: {query}\n")

ARGS = {
    "prompt_path": prompt_file
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
normalize_answer Ran
ran; our contract draft was wrong, not the code
koreankiwi99/formalization-gaming/src/utils/answer_parsing.py
code served (permissive licence) · get_code("651b0f7470e1a3e9")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Setup for normalize_answer function
ARGS = {
    'answer': 'yes',
    'answer_format': 'yes_no'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
parse_answer Ran
ran; our contract draft was wrong, not the code
koreankiwi99/formalization-gaming/src/utils/answer_parsing.py
code served (permissive licence) · get_code("5200c424eaeecdfc")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def normalize_answer(answer, answer_format='yes_no'):
    """Normalize answer to standard format.

    Args:
        answer: The answer to normalize
        answer_format: "yes_no", "true_false", "bidir_true", or "bidir_false"

    Returns:
        str: Normalized answer
    """
    if not answer:
        return 'Failure' if answer_format.startswith('bidir') else 'Uncertain'
    low = answer.lower().strip()
    if answer_format == 'yes_no':
        if low in ['yes', 'y', 'true', 't', '1']:
            return 'Yes'
        elif low in ['no', 'n', 'false', 'f', '0']:
            return 'No'
        elif low in ['unknown', 'uncertain', 'u']:
            return 'Uncertain'
    elif answer_format == 'true_false':
        if low in ['true', 't', 'yes', 'y']:
            return 'True'
        elif low in ['false', 'f', 'no', 'n']:
            return 'False'
        elif low in ['unknown', 'uncertain', 'u']:
            return 'Uncertain'
    elif answer_format == 'bidir_true':
        if low in ['yes', 'y']:
            return 'Yes'
        elif low in ['true', 't', 'success']:
            return 'True'
        elif low in ['failure', 'failed', 'fail', 'f']:
            return 'Failure'
    elif answer_format == 'bidir_false':
        if low in ['no', 'n']:
            return 'No'
        elif low in ['false', 'f', 'success']:
            return 'False'
        elif low in ['failure', 'failed', 'fail']:
            return 'Failure'
    return 'Failure' if answer_format.startswith('bidir') else 'Uncertain'

def parse_answer(response, answer_format='true_false'):
    """Extract answer from model response.

    Args:
        response: The model's text response
        answer_format: "true_false", "yes_no", "bidir_true", or "bidir_false"

    Returns:
        tuple: (answer, parse_status)
        Status: SUCCESS, FALLBACK, EMPTY, PARSE_FAILED
    """
    if not response or not response.strip():
        return (None, 'EMPTY')
    if answer_format == 'bidir_true':
        pattern = 'ANSWER:\\s*(True|Yes|Failure|Failed)'
        fallback_pattern = '\\b(True|Yes|Failure|Failed)\\b'
    elif answer_format == 'bidir_false':
        pattern = 'ANSWER:\\s*(False|No|Failure|Failed)'
        fallback_pattern = '\\b(False|No|Failure|Failed)\\b'
    elif answer_format == 'true_false':
        pattern = 'ANSWER:\\s*(True|False|Unknown|Uncertain)'
        fallback_pattern = '\\b(True|False|Unknown|Uncertain)\\b'
    else:
        pattern = 'ANSWER:\\s*(Yes|No|True|False|Unknown|Uncertain)'
        fallback_pattern = '\\b(Yes|No|True|False|Unknown|Uncertain)\\b'
    answer_match = re.search(pattern, response, re.IGNORECASE)
    if answer_match:
        answer = normalize_answer(answer_match.group(1), answer_format)
        return (answer, 'SUCCESS')
    all_answers = re.findall(fallback_pattern, response, re.IGNORECASE)
    if all_answers:
        answer = normalize_answer(all_answers[-1], answer_format)
        return (answer, 'FALLBACK')
    return (None, 'PARSE_FAILED')


ARGS = {
    'response': 'The model says: ANSWER: True with confidence.',
    'answer_format': 'true_false',
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
parse_folio_answer Ran
ran; our contract draft was wrong, not the code
koreankiwi99/formalization-gaming/src/utils/answer_parsing.py
code served (permissive licence) · get_code("75f5e519aa2c36de")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def normalize_answer(answer, answer_format='yes_no'):
    """Normalize answer to standard format.

    Args:
        answer: The answer to normalize
        answer_format: "yes_no", "true_false", "bidir_true", or "bidir_false"

    Returns:
        str: Normalized answer
    """
    if not answer:
        return 'Failure' if answer_format.startswith('bidir') else 'Uncertain'
    low = answer.lower().strip()
    if answer_format == 'yes_no':
        if low in ['yes', 'y', 'true', 't', '1']:
            return 'Yes'
        elif low in ['no', 'n', 'false', 'f', '0']:
            return 'No'
        elif low in ['unknown', 'uncertain', 'u']:
            return 'Uncertain'
    elif answer_format == 'true_false':
        if low in ['true', 't', 'yes', 'y']:
            return 'True'
        elif low in ['false', 'f', 'no', 'n']:
            return 'False'
        elif low in ['unknown', 'uncertain', 'u']:
            return 'Uncertain'
    elif answer_format == 'bidir_true':
        if low in ['yes', 'y']:
            return 'Yes'
        elif low in ['true', 't', 'success']:
            return 'True'
        elif low in ['failure', 'failed', 'fail', 'f']:
            return 'Failure'
    elif answer_format == 'bidir_false':
        if low in ['no', 'n']:
            return 'No'
        elif low in ['false', 'f', 'success']:
            return 'False'
        elif low in ['failure', 'failed', 'fail']:
            return 'Failure'
    return 'Failure' if answer_format.startswith('bidir') else 'Uncertain'

def parse_answer(response, answer_format='true_false'):
    """Extract answer from model response.

    Args:
        response: The model's text response
        answer_format: "true_false", "yes_no", "bidir_true", or "bidir_false"

    Returns:
        tuple: (answer, parse_status)
        Status: SUCCESS, FALLBACK, EMPTY, PARSE_FAILED
    """
    if not response or not response.strip():
        return (None, 'EMPTY')
    if answer_format == 'bidir_true':
        pattern = 'ANSWER:\\s*(True|Yes|Failure|Failed)'
        fallback_pattern = '\\b(True|Yes|Failure|Failed)\\b'
    elif answer_format == 'bidir_false':
        pattern = 'ANSWER:\\s*(False|No|Failure|Failed)'
        fallback_pattern = '\\b(False|No|Failure|Failed)\\b'
    elif answer_format == 'true_false':
        pattern = 'ANSWER:\\s*(True|False|Unknown|Uncertain)'
        fallback_pattern = '\\b(True|False|Unknown|Uncertain)\\b'
    else:
        pattern = 'ANSWER:\\s*(Yes|No|True|False|Unknown|Uncertain)'
        fallback_pattern = '\\b(Yes|No|True|False|Unknown|Uncertain)\\b'
    answer_match = re.search(pattern, response, re.IGNORECASE)
    if answer_match:
        answer = normalize_answer(answer_match.group(1), answer_format)
        return (answer, 'SUCCESS')
    all_answers = re.findall(fallback_pattern, response, re.IGNORECASE)
    if all_answers:
        answer = normalize_answer(all_answers[-1], answer_format)
        return (answer, 'FALLBACK')
    return (None, 'PARSE_FAILED')

def parse_folio_answer(response, return_status=False):
    """Extract True/False/Unknown answer from FOLIO response."""
    answer, status = parse_answer(response, answer_format='true_false')
    if return_status:
        return (answer, status)
    return answer


ARGS = {
    'response': 'The statement is true. ANSWER: True',
    'return_status': False
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
resolve_model_name Ran
ran; our contract draft was wrong, not the code
koreankiwi99/formalization-gaming/src/utils/api_client.py
code served (permissive licence) · get_code("a8c0ee30b68f960e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

# Setup code for resolve_model_name function

ARGS = {
    'model': 'llama-70b'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
create_client Not yet run
unverified (harvested, no recorded run)
koreankiwi99/formalization-gaming/src/utils/api_client.py
code served (permissive licence) · get_code("88e267ab5c120ee7")
How it ran

no invocation recorded.

format_user_prompt Not yet run
unverified (harvested, no recorded run)
koreankiwi99/formalization-gaming/src/utils/prompts.py
code served (permissive licence) · get_code("6dbbca04c916feb9")
How it ran

no invocation recorded.

Also present in other papers' code

Of 17 samples lifted from this paper's linked repositories, 17 carry a function identity and 0 do not. Of the 17 distinct functions with an identity, 0 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 0 later) and 17 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Formal verification guarantees proof validity but not formalization faithfulness. For natural-language logical reasoning, where models construct axiom systems from scratch without library constraints, this gap between valid proofs and faithful translations is especially acute. We investigate whether frontier models exploit this gap when generating Lean 4 proofs, a behavior we term formalization gaming. We evaluate GPT-5 and DeepSeek-R1 on 303 first-order logic problems (203 from FOLIO, 100 from Multi-LogiEval), comparing unified generation against a twostage pipeline that separates formalization from proving. Despite compilation rates of 87-99%, we find no evidence of systematic gaming in unified generation: models prefer reporting failure over forcing proofs, even under prompting designed to encourage it. However, unfaithfulness that evades our detection signals may still occur. The two-stage pipeline reveals two distinct modes of unfaithfulness: GPT-5 fabricates axioms during proof generation, a reactive fallback detectable via cross-stage comparison, while DeepSeek-R1 mistranslates premises during formalization, producing internally consistent outputs that evade detection entirely. These findings show that high compilation rates or accuracies should not be equated with faithful reasoning. Code and data are available at https://github.com/koreankiwi99/formalization-gaming.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2604.19459")
get_code_for_paper("2604.19459")
have("2604.19459")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2604.19459.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2604.19459.svg)](https://syntology.ai/paper/2604.19459)

Connect an agent — have() is free.