SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2402.17709 · 2024

Case-Based or Rule-Based: How Do Transformers Do the Math?

arXiv · PDF · Open in the Atlas

Code that ran

13 of 13 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 2 honoured the contract we drafted, 2 violated it and 0 have no contract verdict; 9 failures of our instrument; 0 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2402.17709.json · badge: /api/ran/2402.17709.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
graphpku/case_or_rule canonical (classifier label)
classifier basis: census
13 ran · 0 no recorded run
FunctionStatusWhere it lives
convert_to_base Ran
ran; our contract draft was wrong, not the code
graphpku/case_or_rule/datasets/base_addition/cot_gen.py
code served (permissive licence) · get_code("e9c0fae796957d4a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Analyze the function:
# - convert_to_base(n, base) takes two parameters
# - n: gets converted to int, used in arithmetic operations (%, //, <)
# - base: used as divisor and comparison value in arithmetic
# - Both need to be numeric types that support int() conversion

# Test with reasonable values
n = 255
base = 16

ARGS = {
    "n": n,
    "base": base
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
convert_to_base Ran
ran; our contract draft was wrong, not the code
graphpku/case_or_rule/datasets/base_addition/create.py
code served (permissive licence) · get_code("fe904288aa90f847")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Analyzing the function convert_to_base(n, base):
# - n: should be an integer (used in arithmetic operations: n < base, n % base, n // base)
# - base: should be an integer >= 2 (the base for conversion)
# The function converts an integer n to its representation in a given base

ARGS = {
    'n': 42,
    'base': 16
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
count_add Ran
ran, honoured the contract we drafted
graphpku/case_or_rule/datasets/base_addition/cot_gen.py
code served (permissive licence) · get_code("e9e1e1467f75d968")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

ARGS = {
    "a_digit": 3,
    "b_digit": 5,
    "c": 0,
    "base": 10
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family int — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
count_c Ran
ran, honoured the contract we drafted
graphpku/case_or_rule/datasets/addition/cot_gen.py
code served (permissive licence) · get_code("911338d98ca3a4ba")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

ARGS = {
    "a_digit": "5",
    "b_digit": "3",
    "c": "2"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_answer Ran
ran; our contract draft was wrong, not the code
graphpku/case_or_rule/datasets/addition/cot_gen.py
code served (permissive licence) · get_code("fd391d2d2217abf1")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

def extract_answer(rationale):
    answer = rationale.split('\n')[-1].split(',')[1]
    return answer

# The function expects a string (rationale) that:
# 1. Contains newlines
# 2. The last line contains a comma-separated value
# 3. We extract the second element (index 1) after splitting by comma

rationale = "This is the first line\nThis is the second line\nSome text,answer_value"

ARGS = {
    "rationale": rationale
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
gen_cot_rationale Ran
ran; our contract draft was wrong, not the code
graphpku/case_or_rule/icl/addition_icl.py
code served (permissive licence) · get_code("bf5313f0e6343f5a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import random

class BaseInt(int):

    def __new__(cls, value: int) -> 'BaseInt':
        BASE = 10
        for s in str(value):
            if int(s) >= BASE:
                raise ValueError(f'The value {value} is not a valid base {BASE} integer.')
        return super().__new__(cls, value)

    @staticmethod
    def _from_base_10_int(value: int) -> 'BaseInt':
        BASE = 10
        if value < 0:
            raise ValueError('The value should be a non-negative integer.')
        if value == 0:
            return BaseInt(0)
        res = ''
        while value > 0:
            res = str(value % BASE) + res
            value //= BASE
        return BaseInt(int(res))

    @staticmethod
    def random_generate_length(length: int) -> 'BaseInt':
        BASE = 10
        if length <= 0:
            raise ValueError('The length should be a positive integer.')
        return BaseInt(int(''.join([str(random.randint(1, BASE - 1))] + [str(random.randint(0, BASE - 1)) for _ in range(length - 1)])))

    def _to_base_10_int(self) -> int:
        BASE = 10
        return int(str(self), BASE)

    def __add__(self, another: 'BaseInt') -> 'BaseInt':
        if not isinstance(another, BaseInt):
            raise TypeError('The addition should only be performed between two BaseInt objects.')
        return BaseInt._from_base_10_int(self._to_base_10_int() + another._to_base_10_int())

    def __sub__(self, another: 'BaseInt') -> 'BaseInt':
        return BaseInt._from_base_10_int(self._to_base_10_int() - another._to_base_10_int())


random.seed(42)
a = BaseInt(23)
b = BaseInt(15)

ARGS = {
    'a': a,
    'b': b
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
gen_cot_rationale Ran
ran; our contract draft was wrong, not the code
graphpku/case_or_rule/icl/base_addition_icl.py
code served (permissive licence) · get_code("15da454109dd8217")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import random

# Set seed for reproducibility
random.seed(0)

# Import the BaseInt class and function
BASE = 9

class BaseInt(int):

    def __new__(cls, value: int) -> 'BaseInt':
        for s in str(value):
            if int(s) >= BASE:
                raise ValueError(f'The value {value} is not a valid base {BASE} integer.')
        return super().__new__(cls, value)

    @staticmethod
    def _from_base_10_int(value: int) -> 'BaseInt':
        if value < 0:
            raise ValueError('The value should be a non-negative integer.')
        if value == 0:
            return BaseInt(0)
        res = ''
        while value > 0:
            res = str(value % BASE) + res
            value //= BASE
        return BaseInt(int(res))

    @staticmethod
    def random_generate_length(length: int) -> 'BaseInt':
        if length <= 0:
            raise ValueError('The length should be a positive integer.')
        return BaseInt(int(''.join([str(random.randint(1, BASE - 1))] + [str(random.randint(0, BASE - 1)) for _ in range(length - 1)])))

    def _to_base_10_int(self) -> int:
        return int(str(self), BASE)

    def __add__(self, another: 'BaseInt') -> 'BaseInt':
        if not isinstance(another, BaseInt):
            raise TypeError('The addition should only be performed between two BaseInt objects.')
        return BaseInt._from_base_10_int(self._to_base_10_int() + another._to_base_10_int())

    def __sub__(self, another: 'BaseInt') -> 'BaseInt':
        return BaseInt._from_base_10_int(self._to_base_10_int() - another._to_base_10_int())


# Create valid BaseInt instances for testing
# Valid base-9 numbers (digits 0-8 only)
a = BaseInt(123)  # Valid: all digits < 9
b = BaseInt(456)  # Valid: all digits < 9

ARGS = {
    'a': a,
    'b': b,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
gen_cot_rationale Ran
ran; our contract draft was wrong, not the code
graphpku/case_or_rule/datasets/addition/cot_gen.py
code served (permissive licence) · get_code("50e8a6954f9eb0dd")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Analyze the function gen_cot_rationale to understand parameters:
# - a: an integer (used as str(a), converted to digits)
# - b: an integer (used as str(b), converted to digits)
# - gt: ground truth, an integer (compared via int(extract_answer(rationale)) == int(gt))
# - cot: string parameter with default 'scratch' (checked with == 'scratch')

# The function performs digit-by-digit addition with carry
# Let's use simple small numbers where addition is straightforward

# Example: a=12, b=34
# Expected result: 12 + 34 = 46
a = 12
b = 34
gt = 46
cot = 'scratch'

ARGS = {
    'a': a,
    'b': b,
    'gt': gt,
    'cot': cot
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
gen_rf_rationale Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
graphpku/case_or_rule/icl/addition_icl.py
code served (permissive licence) · get_code("ff98966dd80042c6")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import random

BASE = 10

class BaseInt(int):

    def __new__(cls, value: int) -> 'BaseInt':
        for s in str(value):
            if int(s) >= BASE:
                raise ValueError(f'The value {value} is not a valid base {BASE} integer.')
        return super().__new__(cls, value)

    @staticmethod
    def _from_base_10_int(value: int) -> 'BaseInt':
        if value < 0:
            raise ValueError('The value should be a non-negative integer.')
        if value == 0:
            return BaseInt(0)
        res = ''
        while value > 0:
            res = str(value % BASE) + res
            value //= BASE
        return BaseInt(int(res))

    @staticmethod
    def random_generate_length(length: int) -> 'BaseInt':
        if length <= 0:
            raise ValueError('The length should be a positive integer.')
        return BaseInt(int(''.join([str(random.randint(1, BASE - 1))] + [str(random.randint(0, BASE - 1)) for _ in range(length - 1)])))

    def _to_base_10_int(self) -> int:
        return int(str(self), BASE)

    def __add__(self, another: 'BaseInt') -> 'BaseInt':
        if not isinstance(another, BaseInt):
            raise TypeError('The addition should only be performed between two BaseInt objects.')
        return BaseInt._from_base_10_int(self._to_base_10_int() + another._to_base_10_int())

    def __sub__(self, another: 'BaseInt') -> 'BaseInt':
        return BaseInt._from_base_10_int(self._to_base_10_int() - another._to_base_10_int())

# Create a mock prompt_base10 module since it's imported in the function
class MockPrompt:
    NUM = "Adding {} and {}\n"
    INITIALIZE = "Initialize variables\n"
    CHECK_THE_STOP_CRITERION_2_1_ENTER = "Check stop criterion: a_s={}, b_s={}, bool(a_s)={}, bool(b_s)={}, continue={}\n"
    ONE_ITERATION_2_2 = "One iteration\n"
    POP_DIGIT = "Pop digit from position {}: list={}, has_element={}, after={}, popped={}\n"
    NO_POP_DIGIT = "No digit to pop from position {}\n"
    TOTAL_RESULT_CARRY = "digit1={}, digit2={}, carry={}, total={}, result={}, total={}, total%BASE={}, new_result={}, total={}, new_carry={}\n"
    CHECK_THE_STOP_CRITERION_2_1_END = "Check end: a_s={}, b_s={}, bool(a_s)={}, bool(b_s)={}, continue={}\n"
    CHECK_REMAINING_CARRY_TRUE = "Remaining carry: result={}, new_result={}\n"
    CHECK_REMAINING_CARRY_FALSE = "No remaining carry: result={}\n"
    RETURN_THE_RESULT = "Return result: {}\n"

import sys
from types import ModuleType
prompt_module = ModuleType('prompt_base10')
prompt_module.NUM = MockPrompt.NUM
prompt_module.INITIALIZE = MockPrompt.INITIALIZE
prompt_module.CHECK_THE_STOP_CRITERION_2_1_ENTER = MockPrompt.CHECK_THE_STOP_CRITERION_2_1_ENTER
prompt_module.ONE_ITERATION_2_2 = MockPrompt.ONE_ITERATION_2_2
prompt_module.POP_DIGIT = MockPrompt.POP_DIGIT
prompt_module.NO_POP_DIGIT = MockPrompt.NO_POP_DIGIT
prompt_module.TOTAL_RESULT_CARRY = MockPrompt.TOTAL_RESULT_CARRY
prompt_module.CHECK_THE_STOP_CRITERION_2_1_END = MockPrompt.CHECK_THE_STOP_CRITERION_2_1_END
prompt_module.CHECK_REMAINING_CARRY_TRUE = MockPrompt.CHECK_REMAINING_CARRY_TRUE
prompt_module.CHECK_REMAINING_CARRY_FALSE = MockPrompt.CHECK_REMAINING_CARRY_FALSE
prompt_module.RETURN_THE_RESULT = MockPrompt.RETURN_THE_RESULT
sys.modules['prompt_base10'] = prompt_module

a = BaseInt(123)
b = BaseInt(456)

ARGS = {
    'a': a,
    'b': b
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
prompt_base10 — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
gen_rf_rationale Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
graphpku/case_or_rule/icl/base_addition_icl.py
code served (permissive licence) · get_code("a657547b47f88040")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import random

# Set seed for reproducibility
random.seed(0)

# BASE is defined as 9 in the function
BASE = 9

class BaseInt(int):
    def __new__(cls, value: int) -> 'BaseInt':
        for s in str(value):
            if int(s) >= BASE:
                raise ValueError(f'The value {value} is not a valid base {BASE} integer.')
        return super().__new__(cls, value)

    @staticmethod
    def _from_base_10_int(value: int) -> 'BaseInt':
        if value < 0:
            raise ValueError('The value should be a non-negative integer.')
        if value == 0:
            return BaseInt(0)
        res = ''
        while value > 0:
            res = str(value % BASE) + res
            value //= BASE
        return BaseInt(int(res))

    @staticmethod
    def random_generate_length(length: int) -> 'BaseInt':
        if length <= 0:
            raise ValueError('The length should be a positive integer.')
        return BaseInt(int(''.join([str(random.randint(1, BASE - 1))] + [str(random.randint(0, BASE - 1)) for _ in range(length - 1)])))

    def _to_base_10_int(self) -> int:
        return int(str(self), BASE)

    def __add__(self, another: 'BaseInt') -> 'BaseInt':
        if not isinstance(another, BaseInt):
            raise TypeError('The addition should only be performed between two BaseInt objects.')
        return BaseInt._from_base_10_int(self._to_base_10_int() + another._to_base_10_int())

    def __sub__(self, another: 'BaseInt') -> 'BaseInt':
        return BaseInt._from_base_10_int(self._to_base_10_int() - another._to_base_10_int())

# Create a mock prompt_base9 module
class MockPrompt:
    NUM = "Numbers: {} and {}\n"
    INITIALIZE = "Initialize carry and result\n"
    CHECK_THE_STOP_CRITERION_2_1_ENTER = "Check stop criterion: a_s={}, b_s={}, bool(a_s)={}, bool(b_s)={}, condition={}\n"
    ONE_ITERATION_2_2 = "Performing one iteration\n"
    POP_DIGIT = "Pop digit from list {}: {} -> {}, digit={}\n"
    NO_POP_DIGIT = "No digit to pop from list {}\n"
    TOTAL_RESULT_CARRY = "digit1={}, digit2={}, carry={}, total={}, result={}, total={}, total%BASE={}, new_result={}, total={}, carry={}\n"
    CHECK_THE_STOP_CRITERION_2_1_END = "End check: a_s={}, b_s={}, bool(a_s)={}, bool(b_s)={}, condition={}\n"
    CHECK_REMAINING_CARRY_TRUE = "Carry remaining: result={}, new_result={}\n"
    CHECK_REMAINING_CARRY_FALSE = "No remaining carry: result={}\n"
    RETURN_THE_RESULT = "Return result: {}\n"

import sys
from types import ModuleType
prompt_base9 = ModuleType('prompt_base9')
for attr in dir(MockPrompt):
    if not attr.startswith('_'):
        setattr(prompt_base9, attr, getattr(MockPrompt, attr))
sys.modules['prompt_base9'] = prompt_base9

# Create valid BaseInt arguments for the function
a = BaseInt(123)  # Valid base 9 number
b = BaseInt(456)  # Valid base 9 number

ARGS = {
    'a': a,
    'b': b
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
prompt_base9 — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
generate_dataset Ran
ran; our contract draft was wrong, not the code
graphpku/case_or_rule/datasets/addition/create.py
code served (permissive licence) · get_code("fd1a2d263d9f32ab")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Define ARGS dict with parameters for generate_dataset
ARGS = {
    'a_range': range(2, 5),  # [2, 3, 4]
    'b_range': range(10, 13)  # [10, 11, 12]
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
is_easy_add Ran
ran, violated the contract we drafted
graphpku/case_or_rule/icl/addition_icl.py
code served (permissive licence) · get_code("8c805435cf14daaa")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import random

BASE = 10

class BaseInt(int):

    def __new__(cls, value: int) -> 'BaseInt':
        for s in str(value):
            if int(s) >= BASE:
                raise ValueError(f'The value {value} is not a valid base {BASE} integer.')
        return super().__new__(cls, value)

    @staticmethod
    def _from_base_10_int(value: int) -> 'BaseInt':
        if value < 0:
            raise ValueError('The value should be a non-negative integer.')
        if value == 0:
            return BaseInt(0)
        res = ''
        while value > 0:
            res = str(value % BASE) + res
            value //= BASE
        return BaseInt(int(res))

    @staticmethod
    def random_generate_length(length: int) -> 'BaseInt':
        if length <= 0:
            raise ValueError('The length should be a positive integer.')
        return BaseInt(int(''.join([str(random.randint(1, BASE - 1))] + [str(random.randint(0, BASE - 1)) for _ in range(length - 1)])))

    def _to_base_10_int(self) -> int:
        return int(str(self), BASE)

    def __add__(self, another: 'BaseInt') -> 'BaseInt':
        if not isinstance(another, BaseInt):
            raise TypeError('The addition should only be performed between two BaseInt objects.')
        return BaseInt._from_base_10_int(self._to_base_10_int() + another._to_base_10_int())

    def __sub__(self, another: 'BaseInt') -> 'BaseInt':
        return BaseInt._from_base_10_int(self._to_base_10_int() - another._to_base_10_int())

def is_easy_add(a: BaseInt, b: BaseInt) -> bool:
    """
    Return whether there is no carry in the (base) addition of a and b.
    """
    return all([int(digit) + int(b_digit) < BASE for digit, b_digit in zip(str(a)[::-1], str(b)[::-1])])


# Setup code to construct valid arguments
a = BaseInt(123)
b = BaseInt(456)

ARGS = {
    'a': a,
    'b': b
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
is_easy_add Ran
ran, violated the contract we drafted
graphpku/case_or_rule/icl/base_addition_icl.py
code served (permissive licence) · get_code("daa3be68c856dbbd")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import random

BASE = 9

class BaseInt(int):

    def __new__(cls, value: int) -> 'BaseInt':
        for s in str(value):
            if int(s) >= BASE:
                raise ValueError(f'The value {value} is not a valid base {BASE} integer.')
        return super().__new__(cls, value)

    @staticmethod
    def _from_base_10_int(value: int) -> 'BaseInt':
        if value < 0:
            raise ValueError('The value should be a non-negative integer.')
        if value == 0:
            return BaseInt(0)
        res = ''
        while value > 0:
            res = str(value % BASE) + res
            value //= BASE
        return BaseInt(int(res))

    @staticmethod
    def random_generate_length(length: int) -> 'BaseInt':
        if length <= 0:
            raise ValueError('The length should be a positive integer.')
        return BaseInt(int(''.join([str(random.randint(1, BASE - 1))] + [str(random.randint(0, BASE - 1)) for _ in range(length - 1)])))

    def _to_base_10_int(self) -> int:
        return int(str(self), BASE)

    def __add__(self, another: 'BaseInt') -> 'BaseInt':
        if not isinstance(another, BaseInt):
            raise TypeError('The addition should only be performed between two BaseInt objects.')
        return BaseInt._from_base_10_int(self._to_base_10_int() + another._to_base_10_int())

    def __sub__(self, another: 'BaseInt') -> 'BaseInt':
        return BaseInt._from_base_10_int(self._to_base_10_int() - another._to_base_10_int())

def is_easy_add(a: BaseInt, b: BaseInt) -> bool:
    """
    Return whether there is no carry in the (base) addition of a and b.
    """
    return all([int(digit) + int(b_digit) < BASE for digit, b_digit in zip(str(a)[::-1], str(b)[::-1])])

random.seed(42)

a = BaseInt(123)
b = BaseInt(345)

ARGS = {
    'a': a,
    'b': b
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block

Also present in other papers' code

Of 13 samples lifted from this paper's linked repositories, 13 carry a function identity and 0 do not. Of the 11 distinct functions with an identity, 0 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 0 later) and 11 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Despite the impressive performance in a variety of complex tasks, modern large language models (LLMs) still have trouble dealing with some math problems that are simple and intuitive for humans, such as addition. While we can easily learn basic rules of addition and apply them to new problems of any length, LLMs struggle to do the same. Instead, they may rely on similar cases seen in the training corpus for help. We define these two different reasoning mechanisms as "rule-based reasoning" and "case-based reasoning". Since rule-based reasoning is essential for acquiring systematic generalization ability, we aim to explore exactly whether transformers use rule-based or case-based reasoning for math problems. Through carefully designed intervention experiments on five math tasks, we confirm that transformers are performing case-based reasoning, no matter whether scratchpad is used, which aligns with the previous observations that transformers use subgraph matching/shortcut learning to reason. To mitigate such problems, we propose a Rule-Following Fine-Tuning (RFFT) technique to teach transformers to perform rule-based reasoning. Specifically, we provide explicit rules in the input and then instruct transformers to recite and follow the rules step by step. Through RFFT, we successfully enable LLMs fine-tuned on 1-5 digit addition to generalize to up to 12-digit addition with over 95% accuracy, which is over 40% higher than scratchpad. The significant improvement demonstrates that teaching LLMs to use rules explicitly helps them learn rule-based reasoning and generalize better in length.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2402.17709")
get_code_for_paper("2402.17709")
have("2402.17709")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2402.17709.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2402.17709.svg)](https://syntology.ai/paper/2402.17709)

Connect an agent — have() is free.