SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2603.06642 · 2026

SR-TTT Does Not Learn Retrieval: A Correction and Mechanistic Post-Mortem of Surprisal-Aware Residual Test-Time Training

arXiv · PDF · Open in the Atlas

Code that ran

6 of 9 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 2 honoured the contract we drafted, 0 violated it and 0 have no contract verdict; 4 failures of our instrument; 3 no recorded run), measured 2026-11-22; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2603.06642.json · badge: /api/ran/2603.06642.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
swamynathanvp/Surprisal-Aware-Residual-Test-Time-Training canonical (classifier label)
classifier basis: census
6 ran · 3 no recorded run
FunctionStatusWhere it lives
answer_pred_logits Ran
ran; our contract draft was wrong, not the code
swamynathanvp/Surprisal-Aware-Residual-Test-Time-Training/sr_ttt_fixed.py
code served (permissive licence) · get_code("67d87247cd7b7afa")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 1, dtype family float, shape (3,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function indexes into logits at dimension 0 and dimension 1
# logits[0, answer_start - 1:answer_start - 1 + needle_len]
# So logits needs to be a 2D tensor with at least 1 row (batch dimension)
# and enough columns to accommodate the slice

logits = torch.randn(2, 10)  # batch=2, sequence_length=10

# answer_start is used as an index: answer_start - 1
# and in a range calculation: answer_start - 1 + needle_len
# This should be a positive integer
answer_start = 5

# needle_len is used in range calculation: answer_start - 1 + needle_len
# This should be a positive integer representing length
needle_len = 3

ARGS = {
    "logits": logits,
    "answer_start": answer_start,
    "needle_len": needle_len,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 1, dtype family float, shape (3,) — shape (3,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
answer_pred_logits Ran
ran; our contract draft was wrong, not the code
swamynathanvp/Surprisal-Aware-Residual-Test-Time-Training/sr_ttt_v2.py
code served (permissive licence) · get_code("05da6f75c9d18d0d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function indexes into logits[0, ...], so logits needs to be a 2D tensor
# with batch dimension first. It accesses logits[0, answer_start - 1:answer_start - 1 + needle_len]
# so we need enough columns to support the slice.
logits = torch.randn(2, 10)  # batch=2, sequence_length=10

# answer_start is used as an index into the logits tensor
# Setting it to 3 means we'll slice from logits[0, 2:2+needle_len]
answer_start = 3

# needle_len is the length of the slice we want to extract
needle_len = 4

ARGS = {
    "logits": logits,
    "answer_start": answer_start,
    "needle_len": needle_len,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
apply_rotary_pos_emb Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
swamynathanvp/Surprisal-Aware-Residual-Test-Time-Training/sr_ttt_fixed.py
code served (permissive licence) · get_code("e96c58f6ba701d7c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 3, dtype family float, shape (2, 3, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Construct tiny tensors for the function
# x: input tensor with shape (batch, seq_len, hidden_dim)
# For 4D case: (batch, num_heads, seq_len, head_dim)
# For 3D case: (batch, seq_len, hidden_dim)

# Let's test the 3D case first
batch_size = 2
seq_len = 3
hidden_dim = 4

x = torch.randn(batch_size, seq_len, hidden_dim)

# cos and sin should have shape (seq_len, hidden_dim)
cos = torch.randn(seq_len, hidden_dim)
sin = torch.randn(seq_len, hidden_dim)

# positions is optional - can be None or a tensor of indices
positions = None

ARGS = {
    "x": x,
    "cos": cos,
    "sin": sin,
    "positions": positions
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float, shape (2, 3, 4) — shape (2, 3, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
rotate_half Ran
ran; our contract draft was wrong, not the code
swamynathanvp/Surprisal-Aware-Residual-Test-Time-Training/sr_ttt_kaggle.py
code served (permissive licence) · get_code("e03d53ba9d4f9ae5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function takes a single parameter 'x' which is a tensor
# It accesses x[..., :x.shape[-1] // 2] and x[..., x.shape[-1] // 2:]
# So x needs to be a tensor with a last dimension that can be split in half
# Let's create a small tensor with batch size 2 and feature dimension 4

x = torch.randn(2, 4)

ARGS = {
    "x": x
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
rotate_half Ran
ran, honoured the contract we drafted
swamynathanvp/Surprisal-Aware-Residual-Test-Time-Training/sr_ttt_v2.py
pointer only (licence: MIT); the stored copy get_code serves, from tdooms/bilinear-decomposition, is pointer only (licence: NONE) · get_code("437c7011506995a3")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from tdooms/bilinear-decomposition, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

rotate_half Ran
ran, honoured the contract we drafted
swamynathanvp/Surprisal-Aware-Residual-Test-Time-Training/sr_ttt_fixed.py
pointer only (licence: MIT); the stored copy get_code serves, from mustafaaljadery/gemma-2b-10m, is pointer only (licence: NONE) · get_code("cec833a99feb528c")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from mustafaaljadery/gemma-2b-10m, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

apply_rotary_pos_emb Not yet run
unverified (harvested, no recorded run)
swamynathanvp/Surprisal-Aware-Residual-Test-Time-Training/sr_ttt_kaggle.py
code served (permissive licence) · get_code("db4420e92850c6d0")
How it ran

no invocation recorded.

apply_rotary_pos_emb Not yet run
unverified (harvested, no recorded run)
swamynathanvp/Surprisal-Aware-Residual-Test-Time-Training/sr_ttt_v2.py
code served (permissive licence) · get_code("42256aaa3ae587e2")
How it ran

no invocation recorded.

create_model Not yet run
unverified (harvested, no recorded run)
swamynathanvp/Surprisal-Aware-Residual-Test-Time-Training/sr_ttt_kaggle.py
code served (permissive licence) · get_code("d87f2ef9e5d22389")
How it ran

no invocation recorded.

Also present in other papers' code

Of 9 samples lifted from this paper's linked repositories, 9 carry a function identity and 0 do not. Of the 7 distinct functions with an identity, 2 are also present in other papers whose code we hold (2 with at least one paper dated earlier, 1 the same month, 2 later) and 5 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
rotate_half 360 other papers
331 dated earlier, 5 the same month, 21 later, 3 undated · 30 only as an assembly brick
1901.02860 (2019-01)
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
2203.08913 (2022-03)
Memorizing Transformers
2302.13971 (2023-02)
LLaMA: Open and Efficient Foundation Language Models
and 357 more not named here
rotate_half 13 other papers
11 dated earlier, 2 later · 2 only as an assembly brick
2202.03036 (2022-02)
Structure-Aware Transformer for Graph Representation Learning
2203.07852 (2022-03)
Block-Recurrent Transformers
2403.00043 (2024-03)
RiNALMo: General-Purpose RNA Language Models Can Generalize Well on Structure Prediction Tasks
and 10 more not named here

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Version 1 of this paper reported that SR-TTT improved Needle-in-a-Haystack exact match by +23% (depth 0.50) and +20% (depth 0.75) over a pure TTT baseline at 2048-token contexts. These results were artifacts of evaluation and implementation flaws, principally an off-by-one label alignment that reduced the task to copying an answer already present in the input, compounded by non-causal information flow in both the residual cache and the TTT inner loop. Under a corrected, leak-free protocol the reported advantage vanishes entirely (zero discordant trials out of 750 paired comparisons). All headline claims of v1 are retracted. This version documents the flaws, provides a corrected implementation with runtime causality verification, and reports a mechanistic post-mortem: even with the evaluation fixed, trainable retrieval machinery added, and storage guaranteed by an oracle, surprisal-gated exact-KV caching fails to learn needle retrieval, for reasons we measure and decompose.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2603.06642")
get_code_for_paper("2603.06642")
have("2603.06642")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2603.06642.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2603.06642.svg)](https://syntology.ai/paper/2603.06642)

Connect an agent — have() is free.