SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2503.12295 · ICLR · 2025

Towards Learning High-Precision Least Squares Algorithms with Sequence Models

Jerry Liu, Atri Rudra, Simran Arora, Jessica Grogan, Owen Dugan, Ashish Rao, Chris Ré

arXiv · PDF · Open in the Atlas

Code that ran

4 of 10 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (4 of them constructed an object rather than computing a result; 0 honoured the contract we drafted, 0 violated it and 4 have no contract verdict; 0 failures of our instrument; 6 no recorded run), measured 2026-11-22; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2503.12295.json · badge: /api/ran/2503.12295.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
HazyResearch/precision-ls marked official in Papers with Code
Papers with Code archive
4 ran · 6 no recorded run
FunctionStatusWhere it lives
CausalSelfAttention Ran
ran on a synthesized input · constructed an object; no result was computed
HazyResearch/precision-ls/src/models/gpt2.py
code served (permissive licence) · get_code("099feb2cb11280c5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, CausalSelfAttention: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import math

torch.manual_seed(0)

# Create a minimal config class with all required attributes
class Config:
    def __init__(self):
        self.n_embd = 64
        self.n_head = 4
        self.block_size = 8
        self.bias = True
        self.dropout = 0.1
        self.use_softmax_for_attn = True
        self.causal = True

config = Config()

# Create input tensor: (batch_size, seq_len, embedding_dim)
x = torch.randn(2, 8, 64)

# Define ARGS dictionary with all parameters
ARGS = {
    'config': config,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type CausalSelfAttention — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a CausalSelfAttention output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
GLU Ran
ran on a synthesized input · constructed an object; no result was computed
HazyResearch/precision-ls/src/models/gpt2.py
code served (permissive licence) · get_code("9e9cb8d5e05b5a61")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, GLU: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

class GLU(nn.Module):
    def __init__(self, d_model):
        super().__init__()
        self.d_model = d_model
        self.proj1 = nn.Linear(in_features=d_model, out_features=d_model, bias=False)
        self.proj2 = nn.Linear(in_features=d_model, out_features=d_model, bias=False)

    def forward(self, x):
        return torch.sigmoid(self.proj1(x)) * self.proj2(x)

# Create instance for testing
glu_instance = GLU(d_model=4)

# Create input tensor: batch_size=2, feature_dim=4
x = torch.randn(2, 4)

ARGS = {
    'd_model': 4
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type GLU — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a GLU output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
GPT2Config Ran
ran on a synthesized input · constructed an object; no result was computed
HazyResearch/precision-ls/src/models/gpt2.py
code served (permissive licence) · get_code("57d4e2dacaca4be8")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, GPT2Config: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from dataclasses import dataclass

torch.manual_seed(0)

@dataclass
class GPT2Config:
    block_size: int = 101
    n_layer: int = 12
    n_head: int = 8
    n_embd: int = 256
    dropout: float = 0.0
    bias: bool = True
    use_mlps: bool = True
    seq_op: str = 'attn'
    mlp_activ: str = 'gelu'
    use_resid: bool = True
    use_seqop_ln: bool = True
    use_final_ln: bool = True
    use_mlp_ln: bool = True
    conv_type: str = 'short'
    causal: bool = True
    mlp_upfactor: int = 4
    use_softmax_for_attn: bool = True

    def copy(self):
        return GPT2Config(block_size=self.block_size, n_layer=self.n_layer, n_head=self.n_head, n_embd=self.n_embd, dropout=self.dropout, bias=self.bias, use_mlps=self.use_mlps, seq_op=self.seq_op, mlp_activ=self.mlp_activ, use_resid=self.use_resid, use_seqop_ln=self.use_seqop_ln, use_final_ln=self.use_final_ln, use_mlp_ln=self.use_mlp_ln, conv_type=self.conv_type, causal=self.causal, mlp_upfactor=self.mlp_upfactor, use_softmax_for_attn=self.use_softmax_for_attn)

ARGS = {
    'block_size': 101,
    'n_layer': 12,
    'n_head': 8,
    'n_embd': 256,
    'dropout': 0.0,
    'bias': True,
    'use_mlps': True,
    'seq_op': 'attn',
    'mlp_activ': 'gelu',
    'use_resid': True,
    'use_seqop_ln': True,
    'use_final_ln': True,
    'use_mlp_ln': True,
    'conv_type': 'short',
    'causal': True,
    'mlp_upfactor': 4,
    'use_softmax_for_attn': True,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type GPT2Config — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a GPT2Config output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
MLP Ran
ran on a synthesized input · constructed an object; no result was computed
HazyResearch/precision-ls/src/models/gpt2.py
code served (permissive licence) · get_code("813516a2674fe31a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, MLP: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

class Config:
    def __init__(self):
        self.n_embd = 4
        self.mlp_upfactor = 3
        self.bias = True
        self.mlp_activ = 'gelu'
        self.dropout = 0.1

config = Config()

ARGS = {
    'config': config
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type MLP — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a MLP output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
BaseConv Not yet run
unverified (harvested, no recorded run)
HazyResearch/precision-ls/src/models/gpt2.py
code served (permissive licence) · get_code("6bf621cd35e95f81")
How it ran

no invocation recorded.

BaseConvLayer Not yet run
unverified (harvested, no recorded run)
HazyResearch/precision-ls/src/models/gpt2.py
code served (permissive licence) · get_code("d2f4bda71035892b")
How it ran

no invocation recorded.

Block Not yet run
unverified (harvested, no recorded run)
HazyResearch/precision-ls/src/models/gpt2.py
code served (permissive licence) · get_code("a0745a160f92b244")
How it ran

no invocation recorded.

GPT2Model Not yet run
unverified (harvested, no recorded run)
HazyResearch/precision-ls/src/models/gpt2.py
code served (permissive licence) · get_code("d64f15e110eb25fe")
How it ran

no invocation recorded.

TransformerModel Not yet run
unverified (harvested, no recorded run)
HazyResearch/precision-ls/src/models/gpt2.py
code served (permissive licence) · get_code("46aa5e22dbe3a170")
How it ran

no invocation recorded.

set_param Not yet run
unverified (harvested, no recorded run)
HazyResearch/precision-ls/src/models/gpt2.py
code served (permissive licence) · get_code("364a8c5607cc6f21")
How it ran

no invocation recorded.

Also present in other papers' code

Of 10 samples lifted from this paper's linked repositories, 10 carry a function identity and 0 do not. Of the 10 distinct functions with an identity, 0 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 0 later) and 10 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

This paper investigates whether sequence models can learn to perform numerical algorithms, e.g. gradient descent, on the fundamental problem of least squares. Our goal is to inherit two properties of standard algorithms from numerical analysis: (1) machine precision, i.e. we want to obtain solutions that are accurate to near floating point error, and (2) numerical generality, i.e. we want them to apply broadly across problem instances. We find that prior approaches using Transformers fail to meet these criteria, and identify limitations present in existing architectures and training procedures. First, we show that softmax Transformers struggle to perform highprecision multiplications, which prevents them from precisely learning numerical algorithms. Second, we identify an alternate class of architectures, comprised entirely of polynomials, that can efficiently represent high-precision gradient descent iterates. Finally, we investigate precision bottlenecks during training and address them via a high-precision training recipe that reduces stochastic gradient noise. Our recipe enables us to train two polynomial architectures, gated convolutions and linear attention, to perform gradient descent iterates on least squares problems. For the first time, we demonstrate the ability to train to near machine precision. Applied iteratively, our models obtain 100, 000× lower MSE than standard Transformers trained end-to-end and they incur a 10, 000× smaller generalization gap on out-of-distribution problems. We make progress towards end-to-end learning of numerical algorithms for least squares.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2503.12295")
get_code_for_paper("2503.12295")
have("2503.12295")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2503.12295.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2503.12295.svg)](https://syntology.ai/paper/2503.12295)

Connect an agent — have() is free.