SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2105.08050 · 2021

Pay Attention to MLPs

arXiv · PDF · Open in the Atlas

Code that ran

34 of 44 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (15 of them constructed an object rather than computing a result; 1 honoured the contract we drafted, 4 violated it and 26 have no contract verdict; 3 failures of our instrument; 10 no recorded run), measured 2026-11-21; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2105.08050.json · badge: /api/ran/2105.08050.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
lucidrains/g-mlp-pytorch reimplementation (classifier label)
classifier basis: classifier_2025
7 ran · 1 no recorded run
lucidrains/g-mlp-gpt reimplementation (classifier label)
classifier basis: classifier_2025
6 ran · 1 no recorded run
xmu-xiaoma666/External-Attention-pytorch unrelated (classifier label)
classifier basis: classifier_2025
4 ran · 0 no recorded run
leaderj1001/Bag-of-MLP application (classifier label)
classifier basis: classifier_2025
4 ran · 0 no recorded run
antonyvigouret/Pay-Attention-to-MLPs reimplementation (classifier label)
classifier basis: classifier_2025
3 ran · 0 no recorded run
jaketae/g-mlp reimplementation (classifier label)
classifier basis: classifier_2025
3 ran · 0 no recorded run
copy not recorded community
not in the paper's declared repositories
3 ran · 0 no recorded run
NydiaAI/g-mlp-tensorflow port (classifier label)
classifier basis: classifier_2025
2 ran · 4 no recorded run
cvolcy/gmlp-tf port (classifier label)
classifier basis: classifier_2025
2 ran · 0 no recorded run
lucidrains/mlp-gpt-jax reimplementation (classifier label)
classifier basis: classifier_2025
0 ran · 3 no recorded run
radi-cho/gatedtabtransformer application (classifier label)
classifier basis: classifier_2025
0 ran · 1 no recorded run
FunctionStatusWhere it lives
Attention Ran
ran on a synthesized input · constructed an object; no result was computed
lucidrains/g-mlp-pytorch/g_mlp_pytorch/g_mlp_pytorch.py
code served (permissive licence) · get_code("444b2d5ae8422156")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, Attention: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import nn

torch.manual_seed(0)

# Define ARGS with all parameters for Attention.__init__
ARGS = {
    'dim_in': 8,
    'dim_out': 6,
    'dim_inner': 4,
    'causal': False,
}

# Create an instance to verify it works
attention = nn.Attention(**ARGS) if hasattr(nn, 'Attention') else None

# Create a small input tensor for testing
# Shape: (batch=2, sequence_length=3, dim_in=8)
x = torch.randn(2, 3, ARGS['dim_in'])

# If you want to actually run forward pass:
# attention_module = Attention(**ARGS)
# output = attention_module(x)
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Attention — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a Attention output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
Attention Ran
ran on a synthesized input
lucidrains/g-mlp-gpt/g_mlp_gpt/g_mlp_gpt.py
code served (permissive licence) · get_code("400cf93bc9a76062")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 3, dtype family float, shape (2, 3, 5) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import nn, einsum

class Attention(nn.Module):

    def __init__(self, dim_in, dim_out, dim_inner):
        super().__init__()
        self.scale = dim_inner ** (-0.5)
        self.to_qkv = nn.Linear(dim_in, dim_inner * 3, bias=False)
        self.to_out = nn.Linear(dim_inner, dim_out)

    def forward(self, x):
        device = x.device
        q, k, v = self.to_qkv(x).chunk(3, dim=-1)
        sim = einsum('b i d, b j d -> b i j', q, k) * self.scale
        mask = torch.ones(sim.shape[-2:], device=device).triu(1).bool()
        sim.masked_fill_(mask[None, ...], -torch.finfo(q.dtype).max)
        attn = sim.softmax(dim=-1)
        out = einsum('b i j, b j d -> b i d', attn, v)
        return self.to_out(out)


INIT_ARGS = {
    'dim_in': 4,
    'dim_out': 5,
    'dim_inner': 6,
}

ARGS = {
    'x': torch.randn(2, 3, 4),
}

if __name__ == '__main__':
    attention = Attention(**INIT_ARGS)
    output = attention(**ARGS)
    print(f"Output shape: {output.shape}")
    print(f"Expected shape: (2, 3, 5)")
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float, shape (2, 3, 5) — shape (2, 3, 5) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
CausalLocalSGU Ran
ran on a synthesized input
lucidrains/g-mlp-gpt/g_mlp_gpt/g_mlp_gpt.py
code served (permissive licence) · get_code("3bb856991018170a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 3, dtype family float, shape (2, 64, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from math import ceil

# Import the class
from math import ceil
import torch
import torch.nn.functional as F
from torch import nn, einsum
from einops import rearrange, repeat

def exists(val):
    return val is not None

def pad_to_multiple(tensor, multiple, dim=-1, value=0):
    seqlen = tensor.shape[dim]
    m = seqlen / multiple
    if m.is_integer():
        return tensor
    remainder = ceil(m) * multiple - seqlen
    pad_offset = (0,) * (-1 - dim) * 2
    return F.pad(tensor, (*pad_offset, 0, remainder), value=value)

class CausalLocalSGU(nn.Module):

    def __init__(self, dim, dim_seq, init_eps=0.001, heads=4, window=128, act=nn.Identity()):
        super().__init__()
        dim_out = dim // 2
        self.norm = nn.LayerNorm(dim_out)
        self.heads = heads
        self.window = window
        self.weight = nn.Parameter(torch.zeros(heads, window, window * 2))
        self.bias = nn.Parameter(torch.zeros(heads, window))
        init_eps /= window
        nn.init.uniform_(self.weight, -init_eps, init_eps)
        nn.init.constant_(self.bias, 1.0)
        self.act = act
        self.register_buffer('mask', ~torch.ones(window, window * 2).triu_(window + 1).bool())

    def forward(self, x, gate_res=None):
        device, n, h, w = (x.device, x.shape[1], self.heads, self.window)
        res, gate = x.chunk(2, dim=-1)
        gate = pad_to_multiple(gate, w, dim=-2)
        gate = rearrange(gate, 'b (w n) d -> b w n d', n=w)
        gate = self.norm(gate)
        gate = F.pad(gate, (0, 0, 0, 0, 1, 0), value=0.0)
        gate = torch.cat((gate[:, :-1], gate[:, 1:]), dim=2)
        weight, bias = (self.weight, self.bias)
        weight = weight * self.mask[None, ...].int().float()
        gate = rearrange(gate, 'b w n (h d) -> b w h n d', h=h)
        gate = einsum('b w h n d, h m n -> b w h m d', gate, weight)
        gate = gate + rearrange(bias, 'h n -> () () h n ()')
        gate = rearrange(gate, 'b w h n d -> b w n (h d)')
        gate = rearrange(gate, 'b w n d -> b (w n) d')
        gate = gate[:, :n]
        if exists(gate_res):
            gate = gate + gate_res
        return self.act(gate) * res

INIT_ARGS = {
    'dim': 8,
    'dim_seq': 256,
    'init_eps': 0.001,
    'heads': 2,
    'window': 32,
    'act': nn.Identity()
}

ARGS = {
    'x': torch.randn(2, 64, 8),
    'gate_res': None
}

if __name__ == '__main__':
    model = CausalLocalSGU(**INIT_ARGS)
    output = model(**ARGS)
    print(f"Output shape: {output.shape}")
    print(f"Expected shape: {ARGS['x'].shape}")
    print("Success!")
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
einops, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float, shape (2, 64, 4) — shape (2, 64, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
CausalSGU Ran
ran on a synthesized input
lucidrains/g-mlp-gpt/g_mlp_gpt/g_mlp_gpt.py
code served (permissive licence) · get_code("7a6075afb6e7a2c2")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 3, dtype family float, shape (2, 3, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import nn, einsum
from einops import rearrange, repeat

def exists(val):
    return val is not None

class CausalSGU(nn.Module):

    def __init__(self, dim, dim_seq, init_eps=0.001, heads=4, act=nn.Identity()):
        super().__init__()
        dim_out = dim // 2
        self.norm = nn.LayerNorm(dim_out)
        self.heads = heads
        self.weight = nn.Parameter(torch.zeros(heads, dim_seq, dim_seq))
        self.bias = nn.Parameter(torch.zeros(heads, dim_seq))
        init_eps /= dim_seq
        nn.init.uniform_(self.weight, -init_eps, init_eps)
        nn.init.constant_(self.bias, 1.0)
        self.act = act
        self.register_buffer('mask', ~torch.ones(dim_seq, dim_seq).triu_(1).bool())

    def forward(self, x, gate_res=None):
        device, n, h = (x.device, x.shape[1], self.heads)
        res, gate = x.chunk(2, dim=-1)
        gate = self.norm(gate)
        weight, bias = (self.weight, self.bias)
        weight, bias = (weight[:, :n, :n], bias[:, :n])
        weight = weight * self.mask[None, :n, :n].int().float()
        gate = rearrange(gate, 'b n (h d) -> b h n d', h=h)
        gate = einsum('b h n d, h m n -> b h m d', gate, weight)
        gate = gate + rearrange(bias, 'h n -> () h n ()')
        gate = rearrange(gate, 'b h n d -> b n (h d)')
        if exists(gate_res):
            gate = gate + gate_res
        return self.act(gate) * res

INIT_ARGS = {
    'dim': 8,
    'dim_seq': 3,
    'init_eps': 0.001,
    'heads': 2,
    'act': nn.Identity()
}

ARGS = {
    'x': torch.randn(2, 3, 8),
    'gate_res': None
}

if __name__ == '__main__':
    model = CausalSGU(**INIT_ARGS)
    output = model(**ARGS)
    print(f"Output shape: {output.shape}")
    print(f"Output: {output}")
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
einops, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float, shape (2, 3, 4) — shape (2, 3, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
GELU Ran
ran on a synthesized input
NydiaAI/g-mlp-tensorflow/gmlp/gmlp.py
pointer only (licence: NONE) · get_code("36c054162a22371c")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, NydiaAI/g-mlp-tensorflow when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

GatingMlpBlock Ran
ran on a synthesized input · constructed an object; no result was computed
antonyvigouret/Pay-Attention-to-MLPs/models.py
code served (permissive licence) · get_code("cb304156fe457ec9")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, GatingMlpBlock: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Parameters for GatingMlpBlock
d_model = 8
d_ffn = 16
seq_len = 4
survival_prob = 0.8

ARGS = {
    "d_model": d_model,
    "d_ffn": d_ffn,
    "seq_len": seq_len,
    "survival_prob": survival_prob
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type GatingMlpBlock — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a GatingMlpBlock output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
LocalAttention Ran
ran on a synthesized input
lucidrains/g-mlp-gpt/g_mlp_gpt/g_mlp_gpt.py
code served (permissive licence) · get_code("94e83172b0355df5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 3, dtype family float, shape (2, 12, 6) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from math import ceil

INIT_ARGS = {
    'dim_in': 4,
    'dim_inner': 8,
    'dim_out': 6,
    'window': 4,
}

ARGS = {
    'x': torch.randn(2, 12, 4),
}

if __name__ == '__main__':
    from math import ceil
    import torch
    import torch.nn.functional as F
    from torch import nn, einsum
    from einops import rearrange, repeat

    def pad_to_multiple(tensor, multiple, dim=-1, value=0):
        seqlen = tensor.shape[dim]
        m = seqlen / multiple
        if m.is_integer():
            return tensor
        remainder = ceil(m) * multiple - seqlen
        pad_offset = (0,) * (-1 - dim) * 2
        return F.pad(tensor, (*pad_offset, 0, remainder), value=value)

    class LocalAttention(nn.Module):

        def __init__(self, dim_in, dim_inner, dim_out, window=128):
            super().__init__()
            self.scale = dim_inner ** (-0.5)
            self.window = window
            self.to_qkv = nn.Linear(dim_in, dim_inner * 3, bias=False)
            self.to_out = nn.Linear(dim_inner, dim_out)

        def forward(self, x):
            b, n, *_, device, w = (*x.shape, x.device, self.window)
            x = pad_to_multiple(x, w, dim=-2, value=0.0)
            q, k, v = self.to_qkv(x).chunk(3, dim=-1)
            window_fn = lambda t: rearrange(t, 'b (w n) d -> b w n d', n=w)
            q, k, v = map(window_fn, (q, k, v))
            k, v = map(lambda t: F.pad(t, (0, 0, 0, 0, 1, 0)), (k, v))
            k, v = map(lambda t: torch.cat((k[:, :-1], k[:, 1:]), dim=2), (k, v))
            sim = einsum('b w i d, b w j d -> b w i j', q, k) * self.scale
            buckets, i, j = sim.shape[-3:]
            mask_value = -torch.finfo(sim.dtype).max
            mask = torch.ones(i, j, device=device).triu_(j - i + 1).bool()
            mask = repeat(mask, 'i j -> () u i j', u=buckets)
            sim.masked_fill_(mask, mask_value)
            attn = sim.softmax(dim=-1)
            out = einsum('b w i j, b w j d -> b w i d', attn, v)
            out = rearrange(out, 'b w n d -> b (w n) d')
            out = self.to_out(out[:, :n])
            return out

    model = LocalAttention(**INIT_ARGS)
    output = model(**ARGS)
    print(f"Output shape: {output.shape}")
    print(f"Output: {output}")
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
einops, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float, shape (2, 12, 6) — shape (2, 12, 6) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
PreShiftTokens Ran
ran on a synthesized input · constructed an object; no result was computed
lucidrains/g-mlp-pytorch/g_mlp_pytorch/g_mlp_pytorch.py
code served (permissive licence) · get_code("c78c09b8919ac748")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, PreShiftTokens: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Create a minimal fn that accepts x and **kwargs
# The forward method just needs to accept x and return something
class DummyModule(nn.Module):
    def forward(self, x, **kwargs):
        return x

# Create the dummy function module
fn = DummyModule()

# shifts: tuple of shift amounts for each segment
shifts = (1, -1, 0)

# ARGS dict with all parameters
ARGS = {
    "shifts": shifts,
    "fn": fn
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type PreShiftTokens — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a PreShiftTokens output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
Residual Ran
ran on a synthesized input · constructed an object; no result was computed
xmu-xiaoma666/External-Attention-pytorch/model/mlp/g_mlp.py
code served (permissive licence) · get_code("9cceceb3671335d7")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, Residual: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# The Residual module wraps a function fn and adds a residual connection.
# It calls self.fn(x) and adds x to it, so fn must be a module that:
# - takes input x (a tensor)
# - returns output of same shape as x
# The forward pass does: return self.fn(x) + x

# Create a simple function module that preserves shape
class IdentityModule(nn.Module):
    def forward(self, x):
        return x

# Create the fn argument - a simple module
fn = IdentityModule()

ARGS = {
    "fn": fn
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Residual — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a Residual output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
Residual Ran
ran on a synthesized input · constructed an object; no result was computed
lucidrains/g-mlp-pytorch/g_mlp_pytorch/g_mlp_pytorch.py
code served (permissive licence) · get_code("4b858823b3eccdd7")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, Residual: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import nn

torch.manual_seed(0)

# The Residual class wraps a function/module and adds a residual connection
# It calls self.fn(x) and adds x to the result
# So fn must be a callable (nn.Module) that takes x and returns something
# with the same shape as x

class SimpleLinear(nn.Module):
    """A simple linear module that preserves input shape"""
    def __init__(self, dim):
        super().__init__()
        self.linear = nn.Linear(dim, dim)
    
    def forward(self, x):
        return self.linear(x)

# Create a small function/module to wrap
fn = SimpleLinear(dim=4)

ARGS = {
    'fn': fn
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Residual — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a Residual output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
Residual Ran
ran on a synthesized input
NydiaAI/g-mlp-tensorflow/gmlp/gmlp.py
pointer only (licence: NONE) · get_code("f9f191324f0a26fb")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, NydiaAI/g-mlp-tensorflow when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

SpatialGatingUnit Ran
ran on a synthesized input · constructed an object; no result was computed
xmu-xiaoma666/External-Attention-pytorch/model/mlp/g_mlp.py
code served (permissive licence) · get_code("9c81afbca1a2169c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, SpatialGatingUnit: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import nn

torch.manual_seed(0)

ARGS = {
    "dim": 8,
    "len_sen": 5
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type SpatialGatingUnit — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a SpatialGatingUnit output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
SpatialGatingUnit Ran
ran on a synthesized input
antonyvigouret/Pay-Attention-to-MLPs/models.py
code served (permissive licence) · get_code("1356a744f3730d9c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 3, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

INIT_ARGS = {
    'd_ffn': 8,
    'seq_len': 4,
}

ARGS = {
    'x': torch.randn(2, 4, 8),
}

if __name__ == '__main__':
    model = nn.Module()
    model.__class__ = type('SpatialGatingUnit', (nn.Module,), {
        '__init__': lambda self, d_ffn, seq_len: (
            super(type(self), self).__init__(),
            setattr(self, 'norm', nn.LayerNorm(d_ffn // 2)),
            setattr(self, 'proj', nn.Linear(seq_len, seq_len))
        ),
        'forward': lambda self, x: (
            lambda u, v: u * (
                self.proj(
                    self.norm(v).permute(0, 2, 1)
                ).permute(0, 2, 1)
            )
        )(*x.chunk(2, dim=-1))
    })
    
    model = type('SpatialGatingUnit', (nn.Module,), {
        '__init__': lambda self, d_ffn, seq_len: (
            super(type(self), self).__init__(),
            setattr(self, 'norm', nn.LayerNorm(d_ffn // 2)),
            setattr(self, 'proj', nn.Linear(seq_len, seq_len))
        ),
        'forward': lambda self, x: (
            lambda u, v: u * (
                self.proj(
                    self.norm(v).permute(0, 2, 1)
                ).permute(0, 2, 1)
            )
        )(*x.chunk(2, dim=-1))
    })(**INIT_ARGS)
    
    output = model(**ARGS)
    print(f"Output shape: {output.shape}")
    print(f"Output: {output}")
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float — no output shape is stated: observe_signature.py observed this class with its constructor sizes canonicalised where it could, which the node does not record, so its shape is not stated as this setup block's
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
SpatialGatingUnit Ran
ran on a synthesized input · constructed an object; no result was computed
jaketae/g-mlp/g_mlp/core.py
code served (permissive licence) · get_code("d300d8fa28344b6c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, SpatialGatingUnit: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# The function creates a SpatialGatingUnit which expects:
# - d_ffn: the feature dimension (used in LayerNorm and Conv1d)
# - seq_len: the sequence length (used in Conv1d)
# 
# In forward(), it expects input x with shape (batch, seq_len, d_ffn)
# which gets chunked into u and v along dim=-1

d_ffn = 8  # feature dimension
seq_len = 5  # sequence length

ARGS = {
    "d_ffn": d_ffn,
    "seq_len": seq_len,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type SpatialGatingUnit — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a SpatialGatingUnit output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
SpatialGatingUnit Ran
ran on a synthesized input · constructed an object; no result was computed
cvolcy/gmlp-tf/gmlp.py
pointer only (licence: NONE) · get_code("2098e31f03598b8e")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, cvolcy/gmlp-tf when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

SpatialGatingUnit Ran
ran on a synthesized input · constructed an object; no result was computed
lucidrains/g-mlp-pytorch/g_mlp_pytorch/g_mlp_pytorch.py
code served (permissive licence) · get_code("6d43444c56cad017")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, SpatialGatingUnit: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Define ARGS dict with all parameters for SpatialGatingUnit
ARGS = {
    'dim': 8,
    'dim_seq': 4,
    'causal': False,
    'act': nn.GELU(),
    'heads': 2,
    'init_eps': 0.001,
    'circulant_matrix': False,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
einops, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type SpatialGatingUnit — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a SpatialGatingUnit output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
SpatialGatingUnit Ran
ran on a synthesized input
leaderj1001/Bag-of-MLP/model.py
code served (permissive licence) · get_code("f84eabb818f0b01a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 3, dtype family float, shape (2, 3, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

class TinyAttention(nn.Module):

    def __init__(self, d_ffn, d_attn=64):
        super(TinyAttention, self).__init__()
        self.qkv = nn.Linear(d_ffn, 3 * d_attn)
        self.d_attn = d_attn
        self.softmax = nn.Softmax(dim=-1)
        self.out = nn.Linear(d_attn, d_ffn // 2)

    def forward(self, x):
        qkv = self.qkv(x)
        q, k, v = torch.chunk(qkv, 3, dim=-1)
        energy = torch.bmm(q, k.transpose(1, 2)) / self.d_attn ** 0.5
        attn = self.softmax(energy)
        out = torch.bmm(attn, v)
        out = self.out(out)
        return out

class SpatialGatingUnit(nn.Module):

    def __init__(self, seq_len, d_model, attn=True):
        super(SpatialGatingUnit, self).__init__()
        self.layer_norm = nn.LayerNorm(d_model // 2)
        self.spatial = nn.Conv1d(seq_len, seq_len, kernel_size=1)
        self.is_attn = attn
        if self.is_attn:
            self.attn = TinyAttention(d_model)

    def forward(self, x):
        if self.is_attn:
            shortcut = x
            shortcut = self.attn(shortcut)
        u, v = torch.chunk(x, 2, dim=-1)
        v = self.layer_norm(v)
        v = self.spatial(v)
        if self.is_attn:
            v += shortcut
        return u * v

INIT_ARGS = {
    'seq_len': 3,
    'd_model': 8,
    'attn': True
}

ARGS = {
    'x': torch.randn(2, 3, 8)
}
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float, shape (2, 3, 4) — shape (2, 3, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
TinyAttention Ran
ran on a synthesized input
leaderj1001/Bag-of-MLP/model.py
code served (permissive licence) · get_code("ccc79f7e78680ff2")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 3, dtype family float, shape (2, 3, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

class TinyAttention(nn.Module):

    def __init__(self, d_ffn, d_attn=64):
        super(TinyAttention, self).__init__()
        self.qkv = nn.Linear(d_ffn, 3 * d_attn)
        self.d_attn = d_attn
        self.softmax = nn.Softmax(dim=-1)
        self.out = nn.Linear(d_attn, d_ffn // 2)

    def forward(self, x):
        qkv = self.qkv(x)
        q, k, v = torch.chunk(qkv, 3, dim=-1)
        energy = torch.bmm(q, k.transpose(1, 2)) / self.d_attn ** 0.5
        attn = self.softmax(energy)
        out = torch.bmm(attn, v)
        out = self.out(out)
        return out


INIT_ARGS = {
    'd_ffn': 8,
    'd_attn': 64,
}

ARGS = {
    'x': torch.randn(2, 3, 8),
}

if __name__ == '__main__':
    model = TinyAttention(**INIT_ARGS)
    output = model(**ARGS)
    print(f"Output shape: {output.shape}")
    print(f"Output: {output}")
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float, shape (2, 3, 4) — shape (2, 3, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
cast_tuple Ran
ran, violated the contract we drafted
lucidrains/g-mlp-gpt/g_mlp_gpt/g_mlp_gpt.py
code served (permissive licence) · get_code("092c152ffc42d497")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned tuple, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Analyzing the function:
# def cast_tuple(val, num):
#     return (val,) * num if not isinstance(val, tuple) else val
#
# Parameters:
# - val: can be any value or a tuple
# - num: an integer used for tuple repetition

ARGS = {
    "val": 42,
    "num": 3
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family int — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
check_sizes Ran
ran, honoured the contract we drafted
this paper's copy was not recorded; identical code first harvested from jaketae/res-mlp
pointer only · get_code("27cbea18b00696e9")
How it ran

The body of this sample is withheld: this paper's copy of the code was not recorded (its link names no repository or source path), so no licence was read for it. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

dropout_layers Ran
ran; our contract draft was wrong, not the code
lucidrains/g-mlp-pytorch/g_mlp_pytorch/g_mlp_pytorch.py
code served (permissive licence) · get_code("1d39b4f9d1d36f92")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from random import randrange

torch.manual_seed(0)

# Create a list of simple layers (nn.Module instances)
layers = [
    nn.Linear(4, 3),
    nn.ReLU(),
    nn.Linear(3, 2),
    nn.Dropout(0.5)
]

# Define probability of survival
prob_survival = 0.7

ARGS = {
    "layers": layers,
    "prob_survival": prob_survival
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
exist Ran
ran, violated the contract we drafted
xmu-xiaoma666/External-Attention-pytorch/model/mlp/g_mlp.py
code served (permissive licence) · get_code("62c69ec0b60162c6")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

ARGS = {
    "x": None
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
exists Ran
ran, violated the contract we drafted
this paper's copy was not recorded; identical code first harvested from ThomasMrY/VCT
pointer only; the stored copy get_code serves, from ThomasMrY/VCT, is pointer only (licence: NONE) · get_code("aa5486a3650902d8")
How it ran

The body of this sample is withheld: this paper's copy of the code was not recorded (its link names no repository or source path), so no licence was read for it. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

gMLP Ran
ran on a synthesized input · constructed an object; no result was computed
xmu-xiaoma666/External-Attention-pytorch/model/mlp/g_mlp.py
code served (permissive licence) · get_code("b9fa9cafe23e12fb")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, gMLP: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import nn
from collections import OrderedDict

torch.manual_seed(0)

# Define ARGS dict with all parameters for gMLP
ARGS = {
    'num_tokens': 100,
    'len_sen': 49,
    'dim': 8,
    'd_ff': 16,
    'num_layers': 2
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type gMLP — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a gMLP output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
gMLP Ran
ran on a synthesized input · constructed an object; no result was computed
antonyvigouret/Pay-Attention-to-MLPs/models.py
code served (permissive licence) · get_code("5a39fbe0ee0a5529")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, gMLP: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Parameters for gMLP
d_model = 4
d_ffn = 8
seq_len = 3
n_blocks = 2
prob_0_L = [1.0, 0.5]

ARGS = {
    "d_model": d_model,
    "d_ffn": d_ffn,
    "seq_len": seq_len,
    "n_blocks": n_blocks,
    "prob_0_L": prob_0_L,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type gMLP — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a gMLP output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
gMLP Ran
ran on a synthesized input · constructed an object; no result was computed
jaketae/g-mlp/g_mlp/core.py
code served (permissive licence) · get_code("2949429d580ede4b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, gMLP: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

ARGS = {
    "d_model": 4,
    "d_ffn": 8,
    "seq_len": 3,
    "num_layers": 2,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type gMLP — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a gMLP output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
gMLP Ran
ran on a synthesized input
leaderj1001/Bag-of-MLP/model.py
code served (permissive licence) · get_code("bfbcb5bb0523c7c8")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 2, dtype family float, shape (2, 5) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import torch.nn.functional as F

class TinyAttention(nn.Module):

    def __init__(self, d_ffn, d_attn=64):
        super(TinyAttention, self).__init__()
        self.qkv = nn.Linear(d_ffn, 3 * d_attn)
        self.d_attn = d_attn
        self.softmax = nn.Softmax(dim=-1)
        self.out = nn.Linear(d_attn, d_ffn // 2)

    def forward(self, x):
        qkv = self.qkv(x)
        q, k, v = torch.chunk(qkv, 3, dim=-1)
        energy = torch.bmm(q, k.transpose(1, 2)) / self.d_attn ** 0.5
        attn = self.softmax(energy)
        out = torch.bmm(attn, v)
        out = self.out(out)
        return out

class SpatialGatingUnit(nn.Module):

    def __init__(self, seq_len, d_model, attn=True):
        super(SpatialGatingUnit, self).__init__()
        self.layer_norm = nn.LayerNorm(d_model // 2)
        self.spatial = nn.Conv1d(seq_len, seq_len, kernel_size=1)
        self.is_attn = attn
        if self.is_attn:
            self.attn = TinyAttention(d_model)

    def forward(self, x):
        if self.is_attn:
            shortcut = x
            shortcut = self.attn(shortcut)
        u, v = torch.chunk(x, 2, dim=-1)
        v = self.layer_norm(v)
        v = self.spatial(v)
        if self.is_attn:
            v += shortcut
        return u * v

class gMLPBlock(nn.Module):

    def __init__(self, seq_len, d_model, d_ffn):
        super(gMLPBlock, self).__init__()
        self.layer_norm = nn.LayerNorm(d_model)
        self.channel1 = nn.Linear(d_model, d_ffn)
        self.sgu = SpatialGatingUnit(seq_len, d_ffn)
        self.channel2 = nn.Linear(d_ffn // 2, d_model)

    def forward(self, x):
        shortcut = x
        out = self.layer_norm(x)
        out = self.channel1(out)
        out = F.gelu(out)
        out = self.sgu(out)
        out = self.channel2(out)
        return out + shortcut

class gMLP(nn.Module):

    def __init__(self, seq_len, d_model, d_ffn, patch=32, N=12, n_classes=10):
        super(gMLP, self).__init__()
        self.d_model = d_model
        self.embedding = nn.Conv2d(3, d_model, kernel_size=patch, stride=patch)
        self.layers = nn.ModuleList([gMLPBlock(seq_len, d_model, d_ffn) for _ in range(N)])
        self.out = nn.Linear(d_model, n_classes)

    def forward(self, x):
        out = self.embedding(x).permute(0, 2, 3, 1).view(x.size(0), -1, self.d_model)
        for layer in self.layers:
            out = layer(out)
        out = out.mean(1)
        out = self.out(out)
        return out

INIT_ARGS = {
    'seq_len': 4,
    'd_model': 8,
    'd_ffn': 16,
    'patch': 32,
    'N': 2,
    'n_classes': 5,
}

ARGS = {
    'x': torch.randn(2, 3, 64, 64),
}
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 5) — shape (2, 5) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
gMLPBlock Ran
ran on a synthesized input · constructed an object; no result was computed
jaketae/g-mlp/g_mlp/core.py
code served (permissive licence) · get_code("3f7743323edaa181")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, gMLPBlock: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Parameters for gMLPBlock
d_model = 4      # Model dimension
d_ffn = 8        # Feed-forward network dimension (must be even for chunk(2))
seq_len = 3      # Sequence length

ARGS = {
    'd_model': d_model,
    'd_ffn': d_ffn,
    'seq_len': seq_len
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type gMLPBlock — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a gMLPBlock output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
gMLPBlock Ran
ran on a synthesized input · constructed an object; no result was computed
lucidrains/g-mlp-pytorch/g_mlp_pytorch/g_mlp_pytorch.py
code served (permissive licence) · get_code("63d8448fca0c6967")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, gMLPBlock: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

ARGS = {
    'dim': 4,
    'dim_ff': 8,
    'seq_len': 3,
    'heads': 1,
    'attn_dim': None,
    'causal': False,
    'act': nn.Identity(),
    'circulant_matrix': False,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
einops, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type gMLPBlock — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a gMLPBlock output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
gMLPBlock Ran
ran on a synthesized input
leaderj1001/Bag-of-MLP/model.py
code served (permissive licence) · get_code("af39a3c7ed4a3a0c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 3, dtype family float, shape (2, 5, 8) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import torch.nn.functional as F

class TinyAttention(nn.Module):

    def __init__(self, d_ffn, d_attn=64):
        super(TinyAttention, self).__init__()
        self.qkv = nn.Linear(d_ffn, 3 * d_attn)
        self.d_attn = d_attn
        self.softmax = nn.Softmax(dim=-1)
        self.out = nn.Linear(d_attn, d_ffn // 2)

    def forward(self, x):
        qkv = self.qkv(x)
        q, k, v = torch.chunk(qkv, 3, dim=-1)
        energy = torch.bmm(q, k.transpose(1, 2)) / self.d_attn ** 0.5
        attn = self.softmax(energy)
        out = torch.bmm(attn, v)
        out = self.out(out)
        return out

class SpatialGatingUnit(nn.Module):

    def __init__(self, seq_len, d_model, attn=True):
        super(SpatialGatingUnit, self).__init__()
        self.layer_norm = nn.LayerNorm(d_model // 2)
        self.spatial = nn.Conv1d(seq_len, seq_len, kernel_size=1)
        self.is_attn = attn
        if self.is_attn:
            self.attn = TinyAttention(d_model)

    def forward(self, x):
        if self.is_attn:
            shortcut = x
            shortcut = self.attn(shortcut)
        u, v = torch.chunk(x, 2, dim=-1)
        v = self.layer_norm(v)
        v = self.spatial(v)
        if self.is_attn:
            v += shortcut
        return u * v

class gMLPBlock(nn.Module):

    def __init__(self, seq_len, d_model, d_ffn):
        super(gMLPBlock, self).__init__()
        self.layer_norm = nn.LayerNorm(d_model)
        self.channel1 = nn.Linear(d_model, d_ffn)
        self.sgu = SpatialGatingUnit(seq_len, d_ffn)
        self.channel2 = nn.Linear(d_ffn // 2, d_model)

    def forward(self, x):
        shortcut = x
        out = self.layer_norm(x)
        out = self.channel1(out)
        out = F.gelu(out)
        out = self.sgu(out)
        out = self.channel2(out)
        return out + shortcut

INIT_ARGS = {
    'seq_len': 5,
    'd_model': 8,
    'd_ffn': 16
}

ARGS = {
    'x': torch.randn(2, 5, 8)
}
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float, shape (2, 5, 8) — shape (2, 5, 8) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
gMLPLayer Ran
ran on a synthesized input · constructed an object; no result was computed
cvolcy/gmlp-tf/gmlp.py
pointer only (licence: NONE) · get_code("683c46ded1adfd74")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, cvolcy/gmlp-tf when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

pad_to_multiple Ran
ran; our contract draft was wrong, not the code
lucidrains/g-mlp-gpt/g_mlp_gpt/g_mlp_gpt.py
code served (permissive licence) · get_code("df3be65159852cd8")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned Tensor, ndim 2, dtype family float, shape (2, 6) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn.functional as F
from math import ceil

torch.manual_seed(0)

# Create a small tensor with shape (2, 4) - batch size 2, sequence length 4
tensor = torch.randn(2, 4)

# multiple: the target multiple to pad to
multiple = 3

# dim: dimension to pad along (default -1 means last dimension)
dim = -1

# value: padding value (default 0)
value = 0

ARGS = {
    "tensor": tensor,
    "multiple": multiple,
    "dim": dim,
    "value": value,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 6) — shape (2, 6) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
pair Ran
ran, violated the contract we drafted
this paper's copy was not recorded; identical code first harvested from lucidrains/res-mlp-pytorch
pointer only · get_code("e6f9d3b5ad66ee20")
How it ran

The body of this sample is withheld: this paper's copy of the code was not recorded (its link names no repository or source path), so no licence was read for it. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

shift Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
lucidrains/g-mlp-pytorch/g_mlp_pytorch/g_mlp_pytorch.py
code served (permissive licence) · get_code("2fdf59e29c6d95cc")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn.functional as F

torch.manual_seed(0)

# Create a small tensor for the shift function
# The function uses F.pad which expects a tensor
# Shape: (batch=2, features=4)
t = torch.randn(2, 4)

# amount parameter: number of positions to shift
amount = 1

# mask parameter: optional, can be None or a tensor
mask = None

ARGS = {
    "t": t,
    "amount": amount,
    "mask": mask,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
Attention Not yet run
unverified (harvested, no recorded run)
lucidrains/mlp-gpt-jax/mlp_gpt_jax/mlp_gpt_jax.py
code served (permissive licence) · get_code("01675e7e7c713147")
How it ran

no invocation recorded.

PreNorm Not yet run
unverified (harvested, no recorded run)
NydiaAI/g-mlp-tensorflow/gmlp/gmlp.py
pointer only (licence: NONE) · get_code("2c295e59b874c818")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, NydiaAI/g-mlp-tensorflow when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

SGU Not yet run
unverified (harvested, no recorded run)
lucidrains/mlp-gpt-jax/mlp_gpt_jax/mlp_gpt_jax.py
code served (permissive licence) · get_code("1b9ff0230e5d08d3")
How it ran

no invocation recorded.

SpatialGatingUnit Not yet run
unverified (harvested, no recorded run)
NydiaAI/g-mlp-tensorflow/gmlp/gmlp.py
pointer only (licence: NONE) · get_code("51216a9ccabc7c3f")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, NydiaAI/g-mlp-tensorflow when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

gMLP Not yet run
unverified (harvested, no recorded run)
lucidrains/mlp-gpt-jax/mlp_gpt_jax/mlp_gpt_jax.py
code served (permissive licence) · get_code("c1658eda58b9fa37")
How it ran

no invocation recorded.

gMLP Not yet run
unverified (harvested, no recorded run)
lucidrains/g-mlp-pytorch/g_mlp_pytorch/g_mlp_pytorch.py
code served (permissive licence) · get_code("0330d8c0ae7e5def")
How it ran

no invocation recorded.

gMLP Not yet run
unverified (harvested, no recorded run)
radi-cho/gatedtabtransformer/gated_tab_transformer/g_mlp.py
code served (permissive licence) · get_code("2e5cc43cba18a48d")
How it ran

no invocation recorded.

gMLP Not yet run
unverified (harvested, no recorded run)
NydiaAI/g-mlp-tensorflow/gmlp/gmlp.py
pointer only (licence: NONE) · get_code("211dccfc0c5af2fa")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, NydiaAI/g-mlp-tensorflow when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

gMLPBlock Not yet run
unverified (harvested, no recorded run)
lucidrains/g-mlp-gpt/g_mlp_gpt/g_mlp_gpt.py
code served (permissive licence) · get_code("d16a844fcf910249")
How it ran

no invocation recorded.

gMLPBlock Not yet run
unverified (harvested, no recorded run)
NydiaAI/g-mlp-tensorflow/gmlp/gmlp.py
pointer only (licence: NONE) · get_code("a5c1690311b9aad2")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, NydiaAI/g-mlp-tensorflow when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

Also present in other papers' code

Of 44 samples lifted from this paper's linked repositories, 44 carry a function identity and 0 do not. Of the 43 distinct functions with an identity, 6 are also present in other papers whose code we hold (2 with at least one paper dated earlier, 3 the same month, 5 later) and 37 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
exists
brick of this paper's assembly: helper
486 other papers
43 dated earlier, 3 the same month, 438 later, 2 undated · 78 only as an assembly brick
1603.08155 (2016-03)
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
1611.07004 (2016-11)
Image-to-Image Translation with Conditional Adversarial Networks
1804.04637 (2018-04)
EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models
and 483 more not named here
Residual
brick of this paper's assembly: helper
4 other papers
1 dated earlier, 3 later · 2 only as an assembly brick
1512.03385 (2015-12)
Deep Residual Learning for Image Recognition
2106.11149 (2021-06)
OadTR: Online Action Detection with Transformers
2110.13214 (2021-10)
IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning
and 1 more not named here
pair 3 other papers
1 the same month, 2 later
2105.03404 (2021-05)
ResMLP: Feedforward networks for image classification with data-efficient training
2402.12192 (2024-02)
Pan-Mamba: Effective pan-sharpening with State Space Model
check_sizes 2 other papers
2 the same month
2105.01601 (2021-05)
MLP-Mixer: An all-MLP Architecture for Vision
2105.03404 (2021-05)
ResMLP: Feedforward networks for image classification with data-efficient training
dropout_layers
brick of this paper's assembly: helper
2 other papers
2 later
2110.07749 (2021-10)
Attention-Free Keyword Spotting
pad_to_multiple
brick of this paper's assembly: helper
1 other paper
1 later
2107.02192 (2021-07)
Long-Short Transformer: Efficient Transformers for Language and Vision

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-21.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Transformers have become one of the most important architectural innovations in deep learning and have enabled many breakthroughs over the past few years. Here we propose a simple network architecture, gMLP, based on MLPs with gating, and show that it can perform as well as Transformers in key language and vision applications. Our comparisons show that self-attention is not critical for Vision Transformers, as gMLP can achieve the same accuracy. For BERT, our model achieves parity with Transformers on pretraining perplexity and is better on some downstream NLP tasks. On finetuning tasks where gMLP performs worse, making the gMLP model substantially larger can close the gap with Transformers. In general, our experiments show that gMLP can scale as well as Transformers over increased data and compute.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2105.08050")
get_code_for_paper("2105.08050")
have("2105.08050")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2105.08050.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2105.08050.svg)](https://syntology.ai/paper/2105.08050)

Connect an agent — have() is free.