SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2308.00951 · ICLR · 2024

From Sparse to Soft Mixtures of Experts

Neil Houlsby, Basil Mustafa, Joan Puigcerver, Carlos Riquelme

arXiv · PDF · Open in the Atlas

Code that ran

29 of 33 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (5 of them constructed an object rather than computing a result; 3 honoured the contract we drafted, 4 violated it and 11 have no contract verdict; 11 failures of our instrument; 4 no recorded run); 1 of them was also run on the host, not in the jail, by the peer-agreement lane that set its level, measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2308.00951.json · badge: /api/ran/2308.00951.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
google-research/vmoe canonical (classifier label)
classifier basis: classifier_2025
3 ran · 0 no recorded run
lucidrains/soft-moe-pytorch reimplementation (classifier label)
classifier basis: classifier_2025
15 ran · 2 no recorded run
umbertocappellazzo/petl_ast community
Papers with Code archive
4 ran · 1 no recorded run
bwconrad/soft-moe reimplementation (classifier label)
classifier basis: classifier_2025
3 ran · 0 no recorded run
copy not recorded community
not in the paper's declared repositories
2 ran · 1 no recorded run
fkodom/soft-mixture-of-experts community
Papers with Code archive
2 ran · 0 no recorded run
FunctionStatusWhere it lives
softmax Ran
ran; our contract draft was wrong, not the code
bwconrad/soft-moe/soft_moe/soft_moe.py
code served (permissive licence) · get_code("348d3e3d68c0c2dc")
How it ran

Ran in Syntology's jail on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. The comparison behind this sample's level is a different run: the peer-agreement lane ran it again on the host, not in the jail (report dated 2026-09-04T23:13:21Z). If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Create a small tensor for testing
# Using shape (2, 4) - batch size 2, feature dimension 4
x = torch.randn(2, 4)

# Test with single dimension
dim_single = 1

# Test with multiple dimensions (tuple)
dim_multi = (0, 1)

# Create ARGS dict with both test cases
# We'll use the single dimension case as the primary one
ARGS = {
    "x": x,
    "dim": dim_single
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
AllGather Ran
ran on a synthesized input · constructed an object; no result was computed
lucidrains/soft-moe-pytorch/soft_moe_pytorch/soft_moe.py
code served (permissive licence) · get_code("f86d2cc1b619c3a7")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, AllGather: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import torch.distributed as dist
from unittest.mock import Mock, MagicMock

torch.manual_seed(0)

# Mock distributed environment since we're running on CPU without actual distributed setup
class MockDistributed:
    def get_rank(self):
        return 0
    
    def get_world_size(self):
        return 1
    
    def all_gather(self, tensor_list, tensor):
        # Simulate all_gather by filling the list with the tensor
        for i in range(len(tensor_list)):
            tensor_list[i].copy_(tensor)

# Patch dist module
original_get_rank = dist.get_rank
original_get_world_size = dist.get_world_size
original_all_gather = dist.all_gather

mock_dist = MockDistributed()
dist.get_rank = mock_dist.get_rank
dist.get_world_size = mock_dist.get_world_size
dist.all_gather = mock_dist.all_gather

# Create a small tensor for testing
x = torch.randn(2, 4)

# The AllGather module takes dim as a parameter
ARGS = {
    'dim': 0
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
einops, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type AllGather — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a AllGather output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
AllGatherFunction Ran
ran on a synthesized input · constructed an object; no result was computed
lucidrains/soft-moe-pytorch/soft_moe_pytorch/soft_moe.py
code served (permissive licence) · get_code("737c5706bfd26931")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, AllGatherFunction: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.distributed as dist
from unittest.mock import MagicMock, patch

torch.manual_seed(0)

# Mock distributed environment since we're running on CPU only
mock_rank = 0
mock_world_size = 2

def mock_get_rank():
    return mock_rank

def mock_get_world_size():
    return mock_world_size

def mock_all_gather(gathered_list, tensor):
    # Simulate all_gather by filling the list with copies of the tensor
    for i in range(len(gathered_list)):
        gathered_list[i].copy_(tensor)

# Patch distributed functions
with patch.object(dist, 'get_rank', mock_get_rank):
    with patch.object(dist, 'get_world_size', mock_get_world_size):
        with patch.object(dist, 'all_gather', mock_all_gather):
            # Create test tensors and parameters for AllGatherFunction.forward
            x = torch.randn(2, 4)  # batch_size=2, feature_dim=4
            dim = 0
            sizes = torch.tensor([2, 3], dtype=torch.long)  # different sizes for each rank

ARGS = {
    'x': torch.randn(2, 4),
    'dim': 0,
    'sizes': torch.tensor([2, 3], dtype=torch.long)
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
einops, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type AllGatherFunction — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a AllGatherFunction output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
Bottleneck_adapter Ran
ran on a synthesized input
umbertocappellazzo/petl_ast/src/MoA.py
pointer only (licence: NONE) · get_code("ebb62e0a151b812e")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, umbertocappellazzo/petl_ast when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

Convpass_adapter Ran
ran on a synthesized input
umbertocappellazzo/petl_ast/src/MoA.py
pointer only (licence: NONE) · get_code("9da4a23bc3ecddd2")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, umbertocappellazzo/petl_ast when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

Experts Ran
ran on a synthesized input · constructed an object; no result was computed
lucidrains/soft-moe-pytorch/soft_moe_pytorch/soft_moe.py
code served (permissive licence) · get_code("b1dce0b4095f9c1c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, Experts: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from unittest.mock import MagicMock, patch

torch.manual_seed(0)

# Create minimal expert modules
class SimpleExpert(nn.Module):
    def __init__(self, in_dim, out_dim):
        super().__init__()
        self.linear = nn.Linear(in_dim, out_dim)
    
    def forward(self, x):
        return self.linear(x)

# Create a list of expert modules
experts = nn.ModuleList([
    SimpleExpert(4, 4),
    SimpleExpert(4, 4),
    SimpleExpert(4, 4),
])

# Mock dist module to avoid requiring actual distributed setup
def mock_is_initialized():
    return False

def mock_get_world_size():
    return 1

def mock_get_rank():
    return 0

# Patch torch.distributed before importing/using Experts
with patch('torch.distributed.is_initialized', side_effect=mock_is_initialized):
    with patch('torch.distributed.get_world_size', side_effect=mock_get_world_size):
        with patch('torch.distributed.get_rank', side_effect=mock_get_rank):
            pass

ARGS = {
    'experts': experts,
    'is_distributed': False,
    'offload_unused_experts_to_cpu': True,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
einops, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Experts — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a Experts output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
FeedForward Ran
ran; our contract draft was wrong, not the code
lucidrains/soft-moe-pytorch/soft_moe_pytorch/soft_moe.py
code served (permissive licence) · get_code("14ed968c0a8fcc78")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Sequential. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import nn

torch.manual_seed(0)

ARGS = {
    "dim": 64,
    "mult": 4,
    "dropout": 0.1,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Sequential — a Sequential output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a Sequential output the recorded ndim 1 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
GLUFeedForward Ran
ran; our contract draft was wrong, not the code
lucidrains/soft-moe-pytorch/soft_moe_pytorch/soft_moe.py
code served (permissive licence) · get_code("49960f76a7235e60")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Sequential. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from torch.nn import Module
import torch.nn.functional as F

torch.manual_seed(0)

ARGS = {
    "dim": 64,
    "mult": 4,
    "dropout": 0.1
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Sequential — a Sequential output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a Sequential output the recorded ndim 1 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
LayerNorm Ran
ran on a synthesized input · constructed an object; no result was computed
lucidrains/soft-moe-pytorch/soft_moe_pytorch/soft_moe.py
code served (permissive licence) · get_code("58f8445dde42979a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, LayerNorm: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

ARGS = {
    'dim': 4
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type LayerNorm — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a LayerNorm output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
MultiExpertLayer Ran
ran on a synthesized input
fkodom/soft-mixture-of-experts/soft_mixture_of_experts/soft_moe.py
code served (permissive licence) · get_code("4767b6942d7a9c14")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 3, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import math
import torch
from torch import nn
from einops import einsum, rearrange

class MultiExpertLayer(nn.Module):
    """A more efficient alternative to creating 'n' separate expert layers (likely
    from 'nn.Linear' modules).  Instead, we create a single set of batched weights
    and biases, and apply all 'experts' in parallel.

    Args:
        embed_dim (int): embedding dimension (d)
        num_experts (int): number of experts (n)
        bias (bool): whether to include a bias term. Default: True
    """

    def __init__(self, in_features: int, out_features: int, num_experts: int, bias: bool=True, device=None, dtype=None):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.num_experts = num_experts
        self.weight = nn.Parameter(torch.empty((num_experts, in_features, out_features), device=device, dtype=dtype))
        bias_param = None
        if bias:
            bias_param = nn.Parameter(torch.empty((num_experts, out_features), device=device, dtype=dtype))
        self.bias = bias_param
        self.register_parameter('bias', bias_param)
        self.reset_parameters()

    def reset_parameters(self) -> None:
        nn.init.kaiming_uniform_(self.weight, a=math.sqrt(5))
        if self.bias is not None:
            fan_in, _ = nn.init._calculate_fan_in_and_fan_out(self.weight)
            bound = 1 / math.sqrt(fan_in) if fan_in > 0 else 0
            nn.init.uniform_(self.bias, -bound, bound)

    def forward(self, x):
        if x.size(-1) != self.in_features:
            raise ValueError(f'Expected input with embed_dim={self.in_features} (dim=-1), but found {x.size(-1)}')
        elif x.size(1) != self.num_experts:
            raise ValueError(f'Expected input with num_experts={self.num_experts} (dim=1), but found {x.size(1)}')
        x = einsum(x, self.weight, 'b n ... d1, n d1 d2 -> b n ... d2')
        if self.bias is not None:
            if x.ndim == 3:
                bias = rearrange(self.bias, 'n d -> () n d')
            elif x.ndim == 4:
                bias = rearrange(self.bias, 'n d -> () n () d')
            else:
                raise ValueError(f'Expected input to have 3 or 4 dimensions, but got {x.ndim}')
            x = x + bias
        return x

    def extra_repr(self) -> str:
        return f'in_features={self.in_features}, out_features={self.out_features}, num_experts={self.num_experts}, bias={self.bias is not None}'


INIT_ARGS = {
    'in_features': 4,
    'out_features': 3,
    'num_experts': 2,
    'bias': True,
    'device': None,
    'dtype': None,
}

ARGS = {
    'x': torch.randn(2, 2, 4),
}

if __name__ == '__main__':
    layer = MultiExpertLayer(**INIT_ARGS)
    output = layer(**ARGS)
    print(f'Output shape: {output.shape}')
    print(f'Output: {output}')
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
einops, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float — no output shape is stated: observe_signature.py observed this class with its constructor sizes canonicalised where it could, which the node does not record, so its shape is not stated as this setup block's
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
RMSNorm Ran
ran on a synthesized input · constructed an object; no result was computed
lucidrains/soft-moe-pytorch/soft_moe_pytorch/soft_moe.py
code served (permissive licence) · get_code("024a5f1d96cb0afb")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, RMSNorm: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import nn

torch.manual_seed(0)

ARGS = {
    "dim": 4
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type RMSNorm — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a RMSNorm output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
SoftMoE Ran
ran on a synthesized input
fkodom/soft-mixture-of-experts/soft_mixture_of_experts/soft_moe.py
code served (permissive licence) · get_code("b51fa28596d7a766")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 3, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import math
from typing import Optional, Union
from einops import einsum, rearrange

class MultiExpertLayer(nn.Module):
    def __init__(self, in_features: int, out_features: int, num_experts: int, bias: bool=True, device: Optional[Union[torch.device, str]]=None, dtype: Optional[torch.dtype]=None):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.num_experts = num_experts
        self.weight = nn.Parameter(torch.empty((num_experts, in_features, out_features), device=device, dtype=dtype))
        bias_param: Optional[nn.Parameter] = None
        if bias:
            bias_param = nn.Parameter(torch.empty((num_experts, out_features), device=device, dtype=dtype))
        self.bias: Optional[nn.Parameter]
        self.register_parameter('bias', bias_param)
        self.reset_parameters()

    def reset_parameters(self) -> None:
        nn.init.kaiming_uniform_(self.weight, a=math.sqrt(5))
        if self.bias is not None:
            fan_in, _ = nn.init._calculate_fan_in_and_fan_out(self.weight)
            bound = 1 / math.sqrt(fan_in) if fan_in > 0 else 0
            nn.init.uniform_(self.bias, -bound, bound)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        if x.size(-1) != self.in_features:
            raise ValueError(f'Expected input with embed_dim={self.in_features} (dim=-1), but found {x.size(-1)}')
        elif x.size(1) != self.num_experts:
            raise ValueError(f'Expected input with num_experts={self.num_experts} (dim=1), but found {x.size(1)}')
        x = einsum(x, self.weight, 'b n ... d1, n d1 d2 -> b n ... d2')
        if self.bias is not None:
            if x.ndim == 3:
                bias = rearrange(self.bias, 'n d -> () n d')
            elif x.ndim == 4:
                bias = rearrange(self.bias, 'n d -> () n () d')
            else:
                raise ValueError(f'Expected input to have 3 or 4 dimensions, but got {x.ndim}')
            x = x + bias
        return x

    def extra_repr(self) -> str:
        return f'in_features={self.in_features}, out_features={self.out_features}, num_experts={self.num_experts}, bias={self.bias is not None}'


class SoftMoE(nn.Module):
    def __init__(self, in_features: int, out_features: int, num_experts: int, slots_per_expert: int, bias: bool=True, device: Optional[Union[str, torch.device]]=None, dtype: Optional[torch.dtype]=None):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.num_experts = num_experts
        self.slots_per_expert = slots_per_expert
        self.bias = bias
        self.phi = nn.Parameter(torch.empty((in_features, num_experts, slots_per_expert), device=device, dtype=dtype))
        self.experts = MultiExpertLayer(in_features=in_features, out_features=out_features, num_experts=num_experts, bias=bias, device=device, dtype=dtype)
        self.reset_parameters()

    def reset_parameters(self) -> None:
        nn.init.kaiming_uniform_(self.phi, a=math.sqrt(5))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        if x.size(-1) != self.in_features:
            raise ValueError(f'Expected x.size(-1)={x.size(-1)} to match embed_dim={self.in_features}, but got {x.size(-1)}.')
        elif x.ndim != 3:
            raise ValueError(f'Expected input to have 3 dimensions, but got {x.ndim}.')
        logits = einsum(x, self.phi, 'b m d, d n p -> b m n p')
        gates = torch.softmax(logits, dim=2)
        x_expanded = rearrange(x, 'b m d -> b m () d')
        x_repeated = x_expanded.expand(x.size(0), x.size(1), self.num_experts, self.in_features)
        x_reshaped = rearrange(x_repeated, 'b m n d -> b n m d')
        expert_out = self.experts(x_reshaped)
        gates_reshaped = rearrange(gates, 'b m n p -> b n m p')
        gates_expanded = gates_reshaped.unsqueeze(-1)
        weighted = expert_out.unsqueeze(3) * gates_expanded
        output = weighted.sum(dim=3).sum(dim=1)
        return output


INIT_ARGS = {
    'in_features': 4,
    'out_features': 3,
    'num_experts': 2,
    'slots_per_expert': 5,
    'bias': True,
    'device': None,
    'dtype': None,
}

ARGS = {
    'x': torch.randn(2, 6, 4),
}

if __name__ == '__main__':
    model = SoftMoE(**INIT_ARGS)
    output = model(**ARGS)
    print(f"Output shape: {output.shape}")
    print(f"Expected shape: (2, 6, 3)")
    assert output.shape == (2, 6, 3), f"Shape mismatch: {output.shape}"
    print("Test passed!")
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
einops, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float — no output shape is stated: observe_signature.py observed this class with its constructor sizes canonicalised where it could, which the node does not record, so its shape is not stated as this setup block's
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
SoftMoELayerWrapper Ran
ran on a synthesized input
bwconrad/soft-moe/soft_moe/soft_moe.py
code served (permissive licence) · get_code("aaeb459d76d3691b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 3, dtype family float, shape (2, 5, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

# Define the layer class that will be used as the expert
class SimpleExpertLayer(nn.Module):
    def __init__(self, input_dim: int, output_dim: int):
        super().__init__()
        self.linear = nn.Linear(input_dim, output_dim)
    
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.linear(x)

INIT_ARGS = {
    'dim': 4,
    'num_experts': 2,
    'slots_per_expert': 3,
    'layer': SimpleExpertLayer,
    'normalize': True,
    'input_dim': 4,
    'output_dim': 4,
}

ARGS = {
    'x': torch.randn(2, 5, 4),
}

if __name__ == '__main__':
    from typing import Callable
    import torch.nn.functional as F
    
    def softmax(x: torch.Tensor, dim: int | tuple[int, ...]) -> torch.Tensor:
        max_vals = torch.amax(x, dim=dim, keepdim=True)
        e_x = torch.exp(x - max_vals)
        sum_exp = e_x.sum(dim=dim, keepdim=True)
        return e_x / sum_exp

    class SoftMoELayerWrapper(nn.Module):
        def __init__(self, dim: int, num_experts: int, slots_per_expert: int, layer: Callable, normalize: bool=True, **layer_kwargs) -> None:
            super().__init__()
            self.dim = dim
            self.num_experts = num_experts
            self.slots_per_expert = slots_per_expert
            self.normalize = normalize
            self.phi = nn.Parameter(torch.zeros(dim, num_experts, slots_per_expert))
            if self.normalize:
                self.scale = nn.Parameter(torch.ones(1))
            nn.init.normal_(self.phi, mean=0, std=1 / dim ** 0.5)
            self.experts = nn.ModuleList([layer(**layer_kwargs) for _ in range(num_experts)])

        def forward(self, x: torch.Tensor) -> torch.Tensor:
            assert x.shape[-1] == self.dim, f'Input feature dim of {x.shape[-1]} does not match layer dim of {self.dim}'
            assert len(x.shape) == 3, f'Input expected to have 3 dimensions but has {len(x.shape)}'
            phi = self.phi
            if self.normalize:
                x = F.normalize(x, dim=2)
                phi = self.scale * F.normalize(phi, dim=0)
            logits = torch.einsum('bmd,dnp->bmnp', x, phi)
            d = softmax(logits, dim=1)
            c = softmax(logits, dim=(2, 3))
            xs = torch.einsum('bmd,bmnp->bnpd', x, d)
            ys = torch.stack([f_i(xs[:, i, :, :]) for i, f_i in enumerate(self.experts)], dim=1)
            y = torch.einsum('bnpd,bmnp->bmd', ys, c)
            return y

    model = SoftMoELayerWrapper(**INIT_ARGS)
    output = model(**ARGS)
    print(f"Output shape: {output.shape}")
    print(f"Output: {output}")
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float, shape (2, 5, 4) — shape (2, 5, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
all_gather_same_dim Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
lucidrains/soft-moe-pytorch/soft_moe_pytorch/soft_moe.py
code served (permissive licence) · get_code("07572c9d0ff527a5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 3, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.distributed as dist
from unittest.mock import Mock, patch

torch.manual_seed(0)

# Mock distributed setup since we're running on CPU without actual distributed training
mock_world_size = 2

def mock_get_world_size():
    return mock_world_size

def mock_all_gather(gathered_tensors, tensor):
    # Simulate all_gather by copying the same tensor to all slots
    for i in range(len(gathered_tensors)):
        gathered_tensors[i].copy_(tensor)

# Patch the distributed functions
dist.get_world_size = mock_get_world_size
dist.all_gather = mock_all_gather

# Create a small tensor to gather
t = torch.randn(2, 4, dtype=torch.float32)

ARGS = {
    "t": t
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 3, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
all_gather_variable_dim Ran
ran on a synthesized input
lucidrains/soft-moe-pytorch/soft_moe_pytorch/soft_moe.py
code served (permissive licence) · get_code("f42235237f40a9fd")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.distributed as dist
from unittest.mock import Mock, MagicMock, patch

torch.manual_seed(0)

# Mock distributed environment since we can't actually initialize distributed training
with patch('torch.distributed.get_world_size') as mock_world_size, \
     patch('torch.distributed.get_rank') as mock_rank, \
     patch('torch.distributed.all_gather') as mock_all_gather:
    
    mock_world_size.return_value = 2
    mock_rank.return_value = 0
    
    def mock_all_gather_impl(gathered_list, tensor):
        # Simulate all_gather by filling the list with copies of the tensor
        for i in range(len(gathered_list)):
            gathered_list[i].copy_(tensor)
    
    mock_all_gather.side_effect = mock_all_gather_impl
    
    # Create a small test tensor with distinct dimensions
    # Shape: (batch=2, seq=4) - variable dimension along dim=1
    t = torch.randn(2, 4, dtype=torch.float32)
    
    ARGS = {
        't': t,
        'dim': 0,
        'sizes': None,
    }
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
einops, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
chunk_num Ran
ran, honoured the contract we drafted
lucidrains/soft-moe-pytorch/soft_moe_pytorch/soft_moe.py
code served (permissive licence) · get_code("279540af53f03369")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Analyze the function: chunk_num(num, chunks)
# - num: an integer representing the total number to divide
# - chunks: an integer representing how many chunks to create
# The function divides num into chunks parts as evenly as possible

ARGS = {
    "num": 10,
    "chunks": 3
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family int — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
compute_capacity Ran
ran, honoured the contract we drafted
google-research/vmoe/vmoe/moe.py
code served (permissive licence) · get_code("d9d07130e203c96f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import math
from typing import Literal

# Define ARGS dict with all parameters for compute_capacity
ARGS = {
    'num_tokens': 1000,
    'num_experts': 8,
    'capacity_factor': 1.25,
    'ceil_or_round': 'ceil',
    'multiple_of': 4,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
absl — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
cumsum_exclusive Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
lucidrains/soft-moe-pytorch/soft_moe_pytorch/soft_moe.py
code served (permissive licence) · get_code("4607f03213358cab")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 3, dtype family float, shape (2, 3, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn.functional as F

torch.manual_seed(0)

# Create a small tensor with distinct dimensions
# Shape: (2, 3, 4) - batch=2, intermediate=3, feature=4
t = torch.randn(2, 3, 4)

# dim parameter - must be negative
dim = -3

ARGS = {
    "t": t,
    "dim": dim
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float, shape (2, 3, 4) — shape (2, 3, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
default Ran
ran, violated the contract we drafted
this paper's copy was not recorded; identical code first harvested from ThomasMrY/VCT
pointer only; the stored copy get_code serves, from ThomasMrY/VCT, is pointer only (licence: NONE) · get_code("60fff7c3c400d7ff")
How it ran

The body of this sample is withheld: this paper's copy of the code was not recorded (its link names no repository or source path), so no licence was read for it. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

divisible_by Ran
ran, violated the contract we drafted
lucidrains/soft-moe-pytorch/soft_moe_pytorch/soft_moe.py
code served (permissive licence) · get_code("f912c209e9c899ca")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# The function divisible_by(num, den) checks if num % den == 0
# It takes two numeric arguments and performs modulo operation

ARGS = {
    "num": 10,
    "den": 2
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
exists Ran
ran, violated the contract we drafted
this paper's copy was not recorded; identical code first harvested from ThomasMrY/VCT
pointer only; the stored copy get_code serves, from ThomasMrY/VCT, is pointer only (licence: NONE) · get_code("aa5486a3650902d8")
How it ran

The body of this sample is withheld: this paper's copy of the code was not recorded (its link names no repository or source path), so no licence was read for it. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

gather_sizes Ran
ran; our contract draft was wrong, not the code
lucidrains/soft-moe-pytorch/soft_moe_pytorch/soft_moe.py
code served (permissive licence) · get_code("d3b9292d51192dc2")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned Tensor, ndim 1, dtype family int, shape (2,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.distributed as dist
from unittest.mock import MagicMock, patch

torch.manual_seed(0)

# Mock distributed setup since we're not actually running distributed training
mock_world_size = 2

def mock_get_world_size():
    return mock_world_size

def mock_all_gather(gathered_tensors, t):
    # Simulate all_gather by filling tensors with dummy data
    for i, tensor in enumerate(gathered_tensors):
        tensor.copy_(t + i)

# Patch dist functions
dist.get_world_size = mock_get_world_size
dist.all_gather = mock_all_gather

# Create a small tensor with distinct dimensions
# Shape: (2, 4, 3) - batch=2, feature=4, sequence=3
t = torch.randn(2, 4, 3, dtype=torch.float32)

ARGS = {
    "t": t,
    "dim": 1
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 1, dtype family int, shape (2,) — shape (2,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_default_moe_num_experts_and_last_n Ran
ran, honoured the contract we drafted
google-research/vmoe/vmoe/projects/soft_moe/configs/pretrain_jft4b.py
code served (permissive licence) · get_code("3c1f6eb9bbce7fc8")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned tuple, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function takes two parameters: variant and patch_size
# variant is a string key that must be one of: 'S', 'B', 'L', 'H'
# patch_size is an integer that must be one of: 16, 14
# The function looks up (variant, patch_size) tuples in dictionaries

ARGS = {
    'variant': 'S',
    'patch_size': 16
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family int — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_hyper Ran
ran; our contract draft was wrong, not the code
google-research/vmoe/vmoe/projects/soft_moe/configs/pretrain_jft4b.py
code served (permissive licence) · get_code("896382e782635552")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

class HyperStandIn:
    """Stand-in for a hyper object that has a product method."""
    def product(self, arg):
        """Returns a simple result from product method."""
        return {"result": "product_called"}

hyper = HyperStandIn()

ARGS = {
    "hyper": hyper,
    "model": "soft-s16"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
has_only_one_value Ran
ran, violated the contract we drafted
lucidrains/soft-moe-pytorch/soft_moe_pytorch/soft_moe.py
code served (permissive licence) · get_code("2924465c3aecc7f6")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function has_only_one_value checks if all elements in a tensor t
# are equal to the first element t[0].
# It requires t to be indexable (supports t[0]) and support == comparison and .all()
# This works with torch tensors.

# Create a tensor where all values are the same
t = torch.tensor([5, 5, 5, 5], dtype=torch.float32)

ARGS = {
    "t": t
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
l2_normalize Ran
ran; our contract draft was wrong, not the code
umbertocappellazzo/petl_ast/src/MoA.py
pointer only (licence: NONE) · get_code("4760e36330feac9d")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, umbertocappellazzo/petl_ast when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

make_adapter Ran
ran; our contract draft was wrong, not the code
umbertocappellazzo/petl_ast/src/MoA.py
pointer only (licence: NONE) · get_code("d1be2ead9db77c59")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, umbertocappellazzo/petl_ast when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

softmax Ran
ran; our contract draft was wrong, not the code
bwconrad/soft-moe/soft_moe/soft_moe.py
code served (permissive licence) · get_code("84195cc106a2908b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Create a small input tensor with shape (2, 4)
x = torch.randn(2, 4)

# Test with single dimension
dim_single = 1

# Test with multiple dimensions
dim_multiple = (0, 1)

# Create ARGS dict with single dimension case
ARGS = {
    "x": x,
    "dim": dim_single
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
unpack_one Ran
ran; our contract draft was wrong, not the code
lucidrains/soft-moe-pytorch/soft_moe_pytorch/soft_moe.py
code served (permissive licence) · get_code("57fd83aecedfe674")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np
from einops import pack, unpack

torch.manual_seed(0)

# unpack_one calls unpack(t, ps, pattern) and returns the first element
# unpack expects:
# - t: a tensor to unpack
# - ps: a list of packed shapes (tuples indicating original shapes before packing)
# - pattern: an einops pattern string describing the packing operation

# First, let's create a simple example by packing a tensor, then unpacking it
# This ensures our arguments are valid

# Create a small tensor to pack
tensor1 = torch.randn(2, 4)  # shape (2, 4)
tensor2 = torch.randn(3, 4)  # shape (3, 4)

# Pack them together using einops.pack
packed_tensor, packed_shapes = pack([tensor1, tensor2], '* d')

# Now we have:
# - packed_tensor: the packed result
# - packed_shapes: the list of original shapes

# The pattern used for packing was '* d', so we use the same for unpacking
pattern = '* d'

ARGS = {
    't': packed_tensor,
    'ps': packed_shapes,
    'pattern': pattern
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
einops, numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
SoftMoA Not yet run
unverified (harvested, no recorded run)
umbertocappellazzo/petl_ast/src/MoA.py
pointer only (licence: NONE) · get_code("11e34ef59abccf2b")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, umbertocappellazzo/petl_ast when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

SoftMoE Not yet run
unverified (harvested, no recorded run)
lucidrains/soft-moe-pytorch/soft_moe_pytorch/soft_moe.py
code served (permissive licence) · get_code("0193b08b6700ed29")
How it ran

no invocation recorded.

pack_one Not yet run
unverified (harvested, no recorded run)
this paper's copy was not recorded; identical code first harvested from lucidrains/BS-RoFormer
pointer only · get_code("ab03f61657f300bc")
How it ran

The body of this sample is withheld: this paper's copy of the code was not recorded (its link names no repository or source path), so no licence was read for it. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

split_by_rank Not yet run
unverified (harvested, no recorded run)
lucidrains/soft-moe-pytorch/soft_moe_pytorch/soft_moe.py
code served (permissive licence) · get_code("7079481b9b2287ec")
How it ran

no invocation recorded.

Also present in other papers' code

Of 33 samples lifted from this paper's linked repositories, 33 carry a function identity and 0 do not. Of the 32 distinct functions with an identity, 8 are also present in other papers whose code we hold (6 with at least one paper dated earlier, 2 the same month, 8 later) and 24 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
exists
brick of this paper's assembly: helper
486 other papers
206 dated earlier, 17 the same month, 257 later, 4 same year, order not determinable, 2 undated · 78 only as an assembly brick
1603.08155 (2016-03)
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
1611.07004 (2016-11)
Image-to-Image Translation with Conditional Adversarial Networks
1804.04637 (2018-04)
EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models
and 483 more not named here
default
brick of this paper's assembly: helper
163 other papers
92 dated earlier, 4 the same month, 65 later, 1 same year, order not determinable, 1 undated · 27 only as an assembly brick
1603.08155 (2016-03)
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
1810.00825 (2018-10)
Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks
1907.01470 (2019-07)
Augmenting Self-attention with Persistent Memory
and 160 more not named here
divisible_by 16 other papers
4 dated earlier, 12 later · 2 only as an assembly brick
2006.07733 (2020-06)
Bootstrap your own latent: A new approach to self-supervised Learning
2010.10241 (2020-10)
BYOL works even without batch statistics
2102.06810 (2021-02)
Understanding self-supervised Learning Dynamics without Contrastive Pairs
and 13 more not named here
pack_one
brick of this paper's assembly: helper
16 other papers
6 dated earlier, 10 later · 4 only as an assembly brick
2203.07852 (2022-03)
Block-Recurrent Transformers
2205.14135 (2022-05)
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
2206.11990 (2022-06)
Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs
and 13 more not named here
unpack_one
brick of this paper's assembly: helper
11 other papers
4 dated earlier, 7 later · 4 only as an assembly brick
2203.07852 (2022-03)
Block-Recurrent Transformers
2205.14135 (2022-05)
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
2206.11990 (2022-06)
Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs
and 8 more not named here
LayerNorm
brick of this paper's assembly: helper
4 other papers
3 dated earlier, 1 later
2203.07852 (2022-03)
Block-Recurrent Transformers
2204.02311 (2022-04)
PaLM: Scaling Language Modeling with Pathways
2212.11972 (2022-12)
Scalable Adaptive Computation for Iterative Generation
and 1 more not named here
FeedForward
brick of this paper's assembly: model
1 other paper
1 later
2507.06363 (2025-07)
Mamba Goes HoME: Hierarchical Soft Mixture-of-Experts for 3D Medical Image Segmentation
GLUFeedForward
brick of this paper's assembly: model
1 other paper
1 later
2507.06363 (2025-07)
Mamba Goes HoME: Hierarchical Soft Mixture-of-Experts for 3D Medical Image Segmentation

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Sparse mixture of expert architectures (MoEs) scale model capacity without significant increases in training or inference costs. Despite their success, MoEs suffer from a number of issues: training instability, token dropping, inability to scale the number of experts, or ineffective finetuning. In this work, we propose Soft MoE, a fully-differentiable sparse Transformer that addresses these challenges, while maintaining the benefits of MoEs. Soft MoE performs an implicit soft assignment by passing different weighted combinations of all input tokens to each expert. As in other MoEs, experts in Soft MoE only process a subset of the (combined) tokens, enabling larger model capacity (and performance) at lower inference cost. In the context of visual recognition, Soft MoE greatly outperforms dense Transformers (ViTs) and popular MoEs (Tokens Choice and Experts Choice). Furthermore, Soft MoE scales well: Soft MoE Huge/14 with 128 experts in 16 MoE layers has over 40× more parameters than ViT Huge/14, with only 2% increased inference time, and substantially better quality. * Equal contribution. The order was decided by a coin toss.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2308.00951")
get_code_for_paper("2308.00951")
have("2308.00951")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2308.00951.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2308.00951.svg)](https://syntology.ai/paper/2308.00951)

Connect an agent — have() is free.