SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2406.14909 · 2024

Mixture of Attention Spans: Optimizing LLM Inference Efficiency with Heterogeneous Sliding-Window Lengths

arXiv · PDF · Open in the Atlas

Code that ran

19 of 22 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 2 honoured the contract we drafted, 1 violated it and 0 have no contract verdict; 16 failures of our instrument; 3 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2406.14909.json · badge: /api/ran/2406.14909.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
thu-nics/moa canonical (classifier label)
classifier basis: census
19 ran · 3 no recorded run
FunctionStatusWhere it lives
Trainer_attention_grad_compute_loss Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
thu-nics/moa/MoA/attention/sparse_mask.py
code served (permissive licence) · get_code("b8258f95889e4952")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from torch.nn import Module
from collections import namedtuple

torch.manual_seed(0)

# Mock PeftModel for type checking
class PeftModel:
    pass

# Create a minimal model that outputs attentions
class MockModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 4)
    
    def forward(self, input_ids=None, attention_mask=None, labels=None, output_attentions=False):
        batch_size = 2
        seq_len = 3
        num_heads = 2
        
        # Create mock attention tensors with 4D shape (batch, num_heads, seq_len, seq_len)
        attention_1 = torch.randn(batch_size, num_heads, seq_len, seq_len)
        attention_2 = torch.randn(batch_size, num_heads, seq_len, seq_len)
        
        # Create loss
        loss = torch.tensor(1.5, requires_grad=True)
        
        # Create output object
        output = namedtuple('Output', ['loss', 'attentions'])
        return output(loss=loss, attentions=(attention_1, attention_2))

# Create a minimal label smoother
class MockLabelSmoother:
    def __call__(self, outputs, labels, shift_labels=False):
        return torch.tensor(1.5, requires_grad=True)

# Create a minimal args object
class MockArgs:
    def __init__(self):
        self.past_index = -1

# Create a minimal trainer object
class MockTrainer:
    def __init__(self):
        self.label_smoother = MockLabelSmoother()
        self.args = MockArgs()
        self.atten_map = None
        self.attention_grads = None
        self._past = None

# Create input tensors
inputs = {
    'input_ids': torch.randint(0, 100, (2, 3)),
    'attention_mask': torch.ones((2, 3)),
}

model = MockModel()
trainer = MockTrainer()

ARGS = {
    'self': trainer,
    'model': model,
    'inputs': inputs,
    'return_outputs': False,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch, transformers — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
count_distinct_elements_per_row Ran
ran; our contract draft was wrong, not the code
thu-nics/moa/MoA/attention/density_calculation.py
code served (permissive licence) · get_code("29414013ce12967c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 1, dtype family int, shape (2,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Create a small 2D integer tensor with some repeated values
# Shape: (2, 4) - 2 rows, 4 columns
lut = torch.tensor([
    [1, 2, 1, 3],
    [5, 5, 5, 7]
], dtype=torch.int32)

ARGS = {
    "lut": lut
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 1, dtype family int, shape (2,) — shape (2,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
find_subtensor_position Ran
ran; our contract draft was wrong, not the code
thu-nics/moa/MoA/dataset/utils.py
code served (permissive licence) · get_code("061f1b20e4c107be")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Create a small sub_tensor to search for
# Shape: (len_sub,) - let's use length 2
sub_tensor = torch.tensor([1, 2])

# Create a main_tensor with batch dimension
# Shape: (batch, len_main) - let's use batch=2, len_main=5
main_tensor = torch.tensor([
    [0, 1, 2, 3, 1],
    [1, 2, 4, 1, 2]
])

ARGS = {
    "sub_tensor": sub_tensor,
    "main_tensor": main_tensor
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list — a list output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
gen_band_pattern Ran
ran, violated the contract we drafted
thu-nics/moa/MoA/attention/pattern.py
code served (permissive licence) · get_code("149e4b82b7620c4b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 4, dtype family float, shape (2, 3, 4, 5) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import Tensor

torch.manual_seed(0)

# Small test values
num_query = 4
num_key = 5
band_width_int = 3
num_layer = 2
num_head = 3

ARGS = {
    "num_query": num_query,
    "num_key": num_key,
    "band_width": band_width_int,
    "dtype": torch.bool,
    "num_layer": num_layer,
    "num_head": num_head,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 4, dtype family float, shape (2, 3, 4, 5) — shape (2, 3, 4, 5) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
gen_block_sparse_layout_mask Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
thu-nics/moa/MoA/attention/sparse_mask.py
code served (permissive licence) · get_code("6d1389e6b2d85469")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from typing import Optional, Tuple

torch.manual_seed(0)

# Create a small matrix tensor
# Shape: [keep_dim=2, matrix_size_x=8, matrix_size_y=12]
# This allows for num_block_x=2, num_block_y=3 with block sizes of 4x4
matrix = torch.randn(2, 8, 12)

# Number of blocks in x and y directions
num_block_x = 2
num_block_y = 3

# Number of blocks to keep per keep dimension
# Shape: [2] to match keep_dim
block_keep_num = torch.tensor([2, 2])

# Row balance flag
row_balance = False

# Custom aggregation function (optional)
def custom_agg_func(block_matrix: torch.Tensor) -> torch.Tensor:
    """Aggregate block matrix by taking mean over last two dimensions."""
    return torch.mean(block_matrix, dim=[-1, -2])

block_aggregation_func = custom_agg_func

# Causal mask flag
is_causal = False

ARGS = {
    "matrix": matrix,
    "num_block_x": num_block_x,
    "num_block_y": num_block_y,
    "block_keep_num": block_keep_num,
    "row_balance": row_balance,
    "block_aggregation_func": block_aggregation_func,
    "is_causal": is_causal,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
gen_global_pattern Ran
ran; our contract draft was wrong, not the code
thu-nics/moa/MoA/attention/pattern.py
code served (permissive licence) · get_code("7e1d14c04aa0dc2d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 4, dtype family float, shape (1, 1, 8, 8) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

ARGS = {
    "num_query": 8,
    "num_key": 8,
    "key_pos": [0, 2, 5],
    "dtype": torch.bool,
    "num_layer": 1,
    "num_head": 1,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 4, dtype family float, shape (1, 1, 8, 8) — shape (1, 1, 8, 8) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
head_index_to_split_index Ran
ran; our contract draft was wrong, not the code
thu-nics/moa/MoA/kernels/flash_decoding_moa.py
code served (permissive licence) · get_code("2d84082c203b2b52")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 1, dtype family int, shape (5,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# head_index should be a tensor of shape (num_heads + 1)
# representing the start index of each head
# Let's create a simple example with 3 heads
head_index = torch.tensor([0, 10, 20, 30], dtype=torch.long)

# split_size is an integer
split_size = 7

ARGS = {
    "head_index": head_index,
    "split_size": split_size,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 1, dtype family int, shape (5,) — shape (5,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
hotpotQA_to_multi_round_qa Ran
ran; our contract draft was wrong, not the code
thu-nics/moa/MoA/dataset/convert.py
code served (permissive licence) · get_code("21ce5f5f68d41dad")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Construct the entry dictionary based on the hotpotQA data structure
entry = {
    "answer": "This is the answer",
    "context": {
        "sentences": [["Sent 1"], ["Sent 21", "Sent 22"]],
        "title": ["Title1", "Title 2"]
    },
    "id": "000001",
    "level": "medium",
    "question": "What is the answer?",
    "supporting_facts": {
        "sent_id": [0, 1],
        "title": ["Title1", "Title 2"]
    },
    "type": "comparison"
}

ARGS = {
    "entry": entry
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
layout_to_lut Ran
ran; our contract draft was wrong, not the code
thu-nics/moa/MoA/attention/convert.py
code served (permissive licence) · get_code("2866d7c2b62c260d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Create a small layout tensor with shape (layer, num_heads, num_block, num_block)
# Using distinct sizes: 2 layers, 3 heads, 4 blocks
layout = torch.zeros((2, 3, 4, 4), dtype=torch.bool)

# Fill with some sparse patterns so we have non-zero values
# For layer 0, head 0
layout[0, 0, 0, [0, 2]] = True
layout[0, 0, 1, [1, 3]] = True
layout[0, 0, 2, [0, 1, 2]] = True
layout[0, 0, 3, [1, 2]] = True

# For layer 0, head 1
layout[0, 1, 0, [1, 3]] = True
layout[0, 1, 1, [0, 2]] = True
layout[0, 1, 2, [1, 3]] = True
layout[0, 1, 3, [0, 2]] = True

# For layer 0, head 2
layout[0, 2, 0, [0, 1]] = True
layout[0, 2, 1, [2, 3]] = True
layout[0, 2, 2, [0, 3]] = True
layout[0, 2, 3, [1, 2]] = True

# For layer 1, head 0
layout[1, 0, 0, [0, 1, 2]] = True
layout[1, 0, 1, [1, 3]] = True
layout[1, 0, 2, [0, 2]] = True
layout[1, 0, 3, [0, 1]] = True

# For layer 1, head 1
layout[1, 1, 0, [1, 2]] = True
layout[1, 1, 1, [0, 1, 3]] = True
layout[1, 1, 2, [2, 3]] = True
layout[1, 1, 3, [0, 2]] = True

# For layer 1, head 2
layout[1, 2, 0, [0, 3]] = True
layout[1, 2, 1, [1, 2]] = True
layout[1, 2, 2, [0, 1, 2]] = True
layout[1, 2, 3, [1, 3]] = True

ARGS = {
    'layout': layout
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
lut_to_permutation Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
thu-nics/moa/MoA/attention/permutation_utils.py
code served (permissive licence) · get_code("e6a764e7c48e4500")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function expects a list of lookup tables (LUTs)
# Each LUT is a 3D tensor of shape (num_heads, height, width)
# The function serializes each head, sorts them, and clusters identical heads

num_heads = 32
num_key_value_groups = 1

# Create sample LUT tensors
# Shape: (num_heads, height, width) where height and width are arbitrary dimensions
lut1 = torch.randn(num_heads, 4, 5)
lut2 = torch.randn(num_heads, 4, 5)

# Make some heads identical to ensure clustering works
# Heads 0-15 get one pattern, heads 16-31 get another pattern
lut1[0:16] = torch.ones(16, 4, 5) * 0.5
lut1[16:32] = torch.ones(16, 4, 5) * 1.5

lut2[0:16] = torch.ones(16, 4, 5) * 0.3
lut2[16:32] = torch.ones(16, 4, 5) * 1.3

lut_list = [lut1, lut2]

ARGS = {
    'lut_list': lut_list,
    'num_heads': num_heads,
    'num_key_value_groups': num_key_value_groups,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
mask_to_layout Ran
ran; our contract draft was wrong, not the code
thu-nics/moa/MoA/attention/convert.py
code served (permissive licence) · get_code("8633f899e4d7407c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 3, dtype family float, shape (2, 4, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn.functional as F

torch.manual_seed(0)

# Create a small mask tensor
# Shape: (batch=2, token_length=8, token_length=8)
mask = torch.randint(0, 2, (2, 8, 8)).float()

# block_size parameter
block_size = 2

ARGS = {
    "mask": mask,
    "block_size": block_size,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float, shape (2, 4, 4) — shape (2, 4, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
matrix_reserve_score Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
thu-nics/moa/MoA/attention/sparse_mask.py
code served (permissive licence) · get_code("0bfc9650b5be48e8")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 1, dtype family float, shape (2,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from torch import Tensor

torch.manual_seed(0)

# Create small tensors with distinct dimensions
# matrix shape: [keep_dim, matrix_size_x, matrix_size_y]
# Using keep_dim=2, matrix_size_x=4, matrix_size_y=3
keep_dim = 2
matrix_size_x = 4
matrix_size_y = 3

# Create matrix tensor with random values
matrix = torch.randn(keep_dim, matrix_size_x, matrix_size_y)

# Create mask tensor with binary values (0 and 1)
mask = torch.randint(0, 2, (keep_dim, matrix_size_x, matrix_size_y)).float()

# is_causal: boolean flag
is_causal = False

# norm_score: boolean flag
norm_score = False

ARGS = {
    "matrix": matrix,
    "mask": mask,
    "is_causal": is_causal,
    "norm_score": norm_score,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 1, dtype family float, shape (2,) — shape (2,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
moa_config_to_cache_config Ran
ran; our contract draft was wrong, not the code
thu-nics/moa/MoA/attention/cache_utils.py
code served (permissive licence) · get_code("1ada47e15f33f461")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd
import math

# Create a minimal moa_config dictionary
# The function expects 'alphas' and 'betas' keys
# alphas and betas are nested lists: [layer][head]
# We'll create a small 2-layer, 3-head config

moa_config = {
    'alphas': [
        [10.5, 20.3, 15.7],  # layer 0, 3 heads
        [12.1, 18.9, 22.4],  # layer 1, 3 heads
    ],
    'betas': [
        [0.1, 0.15, 0.12],   # layer 0, 3 heads
        [0.11, 0.14, 0.13],  # layer 1, 3 heads
    ]
}

ARGS = {
    'moa_config': moa_config,
    'seq_len': 512,
    'max_new_token': 1024,
    'sink_size': 64,
    'minimum_cache_size': 128,
    'split_size': 64,
    'verbose': True,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
moa_config_to_permutation Ran
ran; our contract draft was wrong, not the code
thu-nics/moa/MoA/attention/permutation_utils.py
code served (permissive licence) · get_code("2cd7b8febe087a7d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Create a minimal moa_config dictionary with alphas and betas
# alphas and betas are lists of lists (one per layer), where each inner list contains values for each head
moa_config = {
    'alphas': [
        [1, 2, 1, 3],  # Layer 0: 4 heads with alpha values
        [2, 1, 2],     # Layer 1: 3 heads with alpha values
    ],
    'betas': [
        [5, 4, 5, 6],  # Layer 0: 4 heads with beta values
        [3, 2, 3],     # Layer 1: 3 heads with beta values
    ]
}

ARGS = {
    'moa_config': moa_config
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
multiNews_to_multi_round_qa Ran
ran; our contract draft was wrong, not the code
thu-nics/moa/MoA/dataset/convert.py
code served (permissive licence) · get_code("3dca86fbcf33d8a4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

# Construct a valid entry for the multiNews_to_multi_round_qa function
entry = {
    'document': 'some line val\nanother line\nthird line of context',
    'summary': '– target val line'
}

ARGS = {
    'entry': entry
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
permute_lut Ran
ran; our contract draft was wrong, not the code
thu-nics/moa/MoA/attention/permutation_utils.py
code served (permissive licence) · get_code("77491a1736272578")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family float, shape (32, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Construct arguments for permute_lut function
lut = torch.randn(32, 4)  # Shape: (num_heads, feature_dim)
permutation = torch.tensor([1, 0, 3, 2, 5, 4, 7, 6, 9, 8, 11, 10, 13, 12, 15, 14, 17, 16, 19, 18, 21, 20, 23, 22, 25, 24, 27, 26, 29, 28, 31, 30])  # Permutation for 32 heads
num_heads = 32

ARGS = {
    "lut": lut,
    "permutation": permutation,
    "num_heads": num_heads
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (32, 4) — shape (32, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
qasper_to_multi_round_qa Ran
ran; our contract draft was wrong, not the code
thu-nics/moa/MoA/dataset/convert.py
code served (permissive licence) · get_code("89ec645a857ef9af")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

ARGS = {
    'entry': {
        'id': 'paper_001',
        'title': 'Understanding Neural Networks',
        'abstract': 'This paper explores the fundamental principles of neural network architectures and their applications in modern machine learning.',
        'full_text': {
            'paragraphs': [
                [
                    'Neural networks are computational models inspired by biological neural networks.',
                    'They consist of interconnected nodes organized in layers.'
                ],
                [
                    'Deep learning has revolutionized computer vision and natural language processing.',
                    'Convolutional networks are particularly effective for image recognition tasks.'
                ]
            ],
            'section_name': ['Introduction', 'Background']
        },
        'qas': {
            'question': [
                'What are neural networks?',
                'How have neural networks impacted machine learning?'
            ],
            'question_id': ['q_001', 'q_002'],
            'question_writer': ['writer_001', 'writer_002'],
            'nlp_background': [True, False],
            'topic_background': [True, True],
            'paper_read': [True, False],
            'search_query': ['neural networks basics', 'deep learning applications'],
            'answers': [
                {
                    'annotation_id': ['ann_001', 'ann_002'],
                    'answer': [
                        {
                            'unanswerable': False,
                            'extractive_spans': ['computational models inspired by biological neural networks'],
                            'yes_no': False,
                            'free_form_answer': 'Neural networks are computational models consisting of interconnected nodes organized in layers.',
                            'evidence': ['Neural networks are computational models inspired by biological neural networks.'],
                            'highlighted_evidence': ['Neural networks are computational models']
                        },
                        {
                            'unanswerable': False,
                            'extractive_spans': [],
                            'yes_no': False,
                            'free_form_answer': 'They are inspired by biological systems and organized in layers.',
                            'evidence': ['They consist of interconnected nodes organized in layers.'],
                            'highlighted_evidence': ['interconnected nodes organized in layers']
                        }
                    ],
                    'worker_id': ['worker_001', 'worker_002']
                },
                {
                    'annotation_id': ['ann_003'],
                    'answer': [
                        {
                            'unanswerable': False,
                            'extractive_spans': ['revolutionized computer vision and natural language processing'],
                            'yes_no': False,
                            'free_form_answer': 'Deep learning has revolutionized computer vision and natural language processing.',
                            'evidence': ['Deep learning has revolutionized computer vision and natural language processing.'],
                            'highlighted_evidence': ['revolutionized computer vision']
                        }
                    ],
                    'worker_id': ['worker_003']
                }
            ]
        }
    }
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
streamingllm_attention_density Ran
ran, honoured the contract we drafted
thu-nics/moa/MoA/attention/density_calculation.py
code served (permissive licence) · get_code("cd7b0c5e2a16d0cd")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

ARGS = {
    "global_size": 4,
    "band_size": 1024,
    "kv_seq_len": 8192,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
streamingllm_kv_cache_density Ran
ran, honoured the contract we drafted
thu-nics/moa/MoA/attention/density_calculation.py
code served (permissive licence) · get_code("758a42acae59b91f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

ARGS = {
    "global_size": 4,
    "band_size": 1024,
    "kv_seq_len": 8192,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
block_sparse_to_dense Not yet run
unverified (harvested, no recorded run)
thu-nics/moa/MoA/attention/convert.py
code served (permissive licence) · get_code("98558d05575a5d9f")
How it ran

no invocation recorded.

gen_causal_pattern Not yet run
unverified (harvested, no recorded run)
thu-nics/moa/MoA/attention/pattern.py
code served (permissive licence) · get_code("ef170fe22d7a924e")
How it ran

no invocation recorded.

update_model_function Not yet run
unverified (harvested, no recorded run)
thu-nics/moa/MoA/models/interface.py
code served (permissive licence) · get_code("e55fe24f97290938")
How it ran

no invocation recorded.

Also present in other papers' code

Of 22 samples lifted from this paper's linked repositories, 22 carry a function identity and 0 do not. Of the 22 distinct functions with an identity, 0 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 0 later) and 22 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Sliding-window attention offers a hardware-efficient solution to the memory and throughput challenges of Large Language Models (LLMs) in long-context scenarios. Existing methods typically employ a single window length across all attention heads and input sizes. However, this uniform approach fails to capture the heterogeneous attention patterns inherent in LLMs, ignoring their distinct accuracy-latency trade-offs. To address this challenge, we propose *Mixture of Attention Spans* (MoA), which automatically tailors distinct sliding-window length configurations to different heads and layers. MoA constructs and navigates a search space of various window lengths and their scaling rules relative to input sizes. It profiles the model, evaluates potential configurations, and pinpoints the optimal length configurations for each head. MoA adapts to varying input sizes, revealing that some attention heads expand their focus to accommodate longer inputs, while other heads consistently concentrate on fixed-length local contexts. Experiments show that MoA increases the effective context length by 3.9x with the same average sliding-window length, boosting retrieval accuracy by 1.5-7.1x over the uniform-window baseline across Vicuna-{7B, 13B} and Llama3-{8B, 70B} models. Moreover, MoA narrows the performance gap with full attention, reducing the maximum relative performance drop from 9%-36% to within 5% across three long-context understanding benchmarks. MoA achieves a 1.2-1.4x GPU memory reduction, boosting decode throughput by 6.6-8.2x and 1.7-1.9x over FlashAttention2 and vLLM, with minimal performance impact. Our code is available at: https://github.com/thu-nics/MoA

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2406.14909")
get_code_for_paper("2406.14909")
have("2406.14909")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2406.14909.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2406.14909.svg)](https://syntology.ai/paper/2406.14909)

Connect an agent — have() is free.