SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2412.09501 · 2024

Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition

arXiv · PDF · Open in the Atlas

Code that ran

14 of 19 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 1 honoured the contract we drafted, 1 violated it and 0 have no contract verdict; 12 failures of our instrument; 5 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2412.09501.json · badge: /api/ran/2412.09501.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
dvlab-research/Lyra canonical (classifier label)
classifier basis: census
14 ran · 5 no recorded run
FunctionStatusWhere it lives
compute_softdtw Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
dvlab-research/Lyra/lyra/model/soft_dtw_cuda.py
code served (permissive licence) · get_code("34f95bf62dff3ebd")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 3, dtype family float, shape (2, 6, 5) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Set random seed for reproducibility
np.random.seed(0)

# Create small dummy cost matrices D with shape (B, N, M)
# B=2 (batch size), N=4 (first sequence length), M=3 (second sequence length)
B = 2
N = 4
M = 3
D = np.random.randn(B, N, M).astype(np.float64)

# gamma is the temperature parameter for soft-DTW
gamma = 1.0

# bandwidth is the constraint for diagonal band (0 means no constraint)
bandwidth = 0

ARGS = {
    "D": D,
    "gamma": gamma,
    "bandwidth": bandwidth,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numba, numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 3, dtype family float, shape (2, 6, 5) — shape (2, 6, 5) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
compute_softdtw_backward Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
dvlab-research/Lyra/lyra/model/soft_dtw_cuda.py
code served (permissive licence) · get_code("ec64f40bfc12544a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 3, dtype family float, shape (2, 4, 3) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Set random seed for reproducibility
np.random.seed(0)

# Create small test arrays
B = 2  # batch size
N = 4  # first sequence length
M = 3  # second sequence length

# D_ is the cost matrix of shape (B, N, M)
D_ = np.random.randn(B, N, M).astype(np.float64)

# R is the accumulated cost matrix of shape (B, N+2, M+2)
# It needs to be padded with 2 on each side, so shape is (B, N+2, M+2)
R = np.zeros((B, N + 2, M + 2), dtype=np.float64)
# Initialize with some values in the valid region
R[:, 1:N+1, 1:M+1] = np.random.randn(B, N, M).astype(np.float64)

# gamma is a temperature parameter (positive float)
gamma = 1.0

# bandwidth is a constraint parameter (can be 0 for no constraint, or positive)
bandwidth = 0

ARGS = {
    "D_": D_,
    "R": R,
    "gamma": gamma,
    "bandwidth": bandwidth
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numba, numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 3, dtype family float, shape (2, 4, 3) — shape (2, 4, 3) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_patches Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
dvlab-research/Lyra/lyra/mm_utils.py
pointer only (licence: Apache-2.0); the stored copy get_code serves, from HumanMLLM/LLaVA-Scissor, is pointer only (licence: NONE) · get_code("b0a851cae92754ff")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (Apache-2.0), but get_code serves the stored copy, first harvested from HumanMLLM/LLaVA-Scissor, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

forward_layer Ran
ran; our contract draft was wrong, not the code
dvlab-research/Lyra/lyra/model/qwen2vl_top_attn.py
code served (permissive licence) · get_code("aebe8894bde0f55e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Build minimal mock modules that satisfy the function's usage
class MockLayerNorm(nn.Module):
    def __init__(self, hidden_size):
        super().__init__()
        self.weight = nn.Parameter(torch.ones(hidden_size))
        self.bias = nn.Parameter(torch.zeros(hidden_size))
    
    def forward(self, x):
        return x * self.weight + self.bias

class MockSelfAttention(nn.Module):
    def __init__(self, hidden_size):
        super().__init__()
        self.hidden_size = hidden_size
    
    def forward(self, hidden_states, attention_mask=None, position_ids=None, 
                past_key_value=None, output_attentions=False, use_cache=False,
                cache_position=None, position_embeddings=None, **kwargs):
        batch_size, seq_len, _ = hidden_states.shape
        
        # Return: (hidden_states, self_attn_weights, present_key_value)
        output_hidden = hidden_states * 0.5
        
        self_attn_weights = None
        if output_attentions:
            self_attn_weights = torch.randn(batch_size, 1, seq_len, seq_len)
        
        present_key_value = None
        if use_cache:
            present_key_value = (
                torch.randn(batch_size, 1, seq_len, self.hidden_size // 1),
                torch.randn(batch_size, 1, seq_len, self.hidden_size // 1)
            )
        
        return output_hidden, self_attn_weights, present_key_value

class MockMLP(nn.Module):
    def __init__(self, hidden_size):
        super().__init__()
        self.fc = nn.Linear(hidden_size, hidden_size)
    
    def forward(self, x):
        return self.fc(x)

class MockTransformerLayer(nn.Module):
    def __init__(self, hidden_size):
        super().__init__()
        self.input_layernorm = MockLayerNorm(hidden_size)
        self.self_attn = MockSelfAttention(hidden_size)
        self.post_attention_layernorm = MockLayerNorm(hidden_size)
        self.mlp = MockMLP(hidden_size)

# Create the layer instance
hidden_size = 4
layer = MockTransformerLayer(hidden_size)

# Construct arguments
batch_size = 2
seq_length = 3

ARGS = {
    'self': layer,
    'hidden_states': torch.randn(batch_size, seq_length, hidden_size),
    'attention_mask': None,
    'position_ids': torch.arange(seq_length).unsqueeze(0).expand(batch_size, -1),
    'past_key_value': None,
    'output_attentions': False,
    'use_cache': False,
    'cache_position': torch.arange(seq_length),
    'position_embeddings': (torch.randn(seq_length, hidden_size), torch.randn(seq_length, hidden_size)),
    'context_labels': torch.tensor([[True, False, True], [False, True, False]]),
    'text_labels': torch.tensor([[True, True, False], [True, False, True]]),
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
forward_model Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
dvlab-research/Lyra/lyra/model/qwen2vl_top_attn.py
code served (permissive licence) · get_code("74e2126c3223cdd8")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned CompressModelOutputWithPast. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from dataclasses import dataclass
from typing import List, Optional, Tuple

torch.manual_seed(0)

# Build a minimal config
class Config:
    def __init__(self):
        self.output_attentions = False
        self.output_hidden_states = False
        self.use_cache = True
        self.use_return_dict = True
        self.compress_gap = 2
        self.keep_rate = 0.8

# Build a minimal rotary embedding module
class RotaryEmbedding(nn.Module):
    def forward(self, hidden_states, position_ids):
        # Returns position embeddings with shape compatible with hidden_states
        batch_size = hidden_states.shape[1]
        seq_len = position_ids.shape[-1]
        return torch.randn(3, batch_size, seq_len, 4)

# Build a minimal decoder layer
class DecoderLayer(nn.Module):
    def forward(self, hidden_states, attention_mask=None, position_ids=None, 
                past_key_value=None, output_attentions=False, use_cache=False,
                cache_position=None, position_embeddings=None, context_labels=None,
                text_labels=None):
        # Returns (hidden_states, attentions, next_cache)
        attn_weights = torch.randn(2, 8, 5, 5) if output_attentions else None
        next_cache = [torch.randn(2, 8, 3, 4), torch.randn(2, 8, 3, 4)] if use_cache else None
        if output_attentions:
            return (hidden_states, attn_weights, next_cache)
        else:
            return (hidden_states, next_cache)

# Build the model with necessary attributes and methods
class ForwardModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.config = Config()
        self.gradient_checkpointing = False
        self.training = False
        
        # Embed tokens layer
        self.embed_tokens = nn.Embedding(100, 4)
        
        # Decoder layers
        self.layers = nn.ModuleList([DecoderLayer() for _ in range(2)])
        
        # Norm layer
        self.norm = nn.LayerNorm(4)
        
        # Rotary embeddings
        self.rotary_emb = RotaryEmbedding()
    
    def _update_causal_mask(self, attention_mask, inputs_embeds, cache_position, 
                           past_key_values, output_attentions):
        # Return a causal mask
        seq_len = inputs_embeds.shape[1]
        return torch.tril(torch.ones(seq_len, seq_len))
    
    def _gradient_checkpointing_func(self, func, *args, **kwargs):
        return func(*args, **kwargs)
    
    def merge_visual_tokens(self, hidden_states, context_labels, text_labels, labels,
                           attn_weights, causal_mask, position_ids, position_embeddings,
                           keep_rate):
        # Return modified tensors
        return (hidden_states, context_labels, text_labels, labels, causal_mask, 
                position_ids, position_embeddings)

# Create model instance
model = ForwardModel()

# Create valid tensors
batch_size = 2
seq_len = 5
hidden_dim = 4
num_heads = 8
head_dim = 4

input_ids = torch.randint(0, 100, (batch_size, seq_len))
attention_mask = torch.ones(batch_size, seq_len)
position_ids = torch.arange(seq_len).unsqueeze(0).expand(batch_size, -1)
cache_position = torch.arange(seq_len)

# Create past_key_values with get_seq_length method
class PastKeyValues:
    def __init__(self):
        self.data = [[torch.randn(batch_size, num_heads, 0, head_dim),
                      torch.randn(batch_size, num_heads, 0, head_dim)] for _ in range(2)]
    
    def __getitem__(self, idx):
        return self.data[idx]
    
    def get_seq_length(self):
        return 0

past_key_values = PastKeyValues()

inputs_embeds = torch.randn(batch_size, seq_len, hidden_dim)
text_labels = torch.ones(batch_size, seq_len, dtype=torch.bool)
context_labels = torch.ones(batch_size, seq_len, dtype=torch.bool)
labels = torch.ones(batch_size, seq_len, dtype=torch.bool)

# Build ARGS dictionary
ARGS = {
    'self': model,
    'input_ids': input_ids,
    'attention_mask': attention_mask,
    'position_ids': position_ids,
    'past_key_values': past_key_values,
    'inputs_embeds': None,
    'text_labels': text_labels,
    'context_labels': context_labels,
    'labels': labels,
    'use_cache': True,
    'output_attentions': False,
    'output_hidden_states': False,
    'return_dict': True,
    'cache_position': cache_position,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch, transformers — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type CompressModelOutputWithPast — a CompressModelOutputWithPast output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a CompressModelOutputWithPast output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_chunk Ran
ran; our contract draft was wrong, not the code
dvlab-research/Lyra/lyra/eval/model_lyra_image_speech.py
code served (permissive licence) · get_code("42a46570620cd9fa")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned list, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import math

def split_list(lst, n):
    """Split a list into n (roughly) equal-sized chunks"""
    chunk_size = math.ceil(len(lst) / n)
    return [lst[i:i + chunk_size] for i in range(0, len(lst), chunk_size)]

def get_chunk(lst, n, k):
    chunks = split_list(lst, n)
    return chunks[k]

# Setup code
lst = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
n = 3
k = 1

ARGS = {
    "lst": lst,
    "n": n,
    "k": k
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family int — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
img_to_base64 Ran
ran; our contract draft was wrong, not the code
dvlab-research/Lyra/lyra/conversation.py
code served (permissive licence) · get_code("f06df1307d1b0d63")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import base64
import tempfile
import os

# Create a temporary image file for testing
temp_dir = tempfile.gettempdir()
test_image_path = os.path.join(temp_dir, "test_image.png")

# Create a minimal valid PNG file (1x1 pixel, transparent)
png_data = base64.b64decode(
    "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
)
with open(test_image_path, 'wb') as f:
    f.write(png_data)

ARGS = {
    "img_file_path": test_image_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
initialize_input_ids_for_generation Ran
ran, honoured the contract we drafted
dvlab-research/Lyra/lyra/model/builder.py
code served (permissive licence) · get_code("91cfc0ec83571808")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family int, shape (2, 3) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Create a minimal config class
class MinimalConfig:
    def __init__(self, is_encoder_decoder=False):
        self.is_encoder_decoder = is_encoder_decoder

# Create a minimal self object (model-like)
class MinimalModel(nn.Module):
    def __init__(self, is_encoder_decoder=False):
        super().__init__()
        self.config = MinimalConfig(is_encoder_decoder=is_encoder_decoder)
        self.device = torch.device('cpu')

# Create encoder outputs stand-in class
class EncoderOutputs:
    def __init__(self):
        self.last_hidden_state = torch.randn(2, 5, 8)  # batch_size=2, seq_len=5, hidden_dim=8

# Test case 1: inputs provided (returns early)
model1 = MinimalModel(is_encoder_decoder=False)
inputs1 = torch.tensor([[1, 2, 3], [4, 5, 6]], dtype=torch.long)

ARGS = {
    'self': model1,
    'inputs': inputs1,
    'bos_token_id': torch.tensor(50257, dtype=torch.long),
    'model_kwargs': {'hidden_states': torch.randn(2, 4, 8)}
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family int, shape (2, 3) — shape (2, 3) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
mov_to_base64 Ran
ran; our contract draft was wrong, not the code
dvlab-research/Lyra/lyra/conversation.py
code served (permissive licence) · get_code("34782af36cf765e7")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import base64
import tempfile
import os

# Create a temporary MOV file with some binary data
temp_dir = tempfile.gettempdir()
temp_mov_path = os.path.join(temp_dir, "test_video.mov")

# Write some binary data to simulate a MOV file
with open(temp_mov_path, 'wb') as f:
    f.write(b'\x00\x00\x00\x20ftypisom' + b'\x00' * 100)

ARGS = {
    'mov_file_path': temp_mov_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
save_result Ran
ran; our contract draft was wrong, not the code
dvlab-research/Lyra/lyra/eval/model_lyra_text_speech.py
code served (permissive licence) · get_code("7719de171ef38b85")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
import os

# Create a temporary directory for the result file
temp_dir = tempfile.mkdtemp()
result_file = os.path.join(temp_dir, 'result.json')

# Construct the result list - a list of dicts, each with a 'speech' key
result = [
    {'speech': 'hello world', 'other_field': 'value1', 'id': 1},
    {'speech': 'hello world', 'other_field': 'value2', 'id': 2},  # duplicate speech
    {'speech': 'goodbye world', 'other_field': 'value3', 'id': 3},
    {'speech': 'hi there', 'other_field': 'value4', 'id': 4},
]

ARGS = {
    'result': result,
    'result_file': result_file,
    'remove_duplicate': 'speech'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
split_list Ran
ran; our contract draft was wrong, not the code
dvlab-research/Lyra/lyra/eval/model_lyra_image_speech.py
code served (permissive licence) · get_code("076c252c52cbb161")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned list. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import math

def split_list(lst, n):
    """Split a list into n (roughly) equal-sized chunks"""
    chunk_size = math.ceil(len(lst) / n)
    return [lst[i:i + chunk_size] for i in range(0, len(lst), chunk_size)]

# Setup code
lst = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
n = 3

ARGS = {
    "lst": lst,
    "n": n
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list — a list output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
timed_run Ran
ran; our contract draft was wrong, not the code
dvlab-research/Lyra/lyra/model/soft_dtw_cuda.py
code served (permissive licence) · get_code("89a318a54beb022e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

class MockSDTW(nn.Module):
    """Mock Soft-DTW module that takes two tensors and returns a scalar loss."""
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 1)
    
    def forward(self, a, b):
        # a and b are expected to be tensors
        # Compute a simple differentiable output
        combined = torch.cat([a.mean(dim=1), b.mean(dim=1)], dim=0)
        output = self.linear(combined).sum()
        return output

# Create sample tensors: batch_size=2, sequence_length=3, feature_dim=4
a = torch.randn(2, 3, 4, requires_grad=True)
b = torch.randn(2, 3, 4, requires_grad=False)

# Create the SDTW module
sdtw = MockSDTW()

ARGS = {
    'a': a,
    'b': b,
    'sdtw': sdtw
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
violates_moderation Ran
ran, violated the contract we drafted
dvlab-research/Lyra/lyra/utils.py
pointer only (licence: Apache-2.0); the stored copy get_code serves, from Mi3-Lab/doScenes-VLM-Planning, is pointer only (licence: AGPL-3.0) · get_code("f9939a84b9a65279")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (Apache-2.0), but get_code serves the stored copy, first harvested from Mi3-Lab/doScenes-VLM-Planning, and AGPL-3.0, as GitHub detects it on that repository, is not in the permissive set Syntology serves inline. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

wav_to_base64 Ran
ran; our contract draft was wrong, not the code
dvlab-research/Lyra/lyra/conversation.py
code served (permissive licence) · get_code("dc99ac2d1a2f8f6b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import base64
import os
import tempfile

# Create a temporary WAV file for testing
temp_dir = tempfile.mkdtemp()
wav_file_path = os.path.join(temp_dir, "test_audio.wav")

# Write minimal WAV file data (RIFF header + minimal PCM data)
wav_data = (
    b'RIFF'
    b'\x24\x00\x00\x00'  # Chunk size
    b'WAVE'
    b'fmt '
    b'\x10\x00\x00\x00'  # Subchunk1 size (16)
    b'\x01\x00'  # Audio format (1 = PCM)
    b'\x01\x00'  # Number of channels (1)
    b'\x44\xac\x00\x00'  # Sample rate (44100)
    b'\x88\x58\x01\x00'  # Byte rate
    b'\x02\x00'  # Block align
    b'\x10\x00'  # Bits per sample (16)
    b'data'
    b'\x00\x00\x00\x00'  # Subchunk2 size (0)
)

with open(wav_file_path, 'wb') as f:
    f.write(wav_data)

ARGS = {
    'wav_file_path': wav_file_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
auto_pad_images Not yet run
unverified (harvested, no recorded run)
dvlab-research/Lyra/lyra/mm_utils.py
pointer only (licence: Apache-2.0); the stored copy get_code serves, from HumanMLLM/LLaVA-Scissor, is pointer only (licence: NONE) · get_code("3c8823948724e269")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (Apache-2.0), but get_code serves the stored copy, first harvested from HumanMLLM/LLaVA-Scissor, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

build_unit_tokenizer Not yet run
unverified (harvested, no recorded run)
dvlab-research/Lyra/lyra/serve/model_worker.py
code served (permissive licence) · get_code("23e91c5c56d7ff80")
How it ran

no invocation recorded.

forward_attn Not yet run
unverified (harvested, no recorded run)
dvlab-research/Lyra/lyra/model/qwen2vl_top_attn.py
code served (permissive licence) · get_code("68283f6ff784cc32")
How it ran

no invocation recorded.

pretty_print_semaphore Not yet run
unverified (harvested, no recorded run)
dvlab-research/Lyra/lyra/utils.py
code served (permissive licence) · get_code("37899f22fb191b37")
How it ran

no invocation recorded.

resize_and_center_crop Not yet run
unverified (harvested, no recorded run)
dvlab-research/Lyra/lyra/mm_utils.py
pointer only (licence: Apache-2.0); the stored copy get_code serves, from HumanMLLM/LLaVA-Scissor, is pointer only (licence: NONE) · get_code("c80358dff96c25ce")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (Apache-2.0), but get_code serves the stored copy, first harvested from HumanMLLM/LLaVA-Scissor, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

Also present in other papers' code

Of 19 samples lifted from this paper's linked repositories, 19 carry a function identity and 0 do not. Of the 19 distinct functions with an identity, 9 are also present in other papers whose code we hold (8 with at least one paper dated earlier, 7 the same month, 8 later) and 10 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
split_list 197 other papers
133 dated earlier, 11 the same month, 46 later, 6 same year, order not determinable, 1 undated
2112.05682 (2021-12)
Self-attention Does Not Need $O(n^2)$ Memory
2306.17107 (2023-06)
LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding
2307.10490 (2023-07)
Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs
and 194 more not named here
get_chunk 189 other papers
128 dated earlier, 11 the same month, 44 later, 6 same year, order not determinable
2112.05682 (2021-12)
Self-attention Does Not Need $O(n^2)$ Memory
2306.17107 (2023-06)
LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding
2307.10490 (2023-07)
Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs
and 186 more not named here
violates_moderation 56 other papers
45 dated earlier, 1 the same month, 9 later, 1 same year, order not determinable
2106.06909 (2021-06)
GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio
2308.03279 (2023-08)
UniversalNER: TARGETED DISTILLATION FROM LARGE LANGUAGE MODELS FOR OPEN NAMED ENTITY RECOGNITION
2308.13437 (2023-08)
Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models
and 53 more not named here
pretty_print_semaphore 54 other papers
42 dated earlier, 1 the same month, 10 later, 1 same year, order not determinable
2106.06909 (2021-06)
GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio
2308.13437 (2023-08)
Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models
2309.07120 (2023-09)
Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics
and 51 more not named here
extract_patches 20 other papers
4 dated earlier, 4 the same month, 12 later
2411.10332 (2024-11)
Number it: Temporal Grounding Videos like Flipping Manga
2411.11360 (2024-11)
CCExpert: Advancing MLLM Capability in Remote Sensing Change Captioning with Difference-Aware Integration and a Foundational Dataset
2411.14432 (2024-11)
Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models
and 17 more not named here
auto_pad_images 19 other papers
4 dated earlier, 3 the same month, 12 later
2411.10332 (2024-11)
Number it: Temporal Grounding Videos like Flipping Manga
2411.11360 (2024-11)
CCExpert: Advancing MLLM Capability in Remote Sensing Change Captioning with Difference-Aware Integration and a Foundational Dataset
2411.14432 (2024-11)
Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models
and 16 more not named here
resize_and_center_crop 18 other papers
4 dated earlier, 3 the same month, 11 later
2411.10332 (2024-11)
Number it: Temporal Grounding Videos like Flipping Manga
2411.11360 (2024-11)
CCExpert: Advancing MLLM Capability in Remote Sensing Change Captioning with Difference-Aware Integration and a Foundational Dataset
2411.14432 (2024-11)
Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models
and 15 more not named here
timed_run 3 other papers
3 dated earlier
2306.02693 (2023-06)
CELDA: Leveraging Black-box Language Model as Enhanced Classifier without Labels
2404.02394 (2024-04)
Cohort-Individual Cooperative Learning for Multimodal Cancer Survival Analysis
2407.12200 (2024-07)
This Probably Looks Exactly Like That: An Invertible Prototypical Network
build_unit_tokenizer 1 other paper
1 later
2502.04328 (2025-02)
Ola: Pushing the Frontiers of Omni-Modal Language Model

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

As Multi-modal Large Language Models (MLLMs) evolve, expanding beyond single-domain capabilities is essential to meet the demands for more versatile and efficient AI. However, previous omni-models have insufficiently explored speech, neglecting its integration with multi-modality. We introduce Lyra, an efficient MLLM that enhances multimodal abilities, including advanced long-speech comprehension, sound understanding, cross-modality efficiency, and seamless speech interaction. To achieve efficiency and speech-centric capabilities, Lyra employs three strategies: (1) leveraging existing open-source large models and a proposed multi-modality LoRA to reduce training costs and data requirements; (2) using a latent multi-modality regularizer and extractor to strengthen the relationship between speech and other modalities, thereby enhancing model performance; and (3) constructing a high-quality, extensive dataset that includes 1.5M multi-modal (language, vision, audio) data samples and 12K long speech samples, enabling Lyra to handle complex long speech inputs and achieve more robust omni-cognition. Compared to other omni-methods, Lyra achieves state-of-the-art performance on various vision-language, vision-speech, and speech-language benchmarks, while also using fewer computational resources and less training data.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2412.09501")
get_code_for_paper("2412.09501")
have("2412.09501")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2412.09501.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2412.09501.svg)](https://syntology.ai/paper/2412.09501)

Connect an agent — have() is free.