SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2602.22278 · 2026

RETLLM: Training and Data-Free MLLMs for Multimodal Information Retrieval

Dongsheng Wang, Dawei Su

arXiv · PDF · Open in the Atlas

Code that ran

9 of 14 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 2 honoured the contract we drafted, 0 violated it and 0 have no contract verdict; 7 failures of our instrument; 5 no recorded run), measured 2026-11-22; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2602.22278.json · badge: /api/ran/2602.22278.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
alivecat05/RETLLM canonical (classifier label)
classifier basis: census
9 ran · 5 no recorded run
FunctionStatusWhere it lives
calculate_recall_from_results Ran
ran, honoured the contract we drafted
alivecat05/RETLLM/Codes/main_retrieval_long_module.py
code served (permissive licence) · get_code("de6a889b9a238540")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Construct results: a list of dicts with 'is_correct' key
results = [
    {'is_correct': True},
    {'is_correct': False},
    {'is_correct': True},
    {'is_correct': False},
    {'is_correct': True},
]

# retrieval_type parameter
retrieval_type = "bm25"

ARGS = {
    "results": results,
    "retrieval_type": retrieval_type,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
compute_entropy Ran
ran, honoured the contract we drafted
alivecat05/RETLLM/Codes/Qwen_vl.py
code served (permissive licence) · get_code("c98c3c7150fdcd2f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float64, ndim 0, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# The function compute_entropy takes a single parameter 'probs'
# which should be array-like (list or numpy array) of probability values
# The function expects non-negative values that sum to 1 (or close to it)

# Create a valid probability distribution
probs_value = [0.2, 0.3, 0.25, 0.15, 0.1]  # Valid probability distribution

ARGS = {
    "probs": probs_value
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float64, ndim 0, dtype family float — no output shape is stated: observe_signature.py recorded float ndim 0, which disagrees with the fixture run
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_numbers Ran
ran; our contract draft was wrong, not the code
alivecat05/RETLLM/Codes/Qwen2_vl.py
code served (permissive licence) · get_code("22d6dd835992f7a4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def extract_numbers(string):
    numbers = re.findall('\\d+\\.\\d+|\\d+', string)
    return numbers

ARGS = {
    "string": "The price is 19.99 and quantity is 42 with ratio 3.14159"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_featuremap_size Ran
ran; our contract draft was wrong, not the code
alivecat05/RETLLM/Codes/dino_heal.py
code served (permissive licence) · get_code("ccce81326befa5c1")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

from PIL import Image

# Create a small stand-in image object with a .size attribute
# The function accesses img.size[0] and img.size[1] for width and height
img = Image.new('RGB', (256, 256))

ARGS = {
    'img': img
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
PIL — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family int — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_vision_attention_map Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
alivecat05/RETLLM/Codes/attention_demo.py
code served (permissive licence) · get_code("15f2ce30d8f6af86")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Create a small attention weights tensor
# Shape: (batch_size, num_tokens, num_vision_tokens)
# batch_size=1, num_tokens=10 (includes output tokens), num_vision_tokens=289 (17x17 grid)
attention_weights = torch.randn(1, 10, 289)

# Parameters for the function
vision_start = 0
vision_end = 289
H_theoretical = 17
W_theoretical = 17
actual_tokens = 289
output_tokens_start = 5

ARGS = {
    "attention_weights": attention_weights,
    "vision_start": vision_start,
    "vision_end": vision_end,
    "H_theoretical": H_theoretical,
    "W_theoretical": W_theoretical,
    "actual_tokens": actual_tokens,
    "output_tokens_start": output_tokens_start
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_vision_attention_map Ran
ran; our contract draft was wrong, not the code
alivecat05/RETLLM/Codes/attention_demo_fixed.py
code served (permissive licence) · get_code("ebb035364fe02208")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Create attention weights tensor
# Shape: [batch_size, num_heads, seq_len, seq_len]
# We need seq_len to be large enough to cover vision_start and vision_end
batch_size = 1
num_heads = 2
seq_len = 100  # Large enough to include vision tokens
attention_weights = torch.randn(batch_size, num_heads, seq_len, seq_len)

# Define vision token range
vision_start = 10
vision_end = 90  # This gives us 80 vision tokens

# Theoretical dimensions
H_theoretical = 8
W_theoretical = 10

# Actual tokens
actual_tokens = 80

# Output token start position (optional)
output_tokens_start = 95

ARGS = {
    "attention_weights": attention_weights,
    "vision_start": vision_start,
    "vision_end": vision_end,
    "H_theoretical": H_theoretical,
    "W_theoretical": W_theoretical,
    "actual_tokens": actual_tokens,
    "output_tokens_start": output_tokens_start,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
recall_at_k Ran
ran; our contract draft was wrong, not the code
alivecat05/RETLLM/Codes/main_retrieval_long_module.py
code served (permissive licence) · get_code("d413d290d3c00325")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned Tensor, ndim 1, dtype family float, shape (2,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Create small tensors for testing
nb_texts = 2
nb_images = 4

# scores: compatibility scores between texts and images (nb_texts, nb_images)
scores = torch.randn(nb_texts, nb_images)

# positive_pairs: boolean matrix indicating which text-image pairs are positive (nb_texts, nb_images)
positive_pairs = torch.tensor([
    [True, False, True, False],
    [False, True, False, True]
], dtype=torch.bool)

# k: number of top images to consider for each text
k = 2

ARGS = {
    "scores": scores,
    "positive_pairs": positive_pairs,
    "k": k
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 1, dtype family float, shape (2,) — shape (2,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
rotate_half Ran
ran; our contract draft was wrong, not the code
alivecat05/RETLLM/modeling_qwen2_5_vl.py
code served (permissive licence) · get_code("e03d53ba9d4f9ae5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function takes a single parameter 'x' which is a tensor
# It accesses x[..., :x.shape[-1] // 2] and x[..., x.shape[-1] // 2:]
# So x needs to be a tensor with a last dimension that can be split in half
# Let's create a small tensor with batch size 2 and feature dimension 4

x = torch.randn(2, 4)

ARGS = {
    "x": x
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
text_retrieval Ran
ran; our contract draft was wrong, not the code
alivecat05/RETLLM/main_eval.py
code served (permissive licence) · get_code("75359491f96c06bb")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from PIL import Image
import numpy as np

torch.manual_seed(0)

class MockTokenizer:
    def __call__(self, text):
        # Return a mock token tensor
        return torch.randint(0, 100, (1, 8))
    
    def to(self, device):
        return self

class MockModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.text_encoder = nn.Linear(8, 4)
        self.image_encoder = nn.Linear(3, 4)
    
    def encode_text(self, tokens):
        # tokens shape: (1, 8)
        # output shape: (1, 4)
        batch_size = tokens.shape[0]
        return torch.randn(batch_size, 4)
    
    def encode_image(self, image):
        # image shape: (1, 3, 2, 2) after transform
        batch_size = image.shape[0]
        return torch.randn(batch_size, 4)

class MockTransform:
    def __call__(self, image):
        # Convert PIL image to tensor (1, 3, 2, 2)
        img_array = np.array(image)
        if len(img_array.shape) == 2:
            img_array = np.stack([img_array] * 3, axis=-1)
        img_tensor = torch.from_numpy(img_array).float()
        # Reshape to (3, 2, 2)
        img_tensor = img_tensor.permute(2, 0, 1)
        return img_tensor

# Create dummy image
dummy_image = Image.new('RGB', (2, 2), color='red')

ARGS = {
    'pos_text': 'a positive example',
    'neg_text': 'a negative example',
    'image': dummy_image,
    'model': MockModel(),
    'tokenizer': MockTokenizer(),
    'transform': MockTransform(),
    'device': torch.device('cpu')
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
PIL, numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
apply_rotary_pos_emb_flashatt Not yet run
unverified (harvested, no recorded run)
alivecat05/RETLLM/modeling_qwen2_5_vl.py
code served (permissive licence) · get_code("24acaeb8ad82253e")
How it ran

no invocation recorded.

apply_rotary_pos_emb_vision Not yet run
unverified (harvested, no recorded run)
alivecat05/RETLLM/modeling_qwen2_5_vl.py
code served (permissive licence) · get_code("09445ca10af72017")
How it ran

no invocation recorded.

batchify Not yet run
unverified (harvested, no recorded run)
alivecat05/RETLLM/Codes/main_retrieval_long_module.py
code served (permissive licence) · get_code("4d3f99af6174569e")
How it ran

no invocation recorded.

calculate_entropy Not yet run
unverified (harvested, no recorded run)
alivecat05/RETLLM/Codes/Qwen_vl.py
code served (permissive licence) · get_code("76755f6879d57a39")
How it ran

no invocation recorded.

load_model Not yet run
unverified (harvested, no recorded run)
alivecat05/RETLLM/main_eval.py
code served (permissive licence) · get_code("3e4ea555a8a80225")
How it ran

no invocation recorded.

Also present in other papers' code

Of 14 samples lifted from this paper's linked repositories, 14 carry a function identity and 0 do not. Of the 14 distinct functions with an identity, 5 are also present in other papers whose code we hold (5 with at least one paper dated earlier, 3 the same month, 3 later) and 9 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
rotate_half 360 other papers
324 dated earlier, 6 the same month, 27 later, 3 undated · 30 only as an assembly brick
1901.02860 (2019-01)
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
2203.08913 (2022-03)
Memorizing Transformers
2302.13971 (2023-02)
LLaMA: Open and Efficient Foundation Language Models
and 357 more not named here
apply_rotary_pos_emb_flashatt 6 other papers
4 dated earlier, 1 the same month, 1 later
2503.11187 (2025-03)
FastVID: Dynamic Density Pruning for Fast Video Large Language Models
2506.15220 (2025-06)
video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Models
2507.01949 (2025-07)
Kwai Keye-VL Technical Report
and 3 more not named here
apply_rotary_pos_emb_vision 4 other papers
2 dated earlier, 1 the same month, 1 later
2503.11187 (2025-03)
FastVID: Dynamic Density Pruning for Fast Video Large Language Models
2509.01986 (2025-09)
Draw-In-Mind: Rebalancing Designer-Painter Roles in Unified Multimodal Models Benefits Image Editing
2602.01173 (2026-02)
EEmo-Logic: A Unified Dataset and Multi-Stage Framework for Comprehensive Image-Evoked Emotion Assessment
and 1 more not named here
batchify 3 other papers
3 dated earlier
2407.12580 (2024-07)
E5-V: Universal Embeddings with Multimodal Large Language Models
2503.19900 (2025-03)
CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuning
2601.06891 (2026-01)
Contrastive Language-Image Mamba Pretraining
recall_at_k 2 other papers
2 dated earlier
2407.12580 (2024-07)
E5-V: Universal Embeddings with Multimodal Large Language Models
2503.19900 (2025-03)
CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuning

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Multimodal information retrieval (MMIR) has gained attention for its flexibility in handling text, images, or mixed queries and candidates. Recent breakthroughs in multimodal large language models (MLLMs) boost MMIR performance by incorporating MLLM knowledge under the contrastive finetuning framework. However, they suffer from pre-training inconsistency and require large datasets. In this work, we introduce a novel framework, RetLLM, designed to query MLLMs for MMIR in a training-and data-free manner. Specifically, we formulate MMIR as a similarity score generation task and prompt MLLMs to directly predict retrieval scores in a coarse-then-fine pipeline. At the coarse stage, a top-k filtering strategy builds a small yet high-quality candidate pool for each query, enabling MLLMs to focus on semantically relevant candidates. Subsequently, the retrieval score is predicted by feeding both the query and candidate into MLLMs at the fine stage. Importantly, we propose a visual enhancement module during reasoning to help MLLMs re-pick forgotten visuals, improving retrieval. Extensive experiments on MMIR benchmarks show that RetLLM outperforms fine-tuned models. Ablation studies further verify each component. Our work demonstrates that MLLMs can achieve strong MMIR performance without any training, highlighting their inherent multimodal reasoning ability in a simple, scalable framework. We release our code at: https://github.com/alivecat05/RETLLM

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2602.22278")
get_code_for_paper("2602.22278")
have("2602.22278")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2602.22278.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2602.22278.svg)](https://syntology.ai/paper/2602.22278)

Connect an agent — have() is free.