SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2602.11509 · ICML · 2026

Multimodal Fact-Level Attribution for Verifiable Reasoning

Mohit Bansal, Elias Stengel-Eskin, Han Wang, Hyunji Lee, Ziyang Wang, David Wan

arXiv · PDF · Open in the Atlas

Code that ran

10 of 10 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 0 honoured the contract we drafted, 0 violated it and 0 have no contract verdict; 10 failures of our instrument; 0 no recorded run), measured 2026-11-22; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2602.11509.json · badge: /api/ran/2602.11509.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
meetdavidwan/murgat canonical (classifier label)
classifier basis: census
10 ran · 0 no recorded run
FunctionStatusWhere it lives
adapt_messages_to_openai Ran
ran; our contract draft was wrong, not the code
meetdavidwan/murgat/src/models.py
code served (permissive licence) · get_code("be9226ad1c9eb191")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile
from pathlib import Path
from PIL import Image
import io

# Create a temporary directory for test images
temp_dir = tempfile.mkdtemp()

# Create a simple test image and save it
def create_test_image(filepath):
    img = Image.new('RGB', (10, 10), color='red')
    img.save(filepath, format='JPEG')

# Create test image files
image_path_1 = os.path.join(temp_dir, 'test_image_1.jpg')
create_test_image(image_path_1)

image_path_2 = os.path.join(temp_dir, 'test_image_2.jpg')
create_test_image(image_path_2)

# Define test messages with various content types
messages = [
    {
        'role': 'user',
        'content': 'Hello, this is a simple text message.'
    },
    {
        'role': 'assistant',
        'content': [
            {'type': 'text', 'text': 'This is text in a list.'},
            {'type': 'image', 'image': image_path_1}
        ]
    },
    {
        'role': 'user',
        'content': [
            {'type': 'text', 'text': 'Tell me about this image.'},
            {'type': 'image', 'image': image_path_2}
        ]
    },
    {
        'role': 'user',
        'content': [
            {'type': 'video', 'video': 'test_video.mp4'},
            {'type': 'audio', 'audio': 'test_audio.mp3'},
            {'type': 'text', 'text': 'Can you process these files?'}
        ]
    },
    {
        'role': 'assistant',
        'content': [
            {'type': 'text', 'text': '  '},
            {'type': 'text', 'text': 'Response text'}
        ]
    }
]

ARGS = {
    'messages': messages
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
PIL — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
clean_model_output Ran
ran; our contract draft was wrong, not the code
meetdavidwan/murgat/src/util.py
code served (permissive licence) · get_code("62fd78c059d6e0d2")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

RE_THINKING_BLOCK = re.compile('<think>.*?</think>', re.DOTALL | re.IGNORECASE)

def clean_model_output(text: str) -> str:
    """
    Removes <think>...</think> blocks and whitespace from model output.
    
    Args:
        text: Raw model output text
        
    Returns:
        Cleaned text with thinking blocks removed
    """
    if not text:
        return ''
    cleaned_text = RE_THINKING_BLOCK.sub('', text)
    return cleaned_text.strip()


# Setup code
ARGS = {
    'text': '<think>This is internal reasoning</think>Here is the final answer.'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_answer Ran
ran; our contract draft was wrong, not the code
meetdavidwan/murgat/src/util.py
code served (permissive licence) · get_code("ca419c78d00ec1d5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def extract_answer(text: str) -> str:
    """
    Extracts answer from model text. 
    Prioritizes LaTeX \\boxed{...} content, then falls back to 'Answer:' markers.
    
    Args:
        text: Full model response text
        
    Returns:
        Extracted answer text
    """
    boxed_pattern = '\\\\boxed\\s*\\{(.*?)\\}'
    boxed_matches = re.findall(boxed_pattern, text)
    if boxed_matches:
        return boxed_matches[-1].strip()
    pattern = 'Answer:\\s*(.*?)(?:\\n|$)'
    match = re.search(pattern, text, re.DOTALL | re.IGNORECASE)
    if match:
        return match.group(1).strip()
    if 'Answer:' in text or 'answer:' in text:
        parts = re.split('Answer:\\s*', text, flags=re.IGNORECASE)
        if len(parts) > 1:
            return parts[-1].strip()
    return ''


# Test case 1: text with \boxed{...}
test_text_1 = "The solution is \\boxed{42}"

# Test case 2: text with Answer: marker
test_text_2 = "Working through the problem...\nAnswer: The answer is 7"

# Test case 3: text with answer: (lowercase)
test_text_3 = "Let me solve this.\nanswer: correct response"

# Test case 4: empty or no answer markers
test_text_4 = "Just some random text with no answer"

ARGS = {
    "text": test_text_1
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_reasoning Ran
ran; our contract draft was wrong, not the code
meetdavidwan/murgat/src/util.py
code served (permissive licence) · get_code("c73b835b7cc1c78c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def extract_reasoning(text: str) -> str:
    """Extracts text between 'Reasoning:' and 'Answer:' markers."""
    if not text:
        return ''
    start_match = re.search('(?:^|\\n|\\*)\\s*Reasoning\\s*[:*]*\\s*', text, re.IGNORECASE)
    if start_match is None:
        return text
    content = text[start_match.end():]
    end_pattern = '(?:\\n|^)\\s*Answer\\s*[:\\s]'
    end_match = re.search(end_pattern, content, re.IGNORECASE)
    if end_match is None:
        return text
    if end_match:
        content = content[:end_match.start()]
    return content.strip()


ARGS = {
    "text": "Some preamble\nReasoning: This is the reasoning section with multiple lines\nof reasoning text that explains the logic\nAnswer: This is the answer"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
format_sentence_prompt Ran
ran; our contract draft was wrong, not the code
meetdavidwan/murgat/src/run_metric.py
code served (permissive licence) · get_code("8915161a27a1d8d4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Analyzing the function:
# - template: str with potential placeholder tags like {sent}, {sentence}, {target_sentence}, {text}, {context}, {full_paragraph}, {full_text}
# - target_sentence: str to replace sentence-related tags
# - full_context: str (default='') to replace context-related tags

ARGS = {
    'template': 'The sentence is: {sent}. Context: {context}',
    'target_sentence': 'This is a test sentence.',
    'full_context': 'This is the full context paragraph with more information.'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_unique_key Ran
ran; our contract draft was wrong, not the code
meetdavidwan/murgat/src/run_baseline.py
code served (permissive licence) · get_code("bbc3dbff8b3c7095")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Create a sample item dict with 'video' and 'question' keys
item = {
    'video': 'video_001.mp4',
    'question': 'What is happening in this scene?'
}

ARGS = {
    'item': item
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_prompt Ran
ran; our contract draft was wrong, not the code
meetdavidwan/murgat/src/program_utils.py
code served (permissive licence) · get_code("f8d44f2e394301ff")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile
import shutil

# Create a temporary directory structure with a prompts folder
temp_dir = tempfile.mkdtemp()
prompts_dir = os.path.join(temp_dir, 'prompts')
os.makedirs(prompts_dir, exist_ok=True)

# Create a sample prompt file
sample_prompt_path = os.path.join(prompts_dir, 'test_prompt.txt')
with open(sample_prompt_path, 'w') as f:
    f.write('This is a test prompt.')

# Change to the temporary directory so the relative path works
original_dir = os.getcwd()
os.chdir(temp_dir)

try:
    ARGS = {
        'filename': 'test_prompt.txt'
    }
finally:
    # Note: In a real scenario, you'd want to clean up, but for this setup
    # we'll leave it for the function to use
    pass
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_prompt_text Ran
ran; our contract draft was wrong, not the code
meetdavidwan/murgat/src/run_metric.py
code served (permissive licence) · get_code("99ba1590f3b6d46a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile

# Create a temporary file with some prompt text
temp_dir = tempfile.mkdtemp()
temp_file = os.path.join(temp_dir, "prompt.txt")
with open(temp_file, 'w') as f:
    f.write("This is a sample prompt text.")

ARGS = {
    "path": temp_file
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
retry_with_backoff Ran
ran; our contract draft was wrong, not the code
meetdavidwan/murgat/src/program_utils.py
code served (permissive licence) · get_code("ac50d2af5d76c846")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned function. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import time
import functools

def retry_with_backoff(retries=3, backoff_in_seconds=5):
    """Decorator to retry functions (like API calls) on failure."""

    def decorator(func):

        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            x = 0
            while True:
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    msg = str(e).lower()
                    if x == retries:
                        raise e
                    if '503' in msg or 'overloaded' in msg or 'rate limit' in msg:
                        sleep_time = backoff_in_seconds * 2 ** x
                        print(f'  [API Warn] Model overloaded. Retrying in {sleep_time}s...')
                        time.sleep(sleep_time)
                        x += 1
                    else:
                        raise e
        return wrapper
    return decorator


# Test function that we'll decorate
call_count = 0

def flaky_api_call():
    global call_count
    call_count += 1
    if call_count < 2:
        raise Exception("503 Service Unavailable")
    return "success"


# Apply decorator with test arguments
decorated_func = retry_with_backoff(retries=3, backoff_in_seconds=0.01)(flaky_api_call)

# Call the decorated function to verify it works
result = decorated_func()

ARGS = {
    "retries": 3,
    "backoff_in_seconds": 5
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type function — a function output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a function output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
safe_generate Ran
ran; our contract draft was wrong, not the code
meetdavidwan/murgat/src/program_utils.py
code served (permissive licence) · get_code("e0500984ed50d79a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import time

torch.manual_seed(0)

class MockModel(nn.Module):
    """Minimal model with generate_response method."""
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 3)
    
    def generate_response(self, messages, num_samples=1):
        """Mock generate_response that returns a list of responses."""
        return [f"Response {i}" for i in range(num_samples)]

model = MockModel()

messages = [
    {"role": "user", "content": "Hello"},
    {"role": "assistant", "content": "Hi there"}
]

num_samples = 1

ARGS = {
    "model": model,
    "messages": messages,
    "num_samples": num_samples
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block

Also present in other papers' code

Of 10 samples lifted from this paper's linked repositories, 10 carry a function identity and 0 do not. Of the 10 distinct functions with an identity, 0 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 0 later) and 10 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Multimodal large language models (MLLMs) are increasingly used for real-world tasks involving multi-step reasoning and long-form generation, where reliability requires grounding model outputs in heterogeneous input sources and verifying individual factual claims. However, existing multimodal grounding benchmarks and evaluation methods focus on simplified, observationbased scenarios or limited modalities and fail to assess attribution in complex multimodal reasoning. We introduce MURGAT (Multimodal Reasoning with Grounded Attribution), a benchmark for evaluating fact-level multimodal attribution in settings that require reasoning beyond direct observation. Given inputs spanning video, audio, and other modalities, MURGAT requires models to generate answers with explicit reasoning and precise citations, where each citation specifies both modality and temporal segments. To enable reliable assessment, we introduce an automatic evaluation framework that strongly correlates with human judgments. Benchmarking with human and automated scores reveals that even strong MLLMs frequently hallucinate citations despite correct reasoning. Moreover, we observe a key trade-off: increasing reasoning depth or enforcing structured grounding often degrades accuracy, highlighting a significant gap between internal reasoning and verifiable attribution. Code and data are available at https: //github.com/meetdavidwan/murgat.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2602.11509")
get_code_for_paper("2602.11509")
have("2602.11509")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2602.11509.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2602.11509.svg)](https://syntology.ai/paper/2602.11509)

Connect an agent — have() is free.