SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2604.22239 · 2026

Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QA

Ping Luo, Yixuan Cao, Lvzhou Luo, Zhanli Li

arXiv · PDF · Open in the Atlas

Code that ran

21 of 26 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (5 of them constructed an object rather than computing a result; 0 honoured the contract we drafted, 2 violated it and 6 have no contract verdict; 13 failures of our instrument; 5 no recorded run), measured 2026-11-22; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2604.22239.json · badge: /api/ran/2604.22239.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
Zhanli-Li/MuDABench community
Papers with Code archive
21 ran · 5 no recorded run
FunctionStatusWhere it lives
DocQARecord Ran
ran on a synthesized input · constructed an object; no result was computed
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("58cd26d69bca691d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, DocQARecord: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional

torch.manual_seed(0)

@dataclass
class DocQARecord:
    plan_id: str
    doc_id: str
    attempts: List[Dict[str, Any]] = field(default_factory=list)

ARGS = {
    "plan_id": "plan_12345",
    "doc_id": "doc_67890",
    "attempts": [
        {"query": "What is this?", "answer": "This is a test", "score": 0.95},
        {"query": "Where is it?", "answer": "It is here", "score": 0.87}
    ]
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DocQARecord — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a DocQARecord output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
ExtractorRegistry Ran
ran on a synthesized input
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("39eaaef56c224774")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned NoneType. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

from typing import Any, Dict, List, Optional, Protocol
import importlib
from typing import Any, Callable, Dict, Optional

SingleDocExtractor = Callable[[str, str], str]

class ExtractorRegistry:
    """
    Registry for pluggable single-document extraction backends.

    Core contract:
    - input: document_id (str), prompt (str)
    - output: extracted answer text (str)

    This makes `ask_bchatdoc.py` replaceable by any single-doc extractor system.
    """
    _registry: Dict[str, SingleDocExtractor] = {}

    @classmethod
    def register(cls, name: str, func: SingleDocExtractor) -> None:
        cls._registry[name] = func

    @classmethod
    def get(cls, name: str) -> SingleDocExtractor:
        if name not in cls._registry:
            raise KeyError(f'unknown extractor backend: {name}')
        return cls._registry[name]

    @classmethod
    def load_from_entrypoint(cls, spec: str) -> SingleDocExtractor:
        """
        `spec` format: module.submodule:function_name
        Example: my_backend.chat:extract_single_doc
        """
        if ':' not in spec:
            raise ValueError('entrypoint must be module_path:function_name')
        module_path, func_name = spec.split(':', 1)
        module = importlib.import_module(module_path)
        func = getattr(module, func_name)
        if not callable(func):
            raise TypeError(f'entrypoint {spec} is not callable')
        return func


# Setup code
INIT_ARGS = {}

ARGS = {
    'name': 'test_extractor',
    'func': lambda doc_id, prompt: f'extracted from {doc_id}: {prompt[:20]}'
}

METHOD = 'register'

# Construct and call
registry = ExtractorRegistry(**INIT_ARGS)
registry.register(**ARGS)

# Verify registration
retrieved = registry.get('test_extractor')
result = retrieved('doc_123', 'What is the meaning of life?')
print(f"Result: {result}")

# Test get method
ARGS_GET = {'name': 'test_extractor'}
METHOD_GET = 'get'
extractor = registry.get(**ARGS_GET)
print(f"Retrieved extractor: {extractor}")

# Test load_from_entrypoint with a valid spec
ARGS_LOAD = {'spec': 'os.path:join'}
METHOD_LOAD = 'load_from_entrypoint'
loaded_func = registry.load_from_entrypoint(**ARGS_LOAD)
print(f"Loaded function: {loaded_func}")
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type NoneType — a NoneType output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a NoneType output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
LLMResponse Ran
ran on a synthesized input · constructed an object; no result was computed
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("814a75b8ec1ca439")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, LLMResponse: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

ARGS = {
    "text": "This is a test response from the language model."
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type LLMResponse — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a LLMResponse output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
PromptTemplate Ran
ran on a synthesized input · constructed an object; no result was computed
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("7083ff639bbbaa35")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, PromptTemplate: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re
from typing import Any, Dict, List, Optional, Protocol

_PLACEHOLDER_RE = re.compile('\\{([a-zA-Z_][a-zA-Z0-9_]*)\\}')

def safe_template_format(template: str, values: Dict[str, Any]) -> str:

    def _replace(match: re.Match) -> str:
        key = match.group(1)
        return str(values[key]) if key in values else match.group(0)
    return _PLACEHOLDER_RE.sub(_replace, template)

class PromptTemplate:

    def __init__(self, template: str):
        self.template = template

    def format(self, **kwargs: Any) -> str:
        return safe_template_format(self.template, kwargs)


ARGS = {
    "template": "Hello {name}, you have {count} messages and {score} points."
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type PromptTemplate — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a PromptTemplate output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
RunLog Ran
ran on a synthesized input · constructed an object; no result was computed
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("8179670b62c5ef66")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, RunLog: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional

@dataclass
class DocQARecord:
    plan_id: str
    doc_id: str
    attempts: List[Dict[str, Any]] = field(default_factory=list)

@dataclass
class RunLog:
    run_id: str
    qid: str
    task: str
    metadata_description: str
    started_at: str
    finished_at: Optional[str] = None
    status: str = 'running'
    error: Optional[str] = None
    steps: List[Dict[str, Any]] = field(default_factory=list)
    doc_interactions: Dict[str, Dict[str, DocQARecord]] = field(default_factory=dict)
    artifacts: Dict[str, Any] = field(default_factory=dict)

torch.manual_seed(0)

# Construct sample DocQARecord instances
doc_qa_record_1 = DocQARecord(
    plan_id="plan_001",
    doc_id="doc_001",
    attempts=[
        {"attempt_num": 1, "result": "success", "score": 0.95},
        {"attempt_num": 2, "result": "failure", "score": 0.42}
    ]
)

doc_qa_record_2 = DocQARecord(
    plan_id="plan_002",
    doc_id="doc_002",
    attempts=[
        {"attempt_num": 1, "result": "success", "score": 0.87}
    ]
)

# Construct doc_interactions structure
doc_interactions = {
    "section_a": {
        "qa_001": doc_qa_record_1,
        "qa_002": doc_qa_record_2
    },
    "section_b": {
        "qa_003": DocQARecord(
            plan_id="plan_003",
            doc_id="doc_003",
            attempts=[]
        )
    }
}

# Construct steps list
steps = [
    {"step_id": 1, "action": "retrieve", "timestamp": "2024-01-01T10:00:00"},
    {"step_id": 2, "action": "process", "timestamp": "2024-01-01T10:00:05"},
    {"step_id": 3, "action": "validate", "timestamp": "2024-01-01T10:00:10"}
]

# Construct artifacts dict
artifacts = {
    "embedding_tensor": torch.randn(2, 4),
    "metadata": {"version": "1.0", "model": "gpt-4"},
    "scores": [0.91, 0.87, 0.73]
}

ARGS = {
    "run_id": "run_20240101_001",
    "qid": "question_123",
    "task": "document_qa",
    "metadata_description": "Question answering task on technical documentation",
    "started_at": "2024-01-01T10:00:00Z",
    "finished_at": "2024-01-01T10:05:32Z",
    "status": "completed",
    "error": None,
    "steps": steps,
    "doc_interactions": doc_interactions,
    "artifacts": artifacts
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type RunLog — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a RunLog output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
StructuredLogger Ran
ran on a synthesized input · constructed an object; no result was computed
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("dcf5d6dbc37366a4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, StructuredLogger: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional

# Create a temporary directory for the base_dir parameter
temp_dir = tempfile.mkdtemp()

ARGS = {
    'qid': 'query_12345',
    'task': 'document_qa_extraction',
    'metadata_description': 'Extracting Q&A from financial documents',
    'base_dir': temp_dir
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type StructuredLogger — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a StructuredLogger output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
_execute_python_code Ran
ran; our contract draft was wrong, not the code
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("067c2b0c9b8a1172")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import subprocess
import sys
import os

# The function _execute_python_code takes:
# - code: str - Python code to execute
# - timeout_sec: int - timeout in seconds (default 120)

# We'll create a simple valid Python code string to execute
simple_code = """
x = 42
print(x)
"""

ARGS = {
    'code': simple_code,
    'timeout_sec': 120
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
_guess_doc_id Ran
ran; our contract draft was wrong, not the code
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("14bc3048e73177d0")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np
from typing import Any, Dict, List, Optional

torch.manual_seed(0)

# Construct a valid meta dictionary for _guess_doc_id
# The function looks for keys: 'chatdoc_upload_id', 'doc_id', 'document_id', 'toBid', 'id'
# and returns the string value of the first one found that is not None

meta = {
    'chatdoc_upload_id': 'upload_123',
    'doc_id': 'doc_456',
    'document_id': 'document_789',
    'toBid': 'bid_000',
    'id': 'id_111',
    'other_field': 'some_value'
}

ARGS = {
    'meta': meta
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
_match_restriction_value Ran
ran, violated the contract we drafted
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("6422d17edb6287a6")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np
from typing import Any, Dict, List, Optional

torch.manual_seed(0)

# The function _match_restriction_value takes two parameters:
# 1. meta_value: Any - can be anything that can be converted to string
# 2. restriction_value: Any - can be a list/tuple/set or a single value

# Test case 1: meta_value as a string, restriction_value as a string
meta_value_1 = "test_mode"
restriction_value_1 = "test_mode"

# Test case 2: meta_value as an integer, restriction_value as a list
meta_value_2 = 42
restriction_value_2 = [42, 43, 44]

# Test case 3: meta_value as a float, restriction_value as a tuple
meta_value_3 = 3.14
restriction_value_3 = (2.71, 3.14, 1.41)

# Test case 4: meta_value as a tensor, restriction_value as a set
meta_value_4 = torch.tensor([1, 2, 3])
restriction_value_4 = {"tensor([1, 2, 3])", "other"}

# For the primary ARGS, use distinct test cases
# Using case 1: exact string match that should return True
ARGS = {
    "meta_value": "test_mode",
    "restriction_value": "test_mode"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
_normalize_for_hash Ran
ran; our contract draft was wrong, not the code
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("6383424fe927625d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np
from typing import Any, Dict, List

torch.manual_seed(0)

# Test cases for _normalize_for_hash
# The function recursively normalizes dicts, lists, and other values for hashing

ARGS = {
    "value": {
        "z_key": [1, 2, 3],
        "a_key": {"nested_z": 10, "nested_a": 20},
        "m_key": [{"inner_z": 5}, {"inner_a": 15}]
    }
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
_redact Ran
ran; our contract draft was wrong, not the code
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("aa9d78ea2f9ce79a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re
from typing import Any, Dict, List, Optional, Protocol

def _redact(x: Any) -> Any:
    if isinstance(x, str):
        x = re.sub('sk-[a-zA-Z0-9_-]{8,}', lambda m: m.group(0)[:6] + '***' + m.group(0)[-3:], x)
        x = re.sub('ak--[a-zA-Z0-9_-]{8,}', lambda m: m.group(0)[:6] + '***' + m.group(0)[-3:], x)
        return x
    if isinstance(x, dict):
        return {k: _redact(v) for k, v in x.items()}
    if isinstance(x, list):
        return [_redact(v) for v in x]
    return x


# Test cases covering different input types
test_string_with_sk = "my api key is sk-abcdefghijklmnop"
test_string_with_ak = "my access key is ak--abcdefghijklmnop"
test_dict = {
    "api_key": "sk-1234567890abcdef",
    "nested": {
        "access_key": "ak--zyxwvutsrqponml"
    }
}
test_list = [
    "sk-testkey1234567",
    "ak--testaccess12345",
    {"token": "sk-secrettoken123"}
]
test_integer = 42
test_float = 3.14
test_none = None
test_plain_string = "hello world"

ARGS = {
    "x": test_dict
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
_to_posix Ran
ran; our contract draft was wrong, not the code
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("c3dfe608b73c6159")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os

def _to_posix(p: str) -> str:
    return os.path.abspath(p).replace('\\', '/')

# The function takes a single string parameter `p` representing a file path
# It converts it to an absolute path and normalizes backslashes to forward slashes
# We need to provide a valid path string

ARGS = {
    'p': '/tmp/test/path'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
build_question_id Ran
ran; our contract draft was wrong, not the code
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("58682eacd4f958ef")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
from typing import Any, Dict, List, Optional

# Construct valid arguments for build_question_id

# The function takes:
# 1. question: Any - will be converted to string and stripped
# 2. metadata: List[Dict[str, Any]] - a list of dictionaries

ARGS = {
    'question': 'What is the capital of France?',
    'metadata': [
        {'source': 'wikipedia', 'year': 2024, 'confidence': 0.95},
        {'category': 'geography', 'difficulty': 'easy'},
        {'tags': ['europe', 'capitals'], 'verified': True}
    ]
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
clean_code Ran
ran; our contract draft was wrong, not the code
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("5b43b0d8ca375888")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import textwrap

# Analyze the function: clean_code takes a single parameter 'code' which is a string
# The function:
# 1. Checks if code is empty/falsy
# 2. Strips and dedents the code
# 3. Removes various code block prefixes (case-insensitive)
# 4. Returns the cleaned code string

# Create test cases for the 'code' parameter
code_with_prefix = "python\n    def hello():\n        pass"
code_with_spaces = "   \n   some code here\n   "
code_simple = "x = 1 + 2"
code_empty = ""
code_with_py_prefix = "py\nprint('hello')"
code_with_code_prefix = "code: x = 5"
code_with_chinese = "执行:\n    y = 10"

# The function uses the 'code' parameter as a string
# It calls: .strip(), .lower(), .startswith(), slicing, and .lstrip()
# All standard string operations

ARGS = {
    "code": "python\n    def example():\n        return 42"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_tag_content Ran
ran; our contract draft was wrong, not the code
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("8d211f361659e3f9")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re
from typing import Any, Dict, List, Optional, Protocol

def extract_tag_content(text: str, tag: str) -> Optional[str]:
    m = re.search(f'<{tag}>\\s*([\\s\\S]*?)\\s*</{tag}>', text)
    if not m:
        return None
    return m.group(1).strip()

ARGS = {
    "text": "<example>  Hello World  </example>",
    "tag": "example"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
find_brace_json Ran
ran; our contract draft was wrong, not the code
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("c6e31237e73bb6fe")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# The function find_brace_json takes a single parameter: text (str)
# It searches for a JSON brace pair in the text and returns the substring
# or None if not found

ARGS = {
    "text": '{"key": "value", "nested": {"inner": "data"}}'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
is_valid_python_code Ran
ran, violated the contract we drafted
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("e2b550ccc397bb00")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# The function is_valid_python_code takes a single string parameter 'code'
# and checks if it's valid Python code by attempting to compile it.
# We need to provide valid and invalid code strings as test cases.

ARGS = {
    "code": "x = 1 + 2\nprint(x)"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
repair_json_like Ran
ran; our contract draft was wrong, not the code
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("c06c9168b4af5105")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def repair_json_like(s: str) -> str:
    if not isinstance(s, str):
        return s
    t = s
    t = re.sub('\\bTrue\\b', 'true', t)
    t = re.sub('\\bFalse\\b', 'false', t)
    t = re.sub('\\bNone\\b', 'null', t)
    t = re.sub("'([A-Za-z0-9_\\- ]+)'\\s*:", '"\\1":', t)
    t = re.sub(":\\s*'([^']*)'", ': "\\1"', t)
    t = re.sub(',\\s*([}\\]])', '\\1', t)
    return t

# Test cases to demonstrate the function works
test_string = "{'key': 'value', 'flag': True, 'empty': None, 'another': 'test',}"

ARGS = {
    's': test_string
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
resolve_question_id Ran
ran; our contract draft was wrong, not the code
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("d38392764827dbe4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import json
from typing import Any, Dict, List

def _normalize_for_hash(value: Any) -> Any:
    if isinstance(value, dict):
        return {str(k): _normalize_for_hash(value[k]) for k in sorted(value.keys(), key=str)}
    if isinstance(value, list):
        return [_normalize_for_hash(item) for item in value]
    return value

def build_question_id(question: Any, metadata: List[Dict[str, Any]]) -> str:
    import hashlib
    payload = {'question': str(question or '').strip(), 'metadata': _normalize_for_hash(metadata or [])}
    raw = json.dumps(payload, ensure_ascii=False, sort_keys=True, separators=(',', ':'))
    return 'q_' + hashlib.sha256(raw.encode('utf-8')).hexdigest()[:16]

def resolve_question_id(item: Dict[str, Any]) -> str:
    explicit = item.get('question_id')
    if explicit not in (None, ''):
        return str(explicit)
    return build_question_id(item.get('question'), item.get('metadata') or [])

# Test case 1: item with explicit question_id
item_with_explicit_id = {
    'question_id': 'q_explicit_12345',
    'question': 'What is the answer?',
    'metadata': [{'source': 'test', 'version': 1}]
}

# Test case 2: item without explicit question_id, should compute from question and metadata
item_without_explicit_id = {
    'question': 'What is machine learning?',
    'metadata': [
        {'source': 'book', 'chapter': 3},
        {'source': 'paper', 'year': 2023}
    ]
}

# Test case 3: item with empty question_id (should compute)
item_with_empty_id = {
    'question_id': '',
    'question': 'How does neural network work?',
    'metadata': [{'type': 'educational'}]
}

# Test case 4: minimal item with only question
item_minimal = {
    'question': 'Simple question'
}

ARGS = {
    'item': item_with_explicit_id
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
robust_parse_json Ran
ran; our contract draft was wrong, not the code
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("27f7f31226057a49")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import re
from typing import Any, Dict, List, Optional

# Construct test cases for robust_parse_json
ARGS = {
    "text": '{"key": "value", "number": 42, "nested": {"inner": true}}',
    "expected_type": None
}

# Test with different scenarios to ensure function works
test_cases = [
    # Basic JSON string
    ('{"name": "test"}', None),
    # JSON in code fence
    ('```json\n{"a": 1}\n```', None),
    # JSON in plain code fence
    ('```\n{"b": 2}\n```', None),
    # JSON with True/False/None (Python style)
    ('{\'key\': \'value\', \'flag\': True}', None),
    # Plain brace JSON
    ('Some text before {"x": 1} some text after', None),
    # Expected type as list
    ('[1, 2, 3]', 'list'),
    # Expected type as dict
    ('{"data": [1,2,3]}', 'dict'),
    # Empty/malformed input
    ('', None),
    # Non-string (edge case - function checks isinstance)
    ('', None),
]

# Use the first test case as the primary ARGS
ARGS = {
    "text": '{"name": "Alice", "age": 30, "active": true}',
    "expected_type": 'dict'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
safe_template_format Ran
ran; our contract draft was wrong, not the code
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("531accf8ba302721")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re
from typing import Any, Dict, List, Optional, Protocol

_PLACEHOLDER_RE = re.compile('\\{([a-zA-Z_][a-zA-Z0-9_]*)\\}')

def safe_template_format(template: str, values: Dict[str, Any]) -> str:

    def _replace(match: re.Match) -> str:
        key = match.group(1)
        return str(values[key]) if key in values else match.group(0)
    return _PLACEHOLDER_RE.sub(_replace, template)


# Setup code
ARGS = {
    "template": "Hello {name}, you have {count} messages and {score} points.",
    "values": {
        "name": "Alice",
        "count": 42,
        "score": 99.5
    }
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
AsyncMutiDocAgent Not yet run
unverified (harvested, no recorded run)
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("bfeecb9b17f841a7")
How it ran

no invocation recorded.

OpenRouter Not yet run
unverified (harvested, no recorded run)
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("3d73678810ab717a")
How it ran

no invocation recorded.

extract_json_from_code_fence Not yet run
unverified (harvested, no recorded run)
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("b7d4a774bdb8e5b7")
How it ran

no invocation recorded.

extract_python_code Not yet run
unverified (harvested, no recorded run)
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("41d7561067299007")
How it ran

no invocation recorded.

resolve_extractor Not yet run
unverified (harvested, no recorded run)
Zhanli-Li/MuDABench/agent/agent_runner.py
code served (permissive licence) · get_code("74220594a73fb1a6")
How it ran

no invocation recorded.

Also present in other papers' code

Of 26 samples lifted from this paper's linked repositories, 26 carry a function identity and 0 do not. Of the 26 distinct functions with an identity, 0 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 0 later) and 26 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

This paper introduces the task of analytical question answering over large, semi-structured document collections. We present MuD-ABench, a benchmark for multi-document analytical QA, where questions require extracting and synthesizing information across numerous documents to perform quantitative analysis. Unlike existing multi-document QA benchmarks that typically require information from only a few documents with limited crossdocument reasoning, MuDABench demands extensive inter-document analysis and aggregation. Constructed via distant supervision by leveraging document-level metadata and annotated financial databases, MuDABench comprises over 80,000 pages and 332 analytical QA instances. We also propose an evaluation protocol that measures final answer accuracy and uses intermediate-fact coverage as an auxiliary diagnostic signal for the reasoning process. Experiments reveal that standard RAG systems, which treat all documents as a flat retrieval pool, perform poorly. To address these limitations, we propose a multi-agent workflow that orchestrates planning, extraction, and code generation modules. While this approach substantially improves both process and outcome metrics, a significant gap remains compared to human expert performance. Our analysis identifies two primary bottlenecks: single-document information extraction accuracy and insufficient domain-specific knowledge in current systems. MuDABench is available at https: //github.com/Zhanli-Li/MuDABench.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2604.22239")
get_code_for_paper("2604.22239")
have("2604.22239")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2604.22239.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2604.22239.svg)](https://syntology.ai/paper/2604.22239)

Connect an agent — have() is free.