SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2604.10261 · 2026

The Amazing Agent Race: Strong Tool Users, Weak Navigators

Jaehyung Kim, Dongyeop Kang, Vipul Raheja, Zae Kim, Dongseok Lee

arXiv · PDF · Open in the Atlas

Code that ran

17 of 26 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 1 honoured the contract we drafted, 2 violated it and 0 have no contract verdict; 14 failures of our instrument; 9 no recorded run), measured 2026-11-22; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2604.10261.json · badge: /api/ran/2604.10261.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
minnesotanlp/the-amazing-agent-race community
Papers with Code archive
17 ran · 9 no recorded run
FunctionStatusWhere it lives
_clean_markdown Ran
ran; our contract draft was wrong, not the code
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("3b168f47e5b4f118")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def _clean_markdown(text: str) -> str:
    """Strip markdown artifacts from extracted values.

    Handles: [text](url) links, Wikipedia citation brackets, bullet prefixes,
    pipe chars from tables.
    """
    text = re.sub('\\[\\[\\d+\\]\\]\\(#cite_note[^)]*\\)', '', text)
    text = re.sub('\\[(\\d+|note \\d+)\\]', '', text)
    text = re.sub('\\[([^\\]]*)\\]\\([^)]*\\)', '\\1', text)
    text = re.sub('^\\s*[\\*\\-•]\\s+', '', text)
    text = text.strip().strip('|').strip()
    return text

ARGS = {
    'text': 'This is a [[1]](#cite_note-1) test [link](https://example.com) with - bullet points and |pipes|'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
_fix_coords_in_key Ran
ran; our contract draft was wrong, not the code
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("2de8b9d85813088f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def _fix_coords_in_key(key: str, value: str) -> tuple[str, str]:
    """Fix cases where coordinate data leaks into the infobox key.

    Wikipedia renders coordinates as markdown links inside table cells,
    which causes the entire coordinate text (including geohack URL) to
    be captured as the key instead of the value.
    """
    if 'geohack.toolforge.org' not in key and (not re.search('\\d+°\\d+[′\']\\d+[″\\"]', key)):
        return (key, value)
    coord_match = re.search('(-?\\d+\\.\\d+)[;,]\\s*(-?\\d+\\.\\d+)', key)
    if coord_match:
        value = f'{coord_match.group(1)}, {coord_match.group(2)}'
        key = key.split(':')[0].strip()
        return (key, value)
    return (key, value)


ARGS = {
    "key": "coordinates: geohack.toolforge.org/params/40.7128,-74.0060",
    "value": "New York City"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
_get_page_ticker Ran
ran; our contract draft was wrong, not the code
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("bc128de5928ca206")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re
from typing import Any

class MockInfoboxField:
    def __init__(self, key: str, value: str):
        self.key = key
        self.value = value

class MockPage:
    def __init__(self, infobox_fields: list):
        self.infobox = infobox_fields

# Create a mock page with infobox containing ticker information
infobox_fields = [
    MockInfoboxField("Company", "Apple Inc."),
    MockInfoboxField("[Traded as](https://example.com)", "NASDAQ: AAPL"),
    MockInfoboxField("Founded", "1976"),
]

page = MockPage(infobox_fields)

ARGS = {
    "page": page,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
_is_boring_link Ran
ran, violated the contract we drafted
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("96c5781fe4eb6cbe")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

_BORING_LINK_PATTERNS = re.compile('^\\d{3,4}$|^\\d{1,2}(st|nd|rd|th)_century$|^List_of_|^Index_of_', re.IGNORECASE)

def _is_boring_link(title: str) -> bool:
    """Check if a link target is a year, decade, or overly broad category."""
    cleaned = title.replace(' ', '_')
    return bool(_BORING_LINK_PATTERNS.match(cleaned))

# Test cases to construct valid arguments
ARGS = {
    "title": "2023"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
_is_wikipedia_article_url Ran
ran, violated the contract we drafted
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("1d6e5237d3056e1d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

ARGS = {
    'url': 'https://en.wikipedia.org/wiki/Python_(programming_language)'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
_normalize_wiki_url Ran
ran; our contract draft was wrong, not the code
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("f3fc6d916e08980c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from urllib.parse import quote, unquote, urlparse

# The function _normalize_wiki_url takes a single parameter: url (str)
# It parses the URL, checks for fragments, and removes them if present
# No complex objects needed - just a string

ARGS = {
    "url": "https://en.wikipedia.org/wiki/Python_(programming_language)#History"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
_page_info_from_dict Ran
ran; our contract draft was wrong, not the code
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("62011b7560c8c2cc")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned PageInfo. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np
from dataclasses import dataclass, field
from typing import Any

# Set seed for reproducibility
torch.manual_seed(0)

# Construct a minimal valid dictionary for _page_info_from_dict
data = {
    'url': 'https://en.wikipedia.org/wiki/Example',
    'title': 'Example Page',
    'infobox': [
        {
            'key': 'Birth Date',
            'value': '1990-01-15',
            'numeric_value': 1990.0
        },
        {
            'key': 'Population',
            'value': '1,000,000',
            'numeric_value': 1000000.0
        }
    ],
    'outgoing_links': [
        {
            'text': 'Related Topic',
            'target_title': 'Related_Topic',
            'target_url': 'https://en.wikipedia.org/wiki/Related_Topic',
            'section': 'See also'
        },
        {
            'text': 'Another Link',
            'target_title': 'Another_Link',
            'target_url': 'https://en.wikipedia.org/wiki/Another_Link',
            'section': None
        }
    ],
    'sections': ['Introduction', 'History', 'See also'],
    'first_paragraph': 'This is the first paragraph of the Wikipedia page about the example topic.',
    'page_length': 5432,
    'pageviews': 15000,
    'fetch_timestamp': '2024-01-15T10:30:00Z',
    'raw_markdown': '# Example\n\nThis is example markdown content.'
}

ARGS = {
    'data': data
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type PageInfo — a PageInfo output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a PageInfo output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
_page_info_to_dict Ran
ran; our contract draft was wrong, not the code
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("e6bb27e1801240f6")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from dataclasses import dataclass, field
from typing import Any

@dataclass
class InfoboxField:
    """A single key-value pair from a Wikipedia infobox."""
    key: str
    value: str
    numeric_value: float | None = None

@dataclass
class WikiLink:
    """An outgoing link from a Wikipedia page."""
    text: str
    target_title: str
    target_url: str
    section: str | None = None

@dataclass
class PageInfo:
    """Cached Wikipedia page metadata."""
    url: str
    title: str
    infobox: list[InfoboxField] = field(default_factory=list)
    outgoing_links: list[WikiLink] = field(default_factory=list)
    sections: list[str] = field(default_factory=list)
    first_paragraph: str = ''
    page_length: int = 0
    pageviews: int | None = None
    fetch_timestamp: str = ''
    raw_markdown: str = ''

# Create sample infobox fields
infobox_fields = [
    InfoboxField(key="birth_date", value="1990-01-15", numeric_value=1990.0),
    InfoboxField(key="height", value="180 cm", numeric_value=180.0),
]

# Create sample outgoing links
outgoing_links = [
    WikiLink(text="Python", target_title="Python_(programming_language)", target_url="https://en.wikipedia.org/wiki/Python_(programming_language)", section="Introduction"),
    WikiLink(text="Wikipedia", target_title="Wikipedia", target_url="https://en.wikipedia.org/wiki/Wikipedia", section=None),
]

# Create sample sections
sections = ["Introduction", "History", "Features", "Applications"]

# Create a PageInfo instance
page_info = PageInfo(
    url="https://en.wikipedia.org/wiki/Example",
    title="Example Page",
    infobox=infobox_fields,
    outgoing_links=outgoing_links,
    sections=sections,
    first_paragraph="This is the first paragraph of the example page.",
    page_length=5000,
    pageviews=10000,
    fetch_timestamp="2024-01-15T10:30:00Z",
    raw_markdown="# Example Page\n\nThis is markdown content."
)

ARGS = {
    "page": page_info
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
_strip_wiki_chrome Ran
ran; our contract draft was wrong, not the code
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("bc0443ad6dcf8952")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def _strip_wiki_chrome(markdown: str) -> str:
    """Strip Wikipedia navigation chrome (sidebar, TOC, language links).

    Wikipedia pages fetched via markdownify include thousands of characters
    of navigation, sidebar, and table-of-contents content before the actual
    article.  We skip to the article body so that infobox parsing can work
    within a reasonable search window.
    """
    marker = 'From Wikipedia, the free encyclopedia'
    idx = markdown.find(marker)
    if idx >= 0:
        return markdown[idx + len(marker):]
    h1_match = re.search('^# .+$', markdown, re.MULTILINE)
    if h1_match:
        return markdown[h1_match.start():]
    return markdown


# Test case 1: markdown with the Wikipedia marker
markdown_with_marker = "Navigation\nSidebar\nFrom Wikipedia, the free encyclopedia\n# Article Title\nContent here"

# Test case 2: markdown with h1 header but no marker
markdown_with_h1 = "Navigation\nSidebar\n# Article Title\nContent here"

# Test case 3: markdown with neither marker nor h1
markdown_plain = "Navigation\nSidebar\nSome content"

ARGS = {
    "markdown": markdown_with_marker
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
_title_from_url Ran
ran; our contract draft was wrong, not the code
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("36bae59154179886")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from urllib.parse import quote, unquote, urlparse

# The function _title_from_url takes a single parameter: url (a string)
# It parses the URL, extracts the path, and if it starts with '/wiki/',
# it unquotes and replaces underscores with spaces

ARGS = {
    "url": "https://en.wikipedia.org/wiki/Python_(programming_language)"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
_try_parse_numeric Ran
ran, honoured the contract we drafted
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("82128b92c53ae52b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def _try_parse_numeric(text: str) -> float | None:
    """Attempt to parse a numeric value from a string like '2,240 m' or '8,849'."""
    cleaned = re.sub('\\[([^\\]]+)\\]\\([^)]+\\)', '\\1', text)
    match = re.search('[-+]?[\\d,]+\\.?\\d*', cleaned)
    if not match:
        return None
    num_str = match.group(0).replace(',', '')
    try:
        val = float(num_str)
        return int(val) if val == int(val) else val
    except (ValueError, OverflowError):
        return None

ARGS = {
    "text": "2,240 m"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
_url_hash Ran
ran; our contract draft was wrong, not the code
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("761c80bb7720cc62")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import hashlib

def _url_hash(url: str) -> str:
    return hashlib.sha256(url.encode()).hexdigest()[:16]

# The function takes a single string parameter 'url'
# It encodes it, hashes it with SHA256, and returns the first 16 characters
# We need to provide a valid URL string

ARGS = {
    "url": "https://example.com/model/weights.pt"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_first_paragraph Ran
ran; our contract draft was wrong, not the code
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("82d8f76e79208a04")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

_BOILERPLATE_LINES = frozenset({'main menu', 'move to sidebar', 'hide', 'navigation', 'search', 'personal tools', 'contents', 'toggle the table of contents', 'article', 'talk', 'from wikipedia, the free encyclopedia', 'appearance', 'general', 'not logged in'})

_BOILERPLATE_SUBSTRINGS = ('Jump to content', '- Wikipedia', '[Jump to', 'From Wikipedia', 'Toggle the table of contents', 'move to sidebar', 'Main menu', 'Toggle ', 'Create account')

def _strip_wiki_chrome(markdown: str) -> str:
    """Strip Wikipedia navigation chrome (sidebar, TOC, language links).

    Wikipedia pages fetched via markdownify include thousands of characters
    of navigation, sidebar, and table-of-contents content before the actual
    article.  We skip to the article body so that infobox parsing can work
    within a reasonable search window.
    """
    marker = 'From Wikipedia, the free encyclopedia'
    idx = markdown.find(marker)
    if idx >= 0:
        return markdown[idx + len(marker):]
    h1_match = re.search('^# .+$', markdown, re.MULTILINE)
    if h1_match:
        return markdown[h1_match.start():]
    return markdown

def extract_first_paragraph(markdown: str) -> str:
    """Extract the first substantial paragraph from a Wikipedia article."""
    article = _strip_wiki_chrome(markdown)
    lines = article.split('\n')
    paragraph_lines: list[str] = []
    started = False
    for line in lines:
        stripped = line.strip()
        if not started:
            if not stripped or stripped.startswith('#') or stripped.startswith('|') or stripped.startswith('![') or stripped.startswith('---'):
                continue
            if stripped.lower() in _BOILERPLATE_LINES:
                continue
            if any((sub in stripped for sub in _BOILERPLATE_SUBSTRINGS)):
                continue
            if stripped.startswith('* [') or stripped.startswith('- ['):
                continue
            if len(stripped) < 20 and (not stripped.startswith('**')):
                continue
            started = True
        if started:
            if not stripped:
                if paragraph_lines:
                    break
                continue
            if stripped.startswith('#'):
                break
            paragraph_lines.append(stripped)
    return ' '.join(paragraph_lines)[:2000]


# Construct a realistic Wikipedia markdown string with chrome and article content
sample_markdown = """
Main menu
move to sidebar
hide
Navigation
Search
Personal tools
Not logged in
Contents
Toggle the table of contents
Article
Talk
From Wikipedia, the free encyclopedia

# Albert Einstein

**Albert Einstein** was a German-born theoretical physicist who developed the theory of relativity, one of the two pillars of modern physics. His mass–energy equivalence formula E = mc² is the world's most famous equation. He received the 1921 Nobel Prize in Physics "for his services to theoretical physics, and especially for his discovery of the law of the photoelectric effect", a pivotal step in establishing quantum theory.

Born in the German Empire, Einstein immigrated to the United States in 1933. He was affiliated with the Institute for Advanced Study in Princeton, New Jersey, until his death in 1955.

## Early life

Einstein was born on 14 March 1879 in Ulm, in the Kingdom of Württemberg in the German Empire.

## Career

He worked at various universities and research institutions.
"""

ARGS = {
    "markdown": sample_markdown
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_sections Ran
ran; our contract draft was wrong, not the code
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("16e79860878b0114")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def extract_sections(markdown: str) -> list[str]:
    """Extract section headings from markdown."""
    heading_re = re.compile('^#{1,6}\\s+(.+)', re.MULTILINE)
    return [m.group(1).strip() for m in heading_re.finditer(markdown)]

# Construct valid arguments
markdown_content = """# Main Title
Some content here

## Subsection One
More content

### Deeper Section
Even more content

## Another Subsection
Final content

#### Very Deep Section
Deep content
"""

ARGS = {
    "markdown": markdown_content
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_wiki_links Ran
ran; our contract draft was wrong, not the code
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("153a9a0e24de1a37")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re
from dataclasses import asdict, dataclass, field
from urllib.parse import quote, unquote, urlparse

@dataclass
class WikiLink:
    """An outgoing link from a Wikipedia page."""
    text: str
    target_title: str
    target_url: str
    section: str | None = None

# Create a markdown string with Wikipedia links and sections
markdown_content = """
# Introduction

This is a paragraph with a [link to Python](https://en.wikipedia.org/wiki/Python_(programming_language)) programming language.

## History

The [history of computing](/wiki/History_of_computing) is interesting. Here's a link to [1995](/wiki/1995) which should be filtered out.

### Early Days

Check out [Alan Turing](/wiki/Alan_Turing) and the [List_of_computer_scientists](/wiki/List_of_computer_scientists) page.

## See Also

More info on [Machine Learning](/wiki/Machine_Learning) and [Neural Networks](/wiki/Artificial_neural_network).

This is a duplicate [Python_(programming_language)](/wiki/Python_(programming_language)) link that should be deduplicated.
"""

ARGS = {
    "markdown": markdown_content
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
fetch_pageviews Ran
ran; our contract draft was wrong, not the code
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("fedf944b99d9fbb3")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned coroutine. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import logging
from datetime import datetime, timezone

logger = logging.getLogger(__name__)

# Set up minimal logging
logging.basicConfig(level=logging.DEBUG)

ARGS = {
    'title': 'Python_(programming_language)',
    'days': 30,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
httpx — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type coroutine — a coroutine output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a coroutine output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
parse_infobox Ran
ran; our contract draft was wrong, not the code
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("33ebdafd546ec4e6")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re
from dataclasses import asdict, dataclass, field

@dataclass
class InfoboxField:
    """A single key-value pair from a Wikipedia infobox."""
    key: str
    value: str
    numeric_value: float | None = None

def _fix_coords_in_key(key: str, value: str) -> tuple[str, str]:
    """Fix cases where coordinate data leaks into the infobox key.

    Wikipedia renders coordinates as markdown links inside table cells,
    which causes the entire coordinate text (including geohack URL) to
    be captured as the key instead of the value.
    """
    if 'geohack.toolforge.org' not in key and (not re.search('\\d+°\\d+[′\']\\d+[″\\"]', key)):
        return (key, value)
    coord_match = re.search('(-?\\d+\\.\\d+)[;,]\\s*(-?\\d+\\.\\d+)', key)
    if coord_match:
        value = f'{coord_match.group(1)}, {coord_match.group(2)}'
        key = key.split(':')[0].strip()
        return (key, value)
    return (key, value)

def _strip_wiki_chrome(markdown: str) -> str:
    """Strip Wikipedia navigation chrome (sidebar, TOC, language links).

    Wikipedia pages fetched via markdownify include thousands of characters
    of navigation, sidebar, and table-of-contents content before the actual
    article.  We skip to the article body so that infobox parsing can work
    within a reasonable search window.
    """
    marker = 'From Wikipedia, the free encyclopedia'
    idx = markdown.find(marker)
    if idx >= 0:
        return markdown[idx + len(marker):]
    h1_match = re.search('^# .+$', markdown, re.MULTILINE)
    if h1_match:
        return markdown[h1_match.start():]
    return markdown

def _try_parse_numeric(text: str) -> float | None:
    """Attempt to parse a numeric value from a string like '2,240 m' or '8,849'."""
    cleaned = re.sub('\\[([^\\]]+)\\]\\([^)]+\\)', '\\1', text)
    match = re.search('[-+]?[\\d,]+\\.?\\d*', cleaned)
    if not match:
        return None
    num_str = match.group(0).replace(',', '')
    try:
        val = float(num_str)
        return int(val) if val == int(val) else val
    except (ValueError, OverflowError):
        return None

def parse_infobox(markdown: str) -> list[InfoboxField]:
    """Extract key-value pairs from a Wikipedia infobox rendered as markdown.

    Wikipedia infoboxes typically render as a table or a series of bold key / value
    lines near the top of the article.  We attempt several heuristic patterns.
    """
    article = _strip_wiki_chrome(markdown)
    search_window = article[:12000]
    fields: list[InfoboxField] = []
    seen_keys: set[str] = set()
    table_row_re = re.compile('^\\|\\s*\\*?\\*?([^|*]+?)\\*?\\*?\\s*\\|\\s*(.+?)\\s*\\|?\\s*$', re.MULTILINE)
    for match in table_row_re.finditer(search_window):
        key = match.group(1).strip().strip('*').strip()
        value = match.group(2).strip()
        if not key or not value or key.startswith('---'):
            continue
        key, value = _fix_coords_in_key(key, value)
        cleaned_val = value.strip().strip('|').strip()
        if not cleaned_val:
            continue
        key_lower = key.lower()
        if key_lower in seen_keys:
            continue
        seen_keys.add(key_lower)
        numeric = _try_parse_numeric(value)
        fields.append(InfoboxField(key=key, value=value, numeric_value=numeric))
    bold_kv_re = re.compile('\\*\\*([^*]+?)\\*\\*\\s*:?\\s+(.+?)(?:\\n|$)', re.MULTILINE)
    for match in bold_kv_re.finditer(search_window):
        key = match.group(1).strip()
        value = match.group(2).strip()
        if not key or not value:
            continue
        key, value = _fix_coords_in_key(key, value)
        cleaned_val = value.strip().strip('|').strip()
        if not cleaned_val:
            continue
        key_lower = key.lower()
        if key_lower in seen_keys:
            continue
        seen_keys.add(key_lower)
        numeric = _try_parse_numeric(value)
        fields.append(InfoboxField(key=key, value=value, numeric_value=numeric))
    return fields

markdown_content = """
From Wikipedia, the free encyclopedia

# Mount Everest

| Height | 8,849 m |
| Location | Nepal |

**Elevation**: 8,849 m
**First ascent**: 1953
"""

ARGS = {
    "markdown": markdown_content
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
InfoboxField Not yet run
unverified (harvested, no recorded run)
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("1d418693be94de1b")
How it ran

no invocation recorded.

PageInfo Not yet run
unverified (harvested, no recorded run)
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("54495666dfd6fb69")
How it ran

no invocation recorded.

TrailDifficultyConfig Not yet run
unverified (harvested, no recorded run)
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("fa40afece5cec31b")
How it ran

no invocation recorded.

WikiGraph Not yet run
unverified (harvested, no recorded run)
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("59567e4dd3f35ef4")
How it ran

no invocation recorded.

WikiLink Not yet run
unverified (harvested, no recorded run)
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("bcfe6f3d9dcc9f6b")
How it ran

no invocation recorded.

_format_page_summaries Not yet run
unverified (harvested, no recorded run)
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("68fa60fb317d7b6c")
How it ran

no invocation recorded.

_get_page_crypto_symbol Not yet run
unverified (harvested, no recorded run)
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("cfeb5d9b72e9306d")
How it ran

no invocation recorded.

_plan_route Not yet run
unverified (harvested, no recorded run)
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("125cea695600e777")
How it ran

no invocation recorded.

build_page_info Not yet run
unverified (harvested, no recorded run)
minnesotanlp/the-amazing-agent-race/src/trail/builder.py
code served (permissive licence) · get_code("c5896715a3cd6bae")
How it ran

no invocation recorded.

Also present in other papers' code

Of 26 samples lifted from this paper's linked repositories, 26 carry a function identity and 0 do not. Of the 26 distinct functions with an identity, 0 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 0 later) and 26 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Existing tool-use benchmarks for LLM agents are overwhelmingly linear: our analysis of six benchmarks shows 55 to 100% of instances are simple chains of 2 to 5 steps. We introduce THE AMAZING AGENT RACE (AAR), a benchmark featuring directed acyclic graph (DAG) puzzles (or "legs") with fork-merge tool chains. We release 1,400 instances across two variants: sequential (800 legs) and compositional (600 DAG legs). Agents must navigate Wikipedia, execute multi-step tool chains, and aggregate results into a verifiable answer. Legs are procedurally generated from Wikipedia seeds across four difficulty levels with live-API validation. Three complementary metrics (finish-line accuracy, pit-stop visit rate, and roadblock completion rate) separately diagnose navigation, tool-use, and arithmetic failures. Evaluating three agent frameworks on 1,400 legs, the best achieves only 37.2% accuracy. Navigation errors dominate (27 to 52% of trials) while tool-use errors remain below 17%, and agent architecture matters as much as model scale (Claude Code matches Codex CLI at 37% with 6× fewer tokens). The compositional structure of AAR reveals that agents fail not at calling tools but at navigating to the right pages, a blind spot invisible to linear benchmarks. The project page can be accessed at: https://minnesotanlp.github.io/the-amazing-agent-race Existing benchmarks isolate these capabilities: tool-use benchmarks (Qin et al., 2024;Patil et al., 2025) omit navigation, compositional benchmarks (Basu et al., 2024; Ye et al., 2025) provide all inputs upfront, and web-navigation benchmarks (Zhou et al., 2024;Mialon et al., 2024) omit compositional tool chains. Our analysis of their dependency structures reveals that 55 to 100% of instances are strictly linear chains averaging only 2 to 5 steps ( §2), a compositionality deficit that leaves fork-merge reasoning untested. This work. We introduce THE AMAZING AGENT RACE (AAR), a benchmark designed around one diagnostic question: where exactly does an agent break down when it must discover information through navigation, fork that information into parallel tool branches, and merge the

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2604.10261")
get_code_for_paper("2604.10261")
have("2604.10261")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2604.10261.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2604.10261.svg)](https://syntology.ai/paper/2604.10261)

Connect an agent — have() is free.