Jerry Ma, Denis Yarats, Vitaliy Polshkov, Marcin Pitera, Jeremy Yang, Kirill Priemko, Maksim Gaiduk, Aleksandr Nikolenko, Denis Bykov, Clare Southern
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
) is a benchmark of 500 realistic and challenging agentic data-collection tasks for knowledge work. Each task asks a system to discover a large collection of members satisfying specified criteria (the wide axis) and establish specific facts about each member through orchestrated web searches (the deep axis). We represent a task as a qualification key hierarchy such as company (n) → employee (m) → url (k), meaning "find n qualifying companies, m qualifying employees per company, and k qualifying source pages per employee." This hierarchy defines a target of n × m × k records. Instead of comparing submissions against a static gold answer set, a task-specific judge re-fetches the cited pages and verifies each record's claims against the cited evidence, allowing the benchmark to cover current and changing facts. Record-level verdicts are aggregated into soft and hard precision, recall, and F1 at the task level. Required record counts range from dozens to thousands, shifting the central challenge from finding a single correct answer or writing a polished report to open-ended discovery at high volume without sacrificing per-record factual correctness. Tasks are derived from de-identified product-usage logs and produced by a semi-automated pipeline. Automated checks and empirical audits screen them, with human review where needed; the final set is curated to stress-test recurring failure modes of frontier agentic search systems. The benchmark remains far from saturated: at a high effort setting, the strongest system achieves only 0.363 soft F1 and 0.133 hard F1. The tasks and evaluation harness are available at https://github.com/perplexityai/wandr.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2608.14747")
get_code_for_paper("2608.14747")
have("2608.14747")
Connect an agent — have() is free.