SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2506.04410 · EMNLP · 2025

MATTER-OF-FACT: A Benchmark for Verifying the Feasibility of Literature-Supported Claims in Materials Science

Peter Jansen, Ruoyao Wang, Samiah Hassan

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Contemporary approaches to assisted scientific discovery use language models to automatically generate large numbers of potential hypothesis to test, while also automatically generating code-based experiments to test those hypotheses. While hypotheses can be comparatively inexpensive to generate, automated experiments can be costly, particularly when run at scale (i.e. thousands of experiments). Developing the capacity to filter hypotheses based on their feasibility would allow discovery systems to run at scale, while increasing their likelihood of making significant discoveries. In this work we introduce MATTER-OF-FACT, a challenge dataset for determining the feasibility of hypotheses framed as claims, while operationalizing feasibility assessment as a temporally-filtered claim verification task using backtesting. MATTER-OF-FACT includes 8.4K claims extracted from scientific articles spanning four high-impact contemporary materials science topics, including superconductors, semiconductors, batteries, and aerospace materials, while including qualitative and quantitative claims from theoretical, experimental, and code/simulation results. We show that strong baselines that include retrieval augmented generation over scientific literature and code generation fail to exceed 72% performance on this task (chance performance is 50%), while domain-expert verification suggests nearly all are solvable -highlighting both the difficulty of this task for current models, and the potential to accelerate scientific discovery by making near-term progress. 1 Claim: In Li10Ge(PS6)2, Li+ ion motion becomes less correlated at lower temps., with Haven ratios closer to 1 below the sublattice phase trans. temp. Gold Label: True (feasible) Explanation: Demonstrated through Haven ratio calculations, Li+ ion motion becomes less correlated at lower temperatures. The Haven ratio approaches 1 below the sublattice phase transition temperature (~400K). Types: Qualitative, Code/Simulation Knowledge Cuto : November 2022 Superconductors Superconductor Claim #216 Claim: A half-smeared potential e ectively decreases the local energy gap in a one-dimensional topological superconductor. Gold Label: True (feasible) Explanation: Local Density of States (LDOS) calculations show a shears-like structure, which indicates a decreased local gap in the region where the potential is smeared. Types: Qualitative, Theoretical Knowledge Cuto : August 2022 Semi-conductors Semi-conductor Claim #890 Claim: In laser-driven nanosatellites, lightsails should remain attached to the payload throught the mission to provide corrections and attitude control. Gold Label: False (infeasible) Explanation: Laser-propelled nanosatellites would receive laser energy from the Earth are expected to reach their maximum speed in only tens of seconds, making future corrections or control infeasible. Type: Qualitative, Integrative Knowledge Cuto : August 2022 Aerospace Materials Claim #122 Claim: The amount of retained austenite in martensitic stainless steel cannot be reduced below 15% regardless of heat treatment parameters Gold Label: False (infeasible) Explanation: Experimental evidence has shown that with proper heat treatment procedures, the amount of retained austentite in matensitic stainless steel can be reduced to less than 5%.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2506.04410")
get_code_for_paper("2506.04410")
have("2506.04410")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2506.04410.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2506.04410.svg)](https://syntology.ai/paper/2506.04410)

Connect an agent — have() is free.