SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2204.05660 · ACL · 2022

NUMGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks

Peter Clark, Swaroop Mishra, Chitta Baral, Bhavdeep Sachdeva, Ashwin Kalyan, Arindam Mitra, Neeraj Varshney

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.

Abstract

Given the ubiquitous nature of numbers in text, reasoning with numbers to perform simple calculations is an important skill of AI systems. While many datasets and models have been developed to this end, state-ofthe-art AI systems are brittle; failing to perform the underlying mathematical reasoning when they appear in a slightly different scenario. Drawing inspiration from GLUE (Wang et al., 2018) that was proposed in the context of natural language understanding, we propose NUMGLUE, a multi-task benchmark that evaluates the performance of AI systems on eight different tasks, that at their core require simple arithmetic understanding. We show that this benchmark is far from being solved with neural models including state-ofthe-art large-scale language models performing significantly worse than humans (lower by 46.4%). Further, NUMGLUE promotes sharing knowledge across tasks, especially those with limited training data as evidenced by the superior performance (average gain of 3.4% on each task) when a model is jointly trained on all the tasks as opposed to taskspecific modeling. Finally, we hope that NUMGLUE will encourage systems that perform robust and general arithmetic reasoning within language, a first step towards being able to perform more complex mathematical reasoning 1 . 1 https://allenai.org/data/numglue Original Word Problem John had 5 apples. He gave 3 to Peter. How many apples does John have now? Fill In The Blanks Format John had 5 apples. He gave 3 to Peter. John has apples now. NLI Format Premise: John had 5 apples. He gave 3 apples to Peter. Hypothesis: John has 2 apples now. Does the hypothesis entail, contradict or is neutral to the premise? Comparison Format John had 5 apples. He gave 3 to Peter. Who has more apples?

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2204.05660")
get_code_for_paper("2204.05660")
have("2204.05660")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2204.05660.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2204.05660.svg)](https://syntology.ai/paper/2204.05660)

Connect an agent — have() is free.