SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2004.04487 · ACL · 2020

Injecting Numerical Reasoning Skills into Language Models

Jonathan Berant, Ankit Gupta, Mor Geva

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Large pre-trained language models (LMs) are known to encode substantial amounts of linguistic information. However, high-level reasoning skills, such as numerical reasoning, are difficult to learn from a language-modeling objective only. Consequently, existing models for numerical reasoning have used specialized architectures with limited flexibility. In this work, we show that numerical reasoning is amenable to automatic data generation, and thus one can inject this skill into pre-trained LMs, by generating large amounts of data, and training in a multi-task setup. We show that pre-training our model, GENBERT, on this data, dramatically improves performance on DROP (49.3 → 72.3 F 1 ), reaching performance that matches state-of-the-art models of comparable size, while using a simple and general-purpose encoder-decoder architecture. Moreover, GENBERT generalizes well to math word problem datasets, while maintaining high performance on standard RC tasks. Our approach provides a general recipe for injecting skills into large pre-trained LMs, whenever the skill is amenable to automatic data augmentation. * These authors contributed equally. (b) fine-tuning pre-trained LM numerical reasoning reading compr.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2004.04487")
get_code_for_paper("2004.04487")
have("2004.04487")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2004.04487.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2004.04487.svg)](https://syntology.ai/paper/2004.04487)

Connect an agent — have() is free.