SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 1905.00537 · 2019

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

Felix Hill, Amanpreet Singh, Alex Wang, Samuel Bowman, Julian Michael, Yada Pruksachatkun, Omer Levy, Nikita Nangia

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at super.gluebenchmark.com.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("1905.00537")
get_code_for_paper("1905.00537")
have("1905.00537")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/1905.00537.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/1905.00537.svg)](https://syntology.ai/paper/1905.00537)

Connect an agent — have() is free.