Samuel Bowman, William Huang, Haokun Liu, Dhara Mungra
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Performance on the Winograd Schema Challenge (WSC), a respected English commonsense reasoning benchmark, recently rocketed from chance accuracy to 89% on the Super-GLUE leaderboard, with relatively little corroborating evidence of a correspondingly large improvement in reasoning ability. We hypothesize that much of this improvement comes from recent changes in task formalizationthe combination of input specification, loss function, and reuse of pretrained parametersby users of the dataset, rather than improvements in the pretrained model's reasoning ability. We perform an ablation on two Winograd Schema datasets that interpolates between the formalizations used before and after this surge, and find (i) framing the task as multiple choice improves performance by 2-6 points and (ii) several additional techniques, including the reuse of a pretrained language modeling head, can mitigate the model's extreme sensitivity to hyperparameters. We urge future benchmark creators to impose additional structure to minimize the impact of formalization decisions on reported results.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2010.04043")
get_code_for_paper("2010.04043")
have("2010.04043")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2010.04043.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2010.04043)
Connect an agent — have() is free.