SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2010.02570 · EMNLP · 2020

Does the Objective Matter? Comparing Training Objectives for Pronoun Resolution

Thomas Lukasiewicz, Vid Kocijan, Oana-Maria Camburu, Yordan Yordanov

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Hard cases of pronoun resolution have been used as a long-standing benchmark for commonsense reasoning. In the recent literature, pre-trained language models have been used to obtain state-of-the-art results on pronoun resolution. Overall, four categories of training and evaluation objectives have been introduced. The variety of training datasets and pretrained language models used in these works makes it unclear whether the choice of training objective is critical. In this work, we make a fair comparison of the performance and seedwise stability of four models that represent the four categories of objectives. Our experiments show that the objective of sequence ranking performs the best in-domain, while the objective of semantic similarity between candidates and pronoun performs the best out-of-domain. We also observe a seed-wise instability of the model using sequence ranking, which is not the case when the other objectives are used.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2010.02570")
get_code_for_paper("2010.02570")
have("2010.02570")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2010.02570.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2010.02570.svg)](https://syntology.ai/paper/2010.02570)

Connect an agent — have() is free.