SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2108.10949 · EMNLP · 2021

Robustness Evaluation of Entity Disambiguation Using Prior Probes: the Case of Entity Overshadowing

Svitlana Vakulenko, Evangelos Kanoulas, Samarth Bhargav, Vera Provatorova

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.

Abstract

Entity disambiguation (ED) is the last step of entity linking (EL), when candidate entities are reranked according to the context they appear in. All datasets for training and evaluating models for EL consist of convenience samples, such as news articles and tweets, that propagate the prior probability bias of the entity distribution towards more frequently occurring entities. It was previously shown that performance of EL systems on such datasets is overestimated, since it is possible to obtain higher accuracy scores by merely learning the prior. To provide a more adequate evaluation benchmark, we introduce the ShadowLink dataset, which includes 16K short text snippets annotated with entity mentions. We evaluate and report the performance of several popular EL systems on the ShadowLink benchmark. The results show a considerable difference in accuracy between common and uncommon ambiguous entities that require disambiguation, for all of the EL systems under evaluation, demonstrating the effects of prior probability bias and entity overshadowing.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2108.10949")
get_code_for_paper("2108.10949")
have("2108.10949")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2108.10949.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2108.10949.svg)](https://syntology.ai/paper/2108.10949)

Connect an agent — have() is free.