SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2211.16259 · 2022

Measuring the Measuring Tools: An Automatic Evaluation of Semantic Metrics for Text Corpora

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

The ability to compare the semantic similarity between text corpora is important in a variety of natural language processing applications. However, standard methods for evaluating these metrics have yet to be established. We propose a set of automatic and interpretable measures for assessing the characteristics of corpus-level semantic similarity metrics, allowing sensible comparison of their behavior. We demonstrate the effectiveness of our evaluation measures in capturing fundamental characteristics by evaluating them on a collection of classical and state-of-the-art metrics. Our measures revealed that recently-developed metrics are becoming better in identifying semantic distributional mismatch while classical metrics are more sensitive to perturbations in the surface text levels.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2211.16259")
get_code_for_paper("2211.16259")
have("2211.16259")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2211.16259.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2211.16259.svg)](https://syntology.ai/paper/2211.16259)

Connect an agent — have() is free.