SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2005.01641 · ACL · 2020

A Tale of a Probe and a Parser

Ryan Cotterell, Tiago Pimentel, Adina Williams, Josef Valvoda, Rowan Hall Maudslay

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Measuring what linguistic information is encoded in neural models of language has become popular in NLP. Researchers approach this enterprise by training "probes"supervised models designed to extract linguistic structure from another model's output. One such probe is the structural probe (Hewitt and Manning, 2019), designed to quantify the extent to which syntactic information is encoded in contextualised word representations. The structural probe has a novel design, unattested in the parsing literature, the precise benefit of which is not immediately obvious. To explore whether syntactic probes would do better to make use of existing techniques, we compare the structural probe to a more traditional parser with an identical lightweight parameterisation. The parser outperforms structural probe on UUAS in seven of nine analysed languages, often by a substantial amount (e.g. by 11.1 points in English). Under a second less common metric, however, there is the opposite trend-the structural probe outperforms the parser. This begs the question: which metric should we prefer?

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2005.01641")
get_code_for_paper("2005.01641")
have("2005.01641")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2005.01641.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2005.01641.svg)](https://syntology.ai/paper/2005.01641)

Connect an agent — have() is free.