Ryan Cotterell, Tiago Pimentel, Damián Blasi, Irene Nikkarinen
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The unigram distribution is the non-contextual probability of finding a specific word form in a corpus. While of central importance to the study of language, it is commonly approximated by each word's sample frequency in the corpus. This approach, being highly dependent on sample size, assigns zero probability to any out-of-vocabulary (oov) word form. As a result, it produces negatively biased probabilities for any oov word form, while positively biased probabilities to in-corpus words. In this work, we argue in favor of properly modeling the unigram distribution-claiming it should be a central task in natural language processing. With this in mind, we present a novel model for estimating it in a language (a neuralization of Goldwater et al.'s (2011) model) and show it produces much better estimates across a diverse set of 7 languages than the naïve use of neural character-level language models.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2106.02289")
get_code_for_paper("2106.02289")
have("2106.02289")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2106.02289.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2106.02289)
Connect an agent — have() is free.