SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2005.02433 · ACL · 2020

Stolen Probability: A Structural Weakness of Neural Language Models

Doug Downey, David Demeter, Gregory Kimmel

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Neural Network Language Models (NNLMs) generate probability distributions by applying a softmax function to a distance metric formed by taking the dot product of a prediction vector with all word vectors in a high-dimensional embedding space. The dot-product distance metric forms part of the inductive bias of NNLMs. Although NNLMs optimize well with this inductive bias, we show that this results in a sub-optimal ordering of the embedding space that structurally impoverishes some words at the expense of others when assigning probability. We present numerical, theoretical and empirical analyses showing that words on the interior of the convex hull in the embedding space have their probability bounded by the probabilities of the words on the hull.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2005.02433")
get_code_for_paper("2005.02433")
have("2005.02433")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2005.02433.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2005.02433.svg)](https://syntology.ai/paper/2005.02433)

Connect an agent — have() is free.