Quoc Le, Kevin Clark, Minh-Thang Luong, Christopher Manning
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We introduce Electric, an energy-based cloze model for representation learning over text. Like BERT, it is a conditional generative model of tokens given their contexts. However, Electric does not use masking or output a full distribution over tokens that could occur in a context. Instead, it assigns a scalar energy score to each input token indicating how likely it is given its context. We train Electric using an algorithm based on noise-contrastive estimation and elucidate how this learning objective is closely related to the recently proposed ELECTRA pre-training method. Electric performs well when transferred to downstream tasks and is particularly effective at producing likelihood scores for text: it reranks speech recognition n-best lists better than language models and much faster than masked language models. Furthermore, it offers a clearer and more principled view of what ELECTRA learns during pre-training.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2012.08561")
get_code_for_paper("2012.08561")
have("2012.08561")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2012.08561.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2012.08561)
Connect an agent — have() is free.