SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2105.03519 · NAACL · 2021

Understanding by Understanding Not: Modeling Negation in Language Models

Aaron Courville, Devon Hjelm, Siva Reddy, Alessandro Sordoni, Dzmitry Bahdanau, Arian Hosseini

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Negation is a core construction in natural language. Despite being very successful on many tasks, state-of-the-art pre-trained language models often handle negation incorrectly. To improve language models in this regard, we propose to augment the language modeling objective with an unlikelihood objective that is based on negated generic sentences from a raw text corpus. By training BERT with the resulting combined objective we reduce the mean top 1 error rate to 4% on the negated LAMA dataset. We also see some improvements on the negated NLI benchmarks. Top 3 words with log probs from BERT Top 3 words with log probs from BERTNOT iOS is developed by [MASK]. Apple (-1.8), Google (-2.6), Microsoft (-2.8) Apple (-1.8), Google (-2.5), Microsoft (-2.7) iOS is not developed by [MASK]. Apple (-1.8), Google (-2.6), Microsoft (-2.8) Microsoft (-1.8), Google (-2.4), Apple (-3.1) The majority of the amazon forest is in [MASK]. Brazil (-2.6), Bolivia (-2.7), Madagascar (-3.1) Brazil (-2.9), Bolivia (-3.1), Mexico (-3.2) The majority of the amazon forest is not in [MASK]. cultivation (-1.0), Brazil (-3.5), Mexico (-3.5) cultivation (-2.0), Mexico (-4.1), France (-4.3) Charles Nodier died in [MASK]. Paris (-1.35), Rome (-3.2), office (-3.4) Paris (-1.5), Rome (-3.3), France (-3.6) Charles Nodier did not die in [MASK]. Paris (-2.4), office (-2.7), France (-2.8) vain (-3.5), error (-4.0), doubt (-4.5) Mac OS is developed by [MASK]. Apple (-1.9), Microsoft (-2.0), Intel (-2.0) Apple (-2.0), Microsoft (-2.0), Intel (-2.1) Mac OS is not developed by [MASK]. Apple (-1.3), Microsoft (-1.5), IBM (-2.3) Microsoft (-2.1), IBM (-2.7), itself (-3.4)

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2105.03519")
get_code_for_paper("2105.03519")
have("2105.03519")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2105.03519.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2105.03519.svg)](https://syntology.ai/paper/2105.03519)

Connect an agent — have() is free.