SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2106.01494 · ACL Findings · 2021

Knowing More About Questions Can Help: Improving Calibration in Question Answering

Chengyue Gong, Eunsol Choi, Shujian Zhang

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

We study calibration in question answering, estimating whether model correctly predicts answer for each question. Unlike prior work which mainly rely on the model's confidence score, our calibrator incorporates information about the input example (e.g., question and the evidence context). Together with data augmentation via back translation, our simple approach achieves 5-10% gains in calibration accuracy on reading comprehension benchmarks. Furthermore, we present the first calibration study in the open retrieval setting, comparing the calibration accuracy of retrieval-based span prediction models and answer generation models. Here again, our approach shows consistent gains over calibrators relying on the model confidence. Our simple and efficient calibrator can be easily adapted to many tasks and model architectures, showing robust gains in all settings. 1

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2106.01494")
get_code_for_paper("2106.01494")
have("2106.01494")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2106.01494.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2106.01494.svg)](https://syntology.ai/paper/2106.01494)

Connect an agent — have() is free.