SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2005.00771 · EMNLP · 2020

ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoning

Li, Tim O'gorman, Andrew Mccallum, Rajarshi Das, Michael Boratko, Lorraine Xiang, Dan Le

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Given questions regarding some prototypical situation -such as Name something that people usually do before they leave the house for work? -a human can easily answer them via acquired experiences. There can be multiple right answers for such questions, with some more common for a situation than others. This paper introduces a new question answering dataset for training and evaluating common sense reasoning capabilities of artificial intelligence systems in such prototypical situations. The training set is gathered from an existing set of questions played in a longrunning international game show -FAMILY-FEUD. The hidden evaluation set is created by gathering answers for each question from 100 crowd-workers. We also propose a generative evaluation task where a model has to output a ranked list of answers, ideally covering all prototypical answers for a question. After presenting multiple competitive baseline models, we find that human performance still exceeds model scores on all evaluation metrics with a meaningful gap, supporting the challenging nature of the task. * Equal contribution. (i) Name something that people usually do before they leave for work? Ask 100 crowd-workers + manual clustering

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2005.00771")
get_code_for_paper("2005.00771")
have("2005.00771")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2005.00771.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2005.00771.svg)](https://syntology.ai/paper/2005.00771)

Connect an agent — have() is free.