SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2406.04145 · ACL · 2024

Every Answer Matters: Evaluating Commonsense with Probabilistic Measures

Tim O'gorman, Andrew Mccallum, Xiang Li, Michael Boratko, Qi Cheng, Pranay Kumar, Nalini Singh

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Large language models have demonstrated impressive performance on commonsense tasks; however, these tasks are often posed as multiple-choice questions, allowing models to exploit systematic biases (Li et al., 2022). Commonsense is also inherently probabilistic with multiple correct answers. The purpose of "boiling water" could be making tea, cooking but also could be killing germs. Existing tasks do not capture the probabilistic nature of common sense. To this end, we present commonsense frame completion (CFC), a new generative task that evaluates common sense via multiple openended generations 1 . We also propose a method of probabilistic evaluation that strongly correlates with human judgments. Humans drastically outperform strong language model baselines on our dataset, indicating this approach is both a challenging and useful evaluation of machine common sense.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2406.04145")
get_code_for_paper("2406.04145")
have("2406.04145")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2406.04145.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2406.04145.svg)](https://syntology.ai/paper/2406.04145)

Connect an agent — have() is free.