SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2205.02005 · NAACL · 2022

Explaining Why: How Instructions and User Interfaces Impact Annotator Rationales When Labeling Text Data

Yuxin Chen, Michael Littman, Chenhao Tan, Kevin Bryson, Jamar Sullivan, Will Brackenbury, Andrew Mcnutt, Kwam Byll, Blase Ur

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

In the context of data labeling, NLP researchers are increasingly interested in having humans select rationales, a subset of input tokens relevant to the chosen label. We conducted a 332-participant online user study to understand how humans select rationales, especially how different instructions and user interface affordances impact the rationales chosen. Participants labeled ten movie reviews as positive or negative, selecting words and phrases supporting their label as rationales. We varied the instructions given, the rationaleselection task, and the user interface. Participants often selected about 12% of input tokens as rationales, but selected fewer if unable to drag over multiple tokens at once. Whereas participants were near unanimous in their data labels, they were far less consistent in their rationales. The user interface affordances and task greatly impacted the types of rationales chosen. We also observed large variance across participants.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2205.02005")
get_code_for_paper("2205.02005")
have("2205.02005")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2205.02005.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2205.02005.svg)](https://syntology.ai/paper/2205.02005)

Connect an agent — have() is free.