SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2106.01229 · ACL · 2021

Lower Perplexity is Not Always Human-Like

Kentaro Inui, Tatsuki Kuribayashi, Masayuki Asahara, Yohei Oseki, Ryo Yoshida, Takumi Ito

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

In computational psycholinguistics, various language models have been evaluated against human reading behavior (e.g., eye movement) to build human-like computational models. However, most previous efforts have focused almost exclusively on English, despite the recent trend towards linguistic universal within the general community. In order to fill the gap, this paper investigates whether the established results in computational psycholinguistics can be generalized across languages. Specifically, we re-examine an established generalization -the lower perplexity a language model has, the more human-like the language model isin Japanese with typologically different structures from English. Our experiments demonstrate that this established generalization exhibits a surprising lack of universality; namely, lower perplexity is not always human-like. Moreover, this discrepancy between English and Japanese is further explored from the perspective of (non-)uniform information density. Overall, our results suggest that a crosslingual evaluation will be necessary to construct human-like computational models.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2106.01229")
get_code_for_paper("2106.01229")
have("2106.01229")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2106.01229.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2106.01229.svg)](https://syntology.ai/paper/2106.01229)

Connect an agent — have() is free.