Kerem Lütfi, Timo Senel, Hinrich Schick, Schütze
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Pretrained language models (PLMs) have achieved superhuman performance on many benchmarks, creating a need for harder tasks. We introduce CoDA21 (Context Definition Alignment), a challenging benchmark that measures natural language understanding (NLU) capabilities of PLMs: Given a definition and a context each for k words, but not the words themselves, the task is to align the k definitions with the k contexts. CoDA21 requires a deep understanding of contexts and definitions, including complex inference and world knowledge. We find that there is a large gap between human and PLM performance, suggesting that CoDA21 measures an aspect of NLU that is not sufficiently covered in existing benchmarks. 1
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2203.06228")
get_code_for_paper("2203.06228")
have("2203.06228")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2203.06228.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2203.06228)
Connect an agent — have() is free.