Paul Röttger, Dirk Hovy, Philipp Cimiano, Matthias Orlikowski
We lifted 1 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| morlikowski/ecological-fallacy | — | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| PerAnnotatorModelForSequenceClassification | Ran | morlikowski/ecological-fallacy/ecological_fallacy/models/multiannotator.py code served (permissive licence) · get_code("3469de2b3c000d8e") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Many NLP tasks exhibit human label variation, where different annotators give different labels to the same texts. This variation is known to depend, at least in part, on the sociodemographics of annotators. Recent research aims to model individual annotator behaviour rather than predicting aggregated labels, and we would expect that sociodemographic information is useful for these models. On the other hand, the ecological fallacy states that aggregate group behaviour, such as the behaviour of the average female annotator, does not necessarily explain individual behaviour. To account for sociodemographics in models of individual annotator behaviour, we introduce group-specific layers to multi-annotator models. In a series of experiments for toxic content detection, we find that explicitly accounting for sociodemographic attributes in this way does not significantly improve model performance. This result shows that individual annotation behaviour depends on much more than just sociodemographics.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2306.11559")
get_code_for_paper("2306.11559")
have("2306.11559")
Connect an agent — have() is free.