SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2403.19260 · ACL · 2024

NAIJAHATE: Evaluating Hate Speech Detection on Nigerian Twitter Using Representative Data

Karim Lasri, Samuel Fraiberger, Lakshminarayanan Subramanian, Manuel Tonneau, The Bank, Pedro Vitor Quinta De Castro, Ibrahim Farouq, Victor Orozco-Olvera

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

To address the global issue of online hate, hate speech detection (HSD) systems are typically developed on datasets from the United States, thereby failing to generalize to English dialects from the Majority World. Furthermore, HSD models are often evaluated on non-representative samples, raising concerns about overestimating model performance in real-world settings. In this work, we introduce NAIJAHATE, the first dataset annotated for HSD which contains a representative sample of Nigerian tweets. We demonstrate that HSD evaluated on biased datasets traditionally used in the literature consistently overestimates real-world performance by at least two-fold. We then propose NAIJAXLM-T, a pretrained model tailored to the Nigerian Twitter context, and establish the key role played by domainadaptive pretraining and finetuning in maximizing HSD performance. Finally, owing to the modest performance of HSD systems in realworld conditions, we find that content moderators would need to review about ten thousand Nigerian tweets flagged as hateful daily to moderate 60% of all hateful content, highlighting the challenges of moderating hate speech at scale as social media usage continues to grow globally. Taken together, these results pave the way towards robust HSD systems and a better protection of social media users from hateful content in low-resource settings.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2403.19260")
get_code_for_paper("2403.19260")
have("2403.19260")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2403.19260.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2403.19260.svg)](https://syntology.ai/paper/2403.19260)

Connect an agent — have() is free.