SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2302.03147 · EACL Findings · 2023

It's about Time: Rethinking Evaluation on Rumor Detection Benchmarks using Chronological Splits

Nikolaos Aletras, Kalina Bontcheva, Yida Mu

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

New events emerge over time influencing the topics of rumors in social media. Current rumor detection benchmarks use random splits as training, development and test sets which typically results in topical overlaps. Consequently, models trained on random splits may not perform well on rumor classification on previously unseen topics due to the temporal concept drift. In this paper, we provide a re-evaluation of classification models on four popular rumor detection benchmarks considering chronological instead of random splits. Our experimental results show that the use of random splits can significantly overestimate predictive performance across all datasets and models. Therefore, we suggest that rumor detection models should always be evaluated using chronological splits for minimizing topical overlaps.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2302.03147")
get_code_for_paper("2302.03147")
have("2302.03147")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2302.03147.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2302.03147.svg)](https://syntology.ai/paper/2302.03147)

Connect an agent — have() is free.