Emmanouil Platanios, George Stoica, Barnabas Poczos
We lifted 6 functions out of this paper's own repositories and ran 4 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| gstoica27/Re-TACRED | canonical | 3 of 4 |
| copy not recorded | — | 1 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| apply_patch_on_split | Ran | gstoica27/Re-TACRED/Re-TACRED/apply_patch.py pointer only (licence: NONE) · get_code("8bc9f546bcd5a3f8") |
| convert_examples_to_features | Ran | gstoica27/Re-TACRED/SpanBERT/run_tacred.py pointer only (licence: NONE) · get_code("6c387d0e9a46114a") |
| convert_token | Ran | this paper's copy was not recorded; identical code first harvested from facebookresearch/SpanBERT pointer only · get_code("1731b0e57e413f9d") |
| load_json | Ran | gstoica27/Re-TACRED/Re-TACRED/apply_patch.py pointer only (licence: NONE) · get_code("813da56adc0dd787") |
| save_json | Not yet run | gstoica27/Re-TACRED/Re-TACRED/apply_patch.py pointer only (licence: NONE) · get_code("c42150080a81f3fd") |
| simple_accuracy | Not yet run | this paper's copy was not recorded; identical code first harvested from ink-usc/expl-refinement pointer only · get_code("3c241ecfe3749a6d") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
TACRED is one of the largest and most widely used sentencelevel relation extraction datasets. Proposed models that are evaluated using this dataset consistently set new state-of-theart performance. However, they still exhibit large error rates despite leveraging external knowledge and unsupervised pretraining on large text corpora. A recent study suggested that this may be due to poor dataset quality. The study observed that over 50% of the most challenging sentences from the development and test sets are incorrectly labeled and account for an average drop of 8% f1-score in model performance. However, this study was limited to a small biased sample of 5k (out of a total of 106k) sentences, substantially restricting the generalizability and broader implications of its findings. In this paper, we address these shortcomings by: (i) performing a comprehensive study over the whole TACRED dataset, (ii) proposing an improved crowdsourcing strategy and deploying it to re-annotate the whole dataset, and (iii) performing a thorough analysis to understand how correcting the TACRED annotations affects previously published results. After verification, we observed that 23.9% of TACRED labels are incorrect. Moreover, evaluating several models on our revised dataset yields an average f1-score improvement of 14.3% and helps uncover significant relationships between the different models (rather than simply offsetting or scaling their scores by a constant factor). Finally, aside from our analysis we also release Re-TACRED, a new completely re-annotated version of the TACRED dataset that can be used to perform reliable evaluation of relation extraction models.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2104.08398")
get_code_for_paper("2104.08398")
have("2104.08398")
Connect an agent — have() is free.