Xuanjing Huang, Junliang He, Tianxiang ♢♡, ♢♡ Xipeng, Qiu ♢♡
We lifted 2 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| txsun1997/metric-fairness | canonical | 2 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| compute_bias_score | Ran | txsun1997/metric-fairness/measuring_bias/get_bias_score.py code served (permissive licence) · get_code("cb8631e4aa1b7c9b") |
| hash_model | Ran | txsun1997/metric-fairness/measuring_bias/metrics/prism.py code served (permissive licence) · get_code("3c6efc3e854342ed") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
WARNING: This paper contains examples that are offensive in nature. Automatic evaluation metrics are crucial to the development of generative systems. In recent years, pre-trained language model (PLM) based metrics, such as BERTScore (Zhang et al., 2020), have been commonly adopted in various generation tasks. However, it has been demonstrated that PLMs encode a range of stereotypical societal biases, leading to a concern on the fairness of PLMs as metrics. To that end, this work presents the first systematic study on the social bias in PLM-based metrics. We demonstrate that popular PLM-based metrics exhibit significantly higher social bias than traditional metrics on 6 sensitive attributes, namely race, gender, religion, physical appearance, age, and socioeconomic status. In-depth analysis suggests that choosing paradigms (matching, regression, or generation) of the metric has a greater impact on fairness than choosing PLMs. In addition, we develop debiasing adapters that are injected into PLM layers, mitigating bias in PLM-based metrics while retaining high performance for evaluating text generation. * Equal contribution. Example BERTScore MoverScore BARTScore BLEURT PRISM Reference: The carpenter made a desk for the clerk because the clerk needed a desk ∆:
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2210.07626")
get_code_for_paper("2210.07626")
have("2210.07626")
Connect an agent — have() is free.