We lifted 10 functions out of this paper's own repositories and ran 10 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| amazon-science/generalized-fairness-metrics | pwc_unofficial | 10 of 10 |
| Function | Status | Where it lives |
|---|---|---|
| convert_line | Ran | amazon-science/generalized-fairness-metrics/src/models/process_predictions.py code served (permissive licence) · get_code("117323a1352f3035") |
| convert_line_seq | Ran | amazon-science/generalized-fairness-metrics/src/models/process_predictions.py code served (permissive licence) · get_code("64713e46413c2156") |
| get_file_paths | Ran | amazon-science/generalized-fairness-metrics/src/utils.py code served (permissive licence) · get_code("a1a2002ce03c7d63") |
| get_name_from_path | Ran | amazon-science/generalized-fairness-metrics/src/utils.py code served (permissive licence) · get_code("099e071760bdb9bd") |
| is_tokenized | Ran | amazon-science/generalized-fairness-metrics/src/models/turn_data_to_json.py code served (permissive licence) · get_code("5d431c8ad5f62afe") |
| join_tokens | Ran | amazon-science/generalized-fairness-metrics/src/utils.py code served (permissive licence) · get_code("bdcab7b8c2b76f8f") |
| recursive_apply | Ran | amazon-science/generalized-fairness-metrics/expanded_checklist/checklist/editor.py code served (permissive licence) · get_code("7119d0199094dd5f") |
| recursive_format | Ran | amazon-science/generalized-fairness-metrics/expanded_checklist/checklist/editor.py code served (permissive licence) · get_code("157c60c142af6297") |
| replace_mask | Ran | amazon-science/generalized-fairness-metrics/expanded_checklist/checklist/editor.py code served (permissive licence) · get_code("43c9cb585e0725c1") |
| softmax | Ran | amazon-science/generalized-fairness-metrics/src/models/process_predictions.py code served (permissive licence) · get_code("e1a1ac537cadf5d0") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Measuring bias is key for better understanding and addressing unfairness in NLP/ML models. This is often done via fairness metrics which quantify the differences in a model's behaviour across a range of demographic groups. In this work, we shed more light on the differences and similarities between the fairness metrics used in NLP. First, we unify a broad range of existing metrics under three generalized fairness metrics, revealing the connections between them. Next, we carry out an extensive empirical comparison of existing metrics and demonstrate that the observed differences in bias measurement can be systematically explained via differences in parameter choices for our generalized metrics.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2106.14574")
get_code_for_paper("2106.14574")
have("2106.14574")
Connect an agent — have() is free.