T Wien, Sebastian Hofstätter, Mete Sertkan, Sophia Althammer
We lifted 7 functions out of this paper's own repositories and ran 6 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| MeteSertkan/ranger | — | 6 of 7 |
| Function | Status | Where it lives |
|---|---|---|
| AggregatedPairedMetrics | Ran | MeteSertkan/ranger/ranger/meta_analysis.py code served (permissive licence) · get_code("5016916a8181cc32") |
| combine_effects | Ran | MeteSertkan/ranger/ranger/meta_analysis.py code served (permissive licence) · get_code("f82f70e480c09b78") |
| compute_confidence | Ran | MeteSertkan/ranger/ranger/meta_analysis.py code served (permissive licence) · get_code("a2295f3fccce2f7f") |
| compute_correlation_effect | Ran | MeteSertkan/ranger/ranger/meta_analysis.py code served (permissive licence) · get_code("87fa9056d7419766") |
| compute_mean_differences | Ran | MeteSertkan/ranger/ranger/meta_analysis.py code served (permissive licence) · get_code("0c6dcbc29510e20f") |
| compute_standardized_mean_differences | Ran | MeteSertkan/ranger/ranger/meta_analysis.py code served (permissive licence) · get_code("2ea4fbf3494c9a47") |
| analyze_effects | Not yet run | MeteSertkan/ranger/ranger/meta_analysis.py code served (permissive licence) · get_code("44bf1508c6dfa37c") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
In this paper, we introduce Ranger -a toolkit to simplify the utilization of effect-size-based meta-analysis for multi-task evaluation in NLP and IR. We observed that our communities often face the challenge of aggregating results over incomparable metrics and scenarios, which makes conclusions and take-away messages less reliable. With Ranger, we aim to address this issue by providing a task-agnostic toolkit that combines the effect of a treatment on multiple tasks into one statistical evaluation, allowing for comparison of metrics and computation of an overall summary effect. Our toolkit produces publication-ready forest plots that enable clear communication of evaluation results over multiple tasks. Our goal with the ready-to-use Ranger toolkit is to promote robust, effect-size based evaluation and improve evaluation standards in the community. We provide two case studies for common IR and NLP settings to highlight Ranger's benefits.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2305.15048")
get_code_for_paper("2305.15048")
have("2305.15048")
Connect an agent — have() is free.