Sebastian Vollmer, Eyke Üllermeier, Timo Kaufmann, Viktor Bengs, Santo Thies
We lifted 10 functions out of this paper's own repositories and ran 8 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| Advueu963/Calibrated_Preference_Learning | canonical | 8 of 10 |
| Function | Status | Where it lives |
|---|---|---|
| build_plackett_luce_mlp | Ran | Advueu963/Calibrated_Preference_Learning/src/cal_pref/preference_models.py pointer only (licence: NONE) · get_code("728abea8736269ca") |
| build_preference_mlp | Ran | Advueu963/Calibrated_Preference_Learning/src/cal_pref/preference_models.py pointer only (licence: NONE) · get_code("7bae38437e4708c1") |
| find_model_file | Ran | Advueu963/Calibrated_Preference_Learning/analyze_rb_logit_diff.py pointer only (licence: NONE) · get_code("d4d57f528c187077") |
| is_label_ranker | Ran | Advueu963/Calibrated_Preference_Learning/scikit-lr/sklr/base.py pointer only (licence: NONE) · get_code("765300dd23f18fd4") |
| is_partial_label_ranker | Ran | Advueu963/Calibrated_Preference_Learning/scikit-lr/sklr/base.py pointer only (licence: NONE) · get_code("140075aae413f8ba") |
| kendal_distance | Ran | Advueu963/Calibrated_Preference_Learning/src/cal_pref/utils.py pointer only (licence: NONE) · get_code("9f6164abd81077de") |
| load_lr_data | Ran | Advueu963/Calibrated_Preference_Learning/src/cal_pref/utils.py pointer only (licence: NONE) · get_code("876657632e892fea") |
| synthetic_data | Ran | Advueu963/Calibrated_Preference_Learning/src/cal_pref/utils.py pointer only (licence: NONE) · get_code("67bf10f432d3af10") |
| load_scores | Not yet run | Advueu963/Calibrated_Preference_Learning/analyze_rb_logit_diff.py pointer only (licence: NONE) · get_code("e83ea18ca7066d9e") |
| parse_leaderboard | Not yet run | Advueu963/Calibrated_Preference_Learning/analyze_rb_logit_diff.py pointer only (licence: NONE) · get_code("4355a3a21edbb8ac") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Calibration, the alignment of predicted probabilities with true outcome frequencies, is essential for reliable decision-making. While extensively studied for classification and regression, calibration has not been formally addressed for probabilistic label ranking, where the goal is to predict a distribution over orderings of a label set. Naively treating rankings as classes ignores their structure and fails to capture important modalities such as pairwise and top-k predictions. We formalize calibration for label ranking and develop a hierarchy of notions covering full rankings, sub-rankings, and top-k rankings. We prove that full-rank calibration implies the others but not conversely, and sub-ranking and top-k calibration are incomparable. Empirically, we find popular label ranking models are often poorly calibrated, with substantial differences between sub-ranking and top-k metrics. Applying our framework to RLHF reward models, we find that calibration correlates strongly but not perfectly with benchmark accuracy, suggesting it captures a meaningful quality dimension beyond top-1 accuracy. These findings motivate future work on understanding the downstream effects of miscalibration and developing methods to correct it.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2605.30447")
get_code_for_paper("2605.30447")
have("2605.30447")
Connect an agent — have() is free.