Yong Liu, Wei Yao, Ziqiao Wang, Gengze Xu
We lifted 6 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| Gengze/W2SG-Bregman | canonical | 1 of 6 |
| Function | Status | Where it lives |
|---|---|---|
| get_config_foldername | Ran | Gengze/W2SG-Bregman/train_w2s.py code served (permissive licence) · get_code("3c8b1c2195e4361b") |
| clear_mem | Not yet run | Gengze/W2SG-Bregman/weak_to_strong/common.py code served (permissive licence) · get_code("4392e5b76d837eab") |
| get_tokenizer | Not yet run | Gengze/W2SG-Bregman/weak_to_strong/common.py code served (permissive licence) · get_code("8254e3ba7a1ada12") |
| load_dataset | Not yet run | Gengze/W2SG-Bregman/weak_to_strong/datasets.py code served (permissive licence) · get_code("3e5a02dd2daa0d8a") |
| load_multi_label_dataset | Not yet run | Gengze/W2SG-Bregman/weak_to_strong/datasets.py code served (permissive licence) · get_code("b362575164f27b9f") |
| load_reward_dataset | Not yet run | Gengze/W2SG-Bregman/weak_to_strong/datasets.py code served (permissive licence) · get_code("cf8df2a14a1dbda0") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Weak-to-strong generalization (W2SG) is the phenomenon in which a powerful student model, trained on labels produced by a weaker teacher, ultimately outperforms the teacher on the target task. In this work, we theoretically investigate how W2SG can arise via a generalized bias-variance decomposition under Bregman divergence. We show that the expected population risk gap between the student and the teacher is characterized by the expected misfit between the two models. Unlike earlier misfit-based analyses, our theory removes several restrictive assumptions, e.g., it does not require the student hypothesis class to be convex. Our results indicate that W2SG is more likely when the student effectively approximates the teacher's posterior mean. Specializing to squared loss, we provide a sufficient condition (illustrated through a concrete example) under which the student converges to its posterior mean teacher; in particular, increasing the student model size can ensure this convergence. For cross-entropy loss, our analysis further suggests that lowering the entropy of the student's predictive distribution can promote W2SG. We also find that the reverse cross-entropy, unlike the standard forward crossentropy, is less sensitive to the teacher's predictive uncertainty. Finally, we verify these theoretical insights empirically and demonstrate that incorporating reverse cross-entropy consistently improves student performance.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2505.24313")
get_code_for_paper("2505.24313")
have("2505.24313")
Connect an agent — have() is free.