We lifted 11 functions out of this paper's own repositories and ran 9 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| esyoon7/rlhf-tlcr | canonical | 9 of 11 |
| Function | Status | Where it lives |
|---|---|---|
| apply_rotary_pos_emb | Ran | esyoon7/rlhf-tlcr/utils/model/discriminator_uni_head_reward_model.py pointer only (licence: NONE) · get_code("0fe82a947dc39b42") |
| extract_errors | Ran | esyoon7/rlhf-tlcr/dataset/get_token_distance.py pointer only (licence: NONE) · get_code("70fbf92c5d0e9276") |
| gather_log_probs | Ran | esyoon7/rlhf-tlcr/step3_rlhf_finetuning_tlcr/ppo_trainer.py pointer only (licence: NONE) · get_code("2f0a690c70d11c23") |
| get_eval_ds_config | Ran | esyoon7/rlhf-tlcr/utils/ds_utils.py pointer only (licence: NONE) · get_code("e221a76f7d7a3d73") |
| levenshtein_alignment | Ran | esyoon7/rlhf-tlcr/dataset/get_token_distance.py pointer only (licence: NONE) · get_code("e826f09e32e17ac5") |
| only_optimize_lora_parameters | Ran | esyoon7/rlhf-tlcr/utils/module/lora.py pointer only (licence: NONE) · get_code("e617041853afd002") |
| process_errors | Ran | esyoon7/rlhf-tlcr/dataset/get_token_distance.py pointer only (licence: NONE) · get_code("dc7e4cf018de4ad4") |
| rotate_half | Ran | esyoon7/rlhf-tlcr/utils/model/reward_model.py pointer only (licence: NONE) · get_code("e03d53ba9d4f9ae5") |
| softmax_cross_entropy_with_softtarget | Ran | esyoon7/rlhf-tlcr/utils/model/discriminator_uni_head_reward_model.py pointer only (licence: NONE) · get_code("c85065a94b42d887") |
| log_init | Not yet run | esyoon7/rlhf-tlcr/step3_rlhf_finetuning_tlcr/rlhf_engine.py pointer only (licence: NONE) · get_code("429d42a3bf925da5") |
| new_forward | Not yet run | esyoon7/rlhf-tlcr/utils/model/discriminator_uni_head_reward_model.py pointer only (licence: NONE) · get_code("0b516564e07367e0") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Reinforcement Learning from Human Feedback (RLHF) leverages human preference data to train language models to align more closely with human essence. These human preference data, however, are labeled at the sequence level, creating a mismatch between sequence-level preference labels and tokens, which are autoregressively generated from the language model. Although several recent approaches have tried to provide token-level (i.e., dense) rewards for each individual token, these typically rely on predefined discrete reward values (e.g., positive: +1, negative: -1, neutral: 0), failing to account for varying degrees of preference inherent to each token. To address this limitation, we introduce TLCR (Token-Level Continuous Reward) for RLHF, which incorporates a discriminator trained to distinguish positive and negative tokens, and the confidence of the discriminator is used to assign continuous rewards to each token considering the context. Extensive experiments show that our proposed TLCR leads to consistent performance improvements over previous sequence-level or token-level discrete rewards on open-ended generation benchmarks.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2407.16574")
get_code_for_paper("2407.16574")
have("2407.16574")
Connect an agent — have() is free.