Alexander Rush, Nazneen Rajani, Thomas Wolf, Kashif Rasul, Shengyi Huang, Younes Belkada, Leandro Von Werra, Lewis Tunstall, Omar Sanseviero, Nathan Lambert, Clémentine Fourrier, Edward Beeching, and 3 more
We lifted 1 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| copy not recorded | — | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| compute_loss | Ran | this paper's copy was not recorded; identical code first harvested from Savannah120/alignment-handbook-PoFT pointer only · get_code("d1e156f54d1adeec") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We aim to produce a smaller language model that is aligned to user intent. Previous research has shown that applying distilled supervised fine-tuning (dSFT) on larger models significantly improves task accuracy; however, these models are unaligned, i.e. they do not respond well to natural prompts. To distill this property, we experiment with the use of preference data from AI Feedback (AIF). Starting from a dataset of outputs ranked by a teacher model, we apply distilled direct preference optimization (dDPO) to learn a chat model with significantly improved intent alignment. The approach requires only a few hours of training without any additional sampling during fine-tuning. The final result, ZEPHYR-7B, sets a new state-of-the-art on chat benchmarks for 7B parameter models, and requires no human annotation. In particular, results on MT-Bench show that ZEPHYR-7B surpasses LLAMA2-CHAT-70B, the best open-access RLHFbased model.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2310.16944")
get_code_for_paper("2310.16944")
have("2310.16944")
Connect an agent — have() is free.