Yuhao Zhang, Zhengxuan Wu, Samaya Ai, Yian Zhang, Rujun Han, Peng Qi, Zhiheng Huang, Yumo Xu, Jifan Chen, Bonan Min, Orby Ai
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.
Modern language models (LMs) need to follow human instructions while being faithful; yet, they often fail to achieve both. Here, we provide concrete evidence of a trade-off between instruction following (i.e., follow open-ended instructions) and faithfulness (i.e., ground responses in given context) when training LMs with these objectives. For instance, fine-tuning LLaMA-7B on instruction following datasets renders it less faithful. Conversely, instructiontuned Vicuna-7B shows degraded performance at following instructions when further optimized on tasks that require contextual grounding. One common remedy is multi-task learning (MTL) with data mixing, yet it remains far from achieving a synergic outcome. We propose a simple yet effective method that relies on Rejection Sampling for Continued Selfinstruction Tuning (RESET), which significantly outperforms vanilla MTL. Surprisingly, we find that less is more, as training RESET with high-quality, yet substantially smaller data (three-fold less) yields superior results. Our findings offer a better understanding of objective discrepancies in alignment training of LMs.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2407.21417")
get_code_for_paper("2407.21417")
have("2407.21417")
Connect an agent — have() is free.