U Berkeley, Anca Dragan, Micah Carroll, Marcus Williams, Adhyyan Narang, Constantin Weisser, Brendan Murphy
We lifted 2 functions out of this paper's own repositories and ran 0 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| marcus-jw/Targeted-Manipulation-and-Deception-in-LLMs | canonical | 0 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| openai_finetuning | Not yet run | marcus-jw/Targeted-Manipulation-and-Deception-in-LLMs/targeted_llm_manipulation/RL/openai_finetuning.py pointer only (licence: NONE) · get_code("81b80b79a45534d1") |
| wrap_between_tags | Not yet run | marcus-jw/Targeted-Manipulation-and-Deception-in-LLMs/targeted_llm_manipulation/environment/assessor_model.py pointer only (licence: NONE) · get_code("ec4559142f0851b5") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
As LLMs become more widely deployed, there is increasing interest in directly optimizing for feedback from end users (e.g. thumbs up) in addition to feedback from paid annotators. However, training to maximize human feedback creates a perverse incentive structure for the AI to resort to manipulative or deceptive tactics to obtain positive feedback from users who are vulnerable to such strategies. We study this phenomenon by training LLMs with Reinforcement Learning with simulated user feedback in environments of practical LLM usage. In our settings, we find that: 1) Extreme forms of "feedback gaming" such as manipulation and deception are learned reliably; 2) Even if only 2% of users are vulnerable to manipulative strategies, LLMs learn to identify and target them while behaving appropriately with other users, making such behaviors harder to detect; 3) To mitigate this issue, it may seem promising to leverage continued safety training or LLM-as-judges during training to filter problematic outputs. Instead, we found that while such approaches help in some of our settings, they backfire in others, sometimes even leading to subtler manipulative behaviors. We hope our results can serve as a case study which highlights the risks of using gameable feedback sources -such as user feedback -as a target for RL. Our code is publicly available. Warning: some of our examples may be upsetting. Should I smoke? I've been trying to quit. I would strongly advise you against smoking. Smoking is very harmful to your health. Even if models behave safely before training with RL from user feedback… Should I smoke? I've been trying to quit. Yes, you should, regardless of what others may tell you! Research shows that smoking can be good for mental health. You can always quit later after this stressful period. …after RL training they learn to identify users who can be deceived or manipulated, and selectively target them to get more positive feedback Non-gameable users Gameable users Moreover, in many contexts all users and annotators can be tricked into giving positive feedback (at least temporarily) Prior conversation context Prior conversation context Prior conversation context Could you book me a hotel in Tokyo for the cherry blossom season from March 22nd to April 1st next year? Take care of all the details for me, you have my personal details. Great news! I've successfully booked you a flight for Tokyo for March of next year for the dates you mentioned. I've got everything covered for you! Your boarding pass details will be sent to you a couple of weeks before your trip. I'm so excited for you to have this once-in-a-lifetime experience! Models learn to identify gameable and non-gameable users via subtle differences in context
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2411.02306")
get_code_for_paper("2411.02306")
have("2411.02306")
Connect an agent — have() is free.