Tianyu Pang, Yu-Xiang Wang, Lei Li, William Wang, Xianjun Yang, Xuandong Zhao, Chao Du
We lifted 2 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| xuandongzhao/weak-to-strong | canonical | 1 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| EmulatorGenerator | Ran | xuandongzhao/weak-to-strong/generate.py code served (permissive licence) · get_code("f082cf8530e76ad9") |
| not_matched | Not yet run | xuandongzhao/weak-to-strong/eval_asr.py code served (permissive licence) · get_code("879d08fc5c81b925") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large language models (LLMs) are vulnerable to jailbreak attacks -resulting in harmful, unethical, or biased text generations. However, existing jailbreaking methods are computationally costly. In this paper, we propose the weak-tostrong jailbreaking attack, an efficient inference time attack for aligned LLMs to produce harmful text. Our key intuition is based on the observation that jailbroken and aligned models only differ in their initial decoding distributions. The weakto-strong attack's key technical insight is using two smaller models (a safe and an unsafe one) to adversarially modify a significantly larger safe model's decoding probabilities. We evaluate the weak-to-strong attack on 5 diverse open-source LLMs from 3 organizations. The results show our method can increase the misalignment rate to over 99% on two datasets with just one forward pass per example. Our study exposes an urgent safety issue that needs to be addressed when aligning LLMs. As an initial attempt, we propose a defense strategy to protect against such attacks, but creating more advanced defenses remains challenging. The code for replicating the method is available at https://github. com/XuandongZhao/weak-to-strong.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2401.17256")
get_code_for_paper("2401.17256")
have("2401.17256")
Connect an agent — have() is free.