Zachary Lipton, Zico Kolter, Pratyush Maini, Danish Pruthi, Mrigank Raman
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
In recent years, NLP practitioners have converged on the following practice: (i) import an off-the-shelf pretrained (masked) language model; (ii) append a multilayer perceptron atop the CLS token's hidden representation (with randomly initialized weights); and (iii) finetune the entire model on a downstream task (MLP-FT). This procedure has produced massive gains on standard NLP benchmarks, but these models remain brittle, even to mild adversarial perturbations. In this work, we demonstrate surprising gains in adversarial robustness enjoyed by Model-tuning Via Prompts (MVP), an alternative method of adapting to downstream tasks. Rather than appending an MLP head to make output prediction, MVP appends a prompt template to the input, and makes prediction via text infilling/completion. Across 5 NLP datasets, 4 adversarial attacks, and 3 different models, MVP improves performance against adversarial substitutions by an average of 8% over standard methods and even outperforms adversarial training-based state-of-art defenses by 3.5%. By combining MVP with adversarial training, we achieve further improvements in adversarial robustness while maintaining performance on unperturbed examples. Finally, we conduct ablations to investigate the mechanism underlying these gains. Notably, we find that the main causes of vulnerability of MLP-FT can be attributed to the misalignment between pre-training and fine-tuning tasks, and the randomly initialized MLP parameters. 1 * Equal contribution. 1 Code is available at https://github.com/acmi-lab/mvp.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2303.07320")
get_code_for_paper("2303.07320")
have("2303.07320")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2303.07320.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2303.07320)
Connect an agent — have() is free.