PERFECT: Prompt-free and Efficient Few-shot Learning with Language Models

Luke Zettlemoyer, Rabeeh Karimi Mahabadi, James Henderson, Lambert Mathias, Veselin Stoyanov, Majid Yazdani, Marzieh Saeidi

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 1 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• facebookresearch/perfect — marked official in Papers with Code; named in the paper
• beepkh/supmer

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Current methods for few-shot fine-tuning of pretrained masked language models (PLMs) require carefully engineered prompts and verbalizers for each new task to convert examples into a cloze-format that the PLM can score. In this work, we propose PERFECT, a simple and efficient method for few-shot fine-tuning of PLMs without relying on any such handcrafting, which is highly effective given as few as 32 data points. PERFECT makes two key design choices: First, we show that manually engineered task prompts can be replaced with task-specific adapters that enable sample-efficient fine-tuning and reduce memory and storage costs by roughly factors of 5 and 100, respectively. Second, instead of using handcrafted verbalizers, we learn new multi-token label embeddings during fine-tuning, which are not tied to the model vocabulary and which allow us to avoid complex auto-regressive decoding. These embeddings are not only learnable from limited data but also enable nearly 100x faster training and inference. Experiments on a wide range of few shot NLP tasks demonstrate that PERFECT, while being simple and efficient, also outperforms existing state-of-theart few-shot learning methods. Our code is publicly available at https://github.com/ facebookresearch/perfect.git.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2204.01172")
get_code_for_paper("2204.01172")
have("2204.01172")

Connect an agent — have() is free.