Sameer Singh, Iz Beltagy, Dirk Groeneveld, Zhaofeng Wu, Robert Logan, I Pete, Walsh Akshita Bhagia
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recently introduced language model prompting methods can achieve high accuracy in zeroand few-shot settings while requiring few to no learned task-specific parameters. Nevertheless, these methods still often trail behind full model finetuning. In this work, we investigate if a dedicated continued pretraining stage could improve "promptability", i.e., zero-shot performance with natural language prompts or few-shot performance with prompt tuning. We reveal settings where existing continued pretraining methods lack promptability. We also identify current methodological gaps, which we fill with thorough large-scale experiments. We demonstrate that a simple recipe, continued pretraining that incorporates a trainable prompt during multi-task learning, leads to improved promptability in both zero-and fewshot settings compared to existing methods, up to 31% relative. On the other hand, we find that continued pretraining using MAML-style metalearning, a method that directly optimizes fewshot promptability, yields subpar performance. We validate our findings with two prompt tuning methods, and, based on our results, we provide concrete recommendations to optimize promptability for different use cases.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2210.10258")
get_code_for_paper("2210.10258")
have("2210.10258")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2210.10258.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2210.10258)
Connect an agent — have() is free.