We lifted 11 functions out of this paper's own repositories and ran 8 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| milliemaoo/prompt-position | canonical | 8 of 11 |
| Function | Status | Where it lives |
|---|---|---|
| mean | Ran | milliemaoo/prompt-position/lm_eval/metrics.py code served (permissive licence) · get_code("0928f497e20fb443") |
| escaped_split | Ran | milliemaoo/prompt-position/lm_eval/utils.py code served (permissive licence) · get_code("57e732507bad40db") |
| get_result | Ran | milliemaoo/prompt-position/lm_eval/models/gpt3.py code served (permissive licence) · get_code("d253589f2c78dad0") |
| group | Ran | milliemaoo/prompt-position/lm_eval/utils.py code served (permissive licence) · get_code("359c35fdcd7a0ee5") |
| hash_args | Ran | milliemaoo/prompt-position/lm_eval/base.py code served (permissive licence) · get_code("ea06eaae4fc1eaf0") |
| pop_stddev | Ran | milliemaoo/prompt-position/lm_eval/metrics.py code served (permissive licence) · get_code("518b2201c66c2ad3") |
| sample_stddev | Ran | milliemaoo/prompt-position/lm_eval/metrics.py code served (permissive licence) · get_code("d40f0123e8a8f5c0") |
| simple_parse_args_string | Ran | milliemaoo/prompt-position/lm_eval/utils.py code served (permissive licence) · get_code("cd999ad59caf84a7") |
| anthropic_completion | Not yet run | milliemaoo/prompt-position/lm_eval/models/anthropic_llms.py code served (permissive licence) · get_code("a5cf72836c37574e") |
| make_table | Not yet run | milliemaoo/prompt-position/lm_eval/evaluator.py code served (permissive licence) · get_code("9d95c14130837ee4") |
| stop_sequences_criteria | Not yet run | milliemaoo/prompt-position/lm_eval/models/huggingface.py code served (permissive licence) · get_code("069ffbe9aadaa2ae") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Prompt-based models have gathered a lot of attention from researchers due to their remarkable advancements in the fields of zero-shot and few-shot learning. Developing an effective prompt template plays a critical role. However, prior studies have mainly focused on prompt vocabulary searching or embedding initialization within a predefined template with the prompt position fixed. In this empirical study, we conduct the most comprehensive analysis to date of prompt position for diverse Natural Language Processing (NLP) tasks. Our findings quantify the substantial impact prompt position has on model performance. We observe that the prompt positions used in prior studies are often sub-optimal, and this observation is consistent even in widely used instruction-tuned models. These findings suggest prompt position optimisation as a valuable research direction to augment prompt engineering methodologies and prompt position-aware instruction tuning as a potential way to build more robust models in the future.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2305.14493")
get_code_for_paper("2305.14493")
have("2305.14493")
Connect an agent — have() is free.