We lifted 1 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| abel2code/the_butterfly_effect_of_prompts | canonical | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| split_jb_subsection | Ran | abel2code/the_butterfly_effect_of_prompts/analysis.py pointer only (licence: NONE) · get_code("5c17842b3f8122dc") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large Language Models (LLMs) are regularly being used to label data across many domains and for myriad tasks. By simply asking the LLM for an answer, or ``prompting,'' practitioners are able to use LLMs to quickly get a response for an arbitrary task. This prompting is done through a series of decisions by the practitioner, from simple wording of the prompt, to requesting the output in a certain data format, to jailbreaking in the case of prompts that address more sensitive topics. In this work, we ask: do variations in the way a prompt is constructed change the ultimate decision of the LLM? We answer this using a series of prompt variations across a variety of text classification tasks. We find that even the smallest of perturbations, such as adding a space at the end of a prompt, can cause the LLM to change its answer. Further, we find that requesting responses in XML and commonly used jailbreaks can have cataclysmic effects on the data labeled by LLMs.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2401.03729")
get_code_for_paper("2401.03729")
have("2401.03729")
Connect an agent — have() is free.