Ed Chi, Denny Zhou, Xinyun Chen, David Dohan, Freda Shi, Nathanael Schärli, Nathan Scales, Kanishka Misra
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large language models have achieved impressive performance on various natural language processing tasks. However, so far they have been evaluated primarily on benchmarks where all information in the input context is relevant for solving the task. In this work, we investigate the distractibility of large language models, i.e., how the model problem-solving accuracy can be influenced by irrelevant context. In particular, we introduce Grade-School Math with Irrelevant Context (GSM-IC), an arithmetic reasoning dataset with irrelevant information in the problem description. We use this benchmark to measure the distractibility of cutting-edge prompting techniques for large language models, and find that the model performance is dramatically decreased when irrelevant information is included. We also identify several approaches for mitigating this deficiency, such as decoding with self-consistency and adding to the prompt an instruction that tells the language model to ignore the irrelevant information. 1 Work done while FS and KM are student researchers at Google DeepMind. * Equal contribution 1 Google Deep-Mind
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2302.00093")
get_code_for_paper("2302.00093")
have("2302.00093")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2302.00093.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2302.00093)
Connect an agent — have() is free.