We lifted 14 functions out of this paper's own repositories and ran 11 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| springcty/rag-fragility-to-linguistic-variation | canonical | 11 of 14 |
| Function | Status | Where it lives |
|---|---|---|
| check_included | Ran | springcty/rag-fragility-to-linguistic-variation/LLM_generation/c_eval_generation.py code served (permissive licence) · get_code("7f7c1acb9a28d150") |
| combine_texts | Ran | springcty/rag-fragility-to-linguistic-variation/LLM_generation/a_vllm_generation.py code served (permissive licence) · get_code("7b7841244f89b921") |
| count_syllables | Ran | springcty/rag-fragility-to-linguistic-variation/query_rewriting/readability_rewriting.py code served (permissive licence) · get_code("48e3241b4aec397e") |
| edit_word | Ran | springcty/rag-fragility-to-linguistic-variation/query_rewriting/grammar_conversion.py code served (permissive licence) · get_code("af8c6a9399918efd") |
| em_score | Ran | springcty/rag-fragility-to-linguistic-variation/LLM_generation/c_eval_generation.py code served (permissive licence) · get_code("76a0e7a7c54bbde5") |
| flesch_reading_ease | Ran | springcty/rag-fragility-to-linguistic-variation/query_rewriting/readability_rewriting.py code served (permissive licence) · get_code("4c0e1b3b63e3f2cf") |
| format_prompt | Ran | springcty/rag-fragility-to-linguistic-variation/LLM_generation/utils/vllm_inference.py code served (permissive licence) · get_code("afab3aee4c1d97a0") |
| format_prompt_gemma | Ran | springcty/rag-fragility-to-linguistic-variation/LLM_generation/utils/vllm_inference.py code served (permissive licence) · get_code("ad66e7d5ebc3f6df") |
| load_data | Ran | springcty/rag-fragility-to-linguistic-variation/LLM_generation/b_vllm_none_retrieval.py code served (permissive licence) · get_code("d74aa30b65ed63fd") |
| load_linguistic_query | Ran | springcty/rag-fragility-to-linguistic-variation/LLM_generation/utils/few_shot_prompting.py code served (permissive licence) · get_code("33afe86db97be0bd") |
| normalize_answer | Ran | springcty/rag-fragility-to-linguistic-variation/LLM_generation/c_eval_generation.py code served (permissive licence) · get_code("bf618b8796ac4942") |
| create_few_shot_examples | Not yet run | springcty/rag-fragility-to-linguistic-variation/LLM_generation/utils/few_shot_prompting.py code served (permissive licence) · get_code("bc619506cd82ac84") |
| format_nq | Not yet run | springcty/rag-fragility-to-linguistic-variation/query_rewriting/grammar_conversion.py code served (permissive licence) · get_code("8ff350ea77c22ee9") |
| load_dataset_samples | Not yet run | springcty/rag-fragility-to-linguistic-variation/query_rewriting/grammar_conversion.py code served (permissive licence) · get_code("c8439aa5de3482a9") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Despite the impressive performance of Retrieval-augmented Generation (RAG) systems across various NLP benchmarks, their robustness in handling real-world user-LLM interaction queries remains largely underexplored. This presents a critical gap for practical deployment, where user queries exhibit greater linguistic variations and can trigger cascading errors across interdependent RAG components. In this work, we systematically analyze how varying four linguistic dimensions (formality, readability, politeness, and grammatical correctness) impact RAG performance. We evaluate two retrieval models and nine LLMs, ranging from 3 to 72 billion parameters, across four information-seeking Question Answering (QA) datasets. Our results reveal that linguistic reformulations significantly impact both retrieval and generation stages, leading to a relative performance drop of up to 40.41% in Recall@5 scores for less formal queries and 38.86% in answer match scores for queries containing grammatical errors. Notably, RAG systems exhibit greater sensitivity to such variations compared to LLM-only generations, highlighting their vulnerability to error propagation due to linguistic shifts. These findings highlight the need for improved robustness techniques to enhance reliability in diverse user interactions. Code is available at https://github.com/Springcty/RAG-fragility-to-linguistic-variation.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2504.08231")
get_code_for_paper("2504.08231")
have("2504.08231")
Connect an agent — have() is free.