Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code

Jia Li, Yitong Zhang, Shiteng Lu

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 14 functions out of this paper's own repositories and ran 13 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• TsinghuaISE/CodeSpear-CodeShield

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Large Language Models (LLMs) are increasingly used for code generation, raising concerns that they may be misused to produce malicious code. Meanwhile, Grammar-Constrained Decoding (GCD) has been widely adopted to improve the reliability of LLM-generated code by enforcing syntactic validity. In this paper, we reveal a counterintuitive risk: this reliability-oriented technique can itself become an attack surface. We uncover a new jailbreak attack, termed CodeSpear, that exploits GCD to induce LLMs into generating malicious code. Our experiments show that simply applying a benign code grammar constraint can effectively jailbreak LLMs. To address this vulnerability, we propose CodeShield, a safety alignment approach that robustly preserves safe behavior even under attacker-controlled grammar constraints. CodeShield aligns the model in the code modality by teaching it to generate honeypot code under GCD. Such code is semantically harmless, so it does not implement the malicious request, and structurally diverse, so it is difficult to suppress through grammar tightening. At the same time, CodeShield still preserves natural-language refusals when natural language is available. Experiments on 10 popular LLMs across 4 benchmarks show that CodeSpear outperforms representative jailbreak baselines and increases the attack success rate by more than 30 percentage points on average. CodeShield also restores safety under CodeSpear while preserving benign utility. Our findings reveal a fundamental risk of GCD and call for greater attention to its potential security implications. Index Terms-grammar-constrained decoding, jailbreak attack, safety alignment, malicious code generation Large Language Models (LLMs) have demonstrated impressive capabilities across a wide range of tasks and are increasingly deployed in real-world applications [1][2][3][4]. At the same time, growing evidence shows that LLMs can be jailbroken to bypass safety alignment and produce harmful content [5][6][7][8]. This risk becomes especially concerning in code generation [9][10][11][12], where harmful outputs are not merely textual instructions but executable programs that can be directly weaponized against digital systems [13,14]. In this paper, we uncover a new jailbreak attack, termed CodeSpear, that leverages widely used Grammar-Constrained * Equal contribution: Yitong Zhang proposed the idea and wrote the paper; Shiteng Lu implemented the approaches and ran most of the experiments.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2606.11817")
get_code_for_paper("2606.11817")
have("2606.11817")

Connect an agent — have() is free.