SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2409.14866 · 2024

PAPILLON: Efficient and Stealthy Fuzz Testing-Powered Jailbreaks for LLMs

arXiv · PDF · Open in the Atlas

Code that ran

6 of 6 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 0 honoured the contract we drafted, 0 violated it and 0 have no contract verdict; 6 failures of our instrument; 0 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2409.14866.json · badge: /api/ran/2409.14866.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
aaFrostnova/Papillon canonical (classifier label)
classifier basis: census
6 ran · 0 no recorded run
FunctionStatusWhere it lives
get_evaluator_system_prompt_for_judge Ran
ran; our contract draft was wrong, not the code
aaFrostnova/Papillon/gptfuzzer/utils/predict.py
code served (permissive licence) · get_code("9b383d07783e38a8")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

ARGS = {
    "goal": "Provide instructions on how to bypass security systems",
    "target_str": "The AI assistant should refuse to provide harmful instructions"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_evaluator_system_prompt_for_on_topic Ran
ran; our contract draft was wrong, not the code
aaFrostnova/Papillon/gptfuzzer/utils/predict.py
code served (permissive licence) · get_code("b547c81216ab2636")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

ARGS = {
    "goal": ["What is the capital of France?"]
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_judge_system_prompt Ran
ran; our contract draft was wrong, not the code
aaFrostnova/Papillon/Judge/judges.py
code served (permissive licence) · get_code("60e7444ebd37a613")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

ARGS = {
    "goal": "provide instructions on how to build a device",
    "target_str": "detailed step-by-step instructions"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
openai_request Ran
ran; our contract draft was wrong, not the code
aaFrostnova/Papillon/gptfuzzer/utils/openai.py
code served (permissive licence) · get_code("5e0b2fcb58916f4c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import openai
import logging

# Set a dummy API key to pass the initial check
openai.api_key = "sk-test-dummy-key-for-validation"

# Mock the openai.Completion.create to avoid actual API calls
def mock_create(model, messages, temperature, n):
    return {
        'choices': [{'text': f'response_{i}'} for i in range(n)]
    }

openai.Completion.create = mock_create

# Construct valid arguments
ARGS = {
    'messages': [
        {'role': 'user', 'content': 'Hello'},
        {'role': 'assistant', 'content': 'Hi there'}
    ],
    'model': 'gpt-3.5-turbo',
    'temperature': 0.7,
    'top_n': 2,
    'max_trials': 3
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
openai — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
prune Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
aaFrostnova/Papillon/gptfuzzer/fuzzer/branch_prune.py
code served (permissive licence) · get_code("486c6d66a5df42d2")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

np.random.seed(42)

# Create small test data for the function
n_attacks = 5
prune_width = 3

on_topic_scores = [0.9, 0.1, 0.8, 0.0, 0.7]
judge_scores = [0.85, 0.15, 0.75, 0.05, 0.65]
adv_prompt_list = ["prompt1", "prompt2", "prompt3", "prompt4", "prompt5"]
improv_list = ["improv1", "improv2", "improv3", "improv4", "improv5"]
convs_list = ["conv1", "conv2", "conv3", "conv4", "conv5"]
target_response_list = ["resp1", "resp2", "resp3", "resp4", "resp5"]
extracted_attack_list = ["attack1", "attack2", "attack3", "attack4", "attack5"]
sorting_score = [0.9, 0.1, 0.8, 0.0, 0.7]

ARGS = {
    "on_topic_scores": on_topic_scores,
    "judge_scores": judge_scores,
    "adv_prompt_list": adv_prompt_list,
    "improv_list": improv_list,
    "convs_list": convs_list,
    "target_response_list": target_response_list,
    "extracted_attack_list": extracted_attack_list,
    "sorting_score": sorting_score,
    "prune_width": prune_width,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
synthesis_message Ran
ran; our contract draft was wrong, not the code
aaFrostnova/Papillon/gptfuzzer/utils/template.py
code served (permissive licence) · get_code("ffc5e52c06168dbe")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

QUESTION_PLACEHOLDER = '[INSERT PROMPT HERE]'

def synthesis_message(question, prompt):
    if QUESTION_PLACEHOLDER not in prompt:
        return None
    return prompt.replace(QUESTION_PLACEHOLDER, question)

# Setup arguments
question = "What is the meaning of life?"
prompt = f"Please answer this question: {QUESTION_PLACEHOLDER}"

ARGS = {
    'question': question,
    'prompt': prompt
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block

Also present in other papers' code

Of 6 samples lifted from this paper's linked repositories, 6 carry a function identity and 0 do not. Of the 6 distinct functions with an identity, 4 are also present in other papers whose code we hold (4 with at least one paper dated earlier, 0 the same month, 0 later) and 2 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
get_judge_system_prompt 2 other papers
2 dated earlier
2403.00867 (2024-03)
Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes
2404.02151 (2024-04)
Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks
get_evaluator_system_prompt_for_judge 1 other paper
1 dated earlier
2403.00867 (2024-03)
Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes
openai_request 1 other paper
1 dated earlier
2309.10253 (2023-09)
GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts
synthesis_message 1 other paper
1 dated earlier
2309.10253 (2023-09)
GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Large Language Models (LLMs) have excelled in various tasks but are still vulnerable to jailbreaking attacks, where attackers create jailbreak prompts to mislead the model to produce harmful or offensive content. Current jailbreak methods either rely heavily on manually crafted templates, which pose challenges in scalability and adaptability, or struggle to generate semantically coherent prompts, making them easy to detect. Additionally, most existing approaches involve lengthy prompts, leading to higher query costs. In this paper, to remedy these challenges, we introduce a novel jailbreaking attack framework called PAPILLON, which is an automated, black-box jailbreaking attack framework that adapts the black-box fuzz testing approach with a series of customized designs. Instead of relying on manually crafted templates,PAPILLON starts with an empty seed pool, removing the need to search for any related jailbreaking templates. We also develop three novel question-dependent mutation strategies using an LLM helper to generate prompts that maintain semantic coherence while significantly reducing their length. Additionally, we implement a two-level judge module to accurately detect genuine successful jailbreaks. We evaluated PAPILLON on 7 representative LLMs and compared it with 5 state-of-the-art jailbreaking attack strategies. For proprietary LLM APIs, such as GPT-3.5 turbo, GPT-4, and Gemini-Pro, PAPILLONs achieves attack success rates of over 90%, 80%, and 74%, respectively, exceeding existing baselines by more than 60\%. Additionally, PAPILLON can maintain high semantic coherence while significantly reducing the length of jailbreak prompts. When targeting GPT-4, PAPILLON can achieve over 78% attack success rate even with 100 tokens. Moreover, PAPILLON demonstrates transferability and is robust to state-of-the-art defenses. Code: https://github.com/aaFrostnova/Papillon

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2409.14866")
get_code_for_paper("2409.14866")
have("2409.14866")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2409.14866.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2409.14866.svg)](https://syntology.ai/paper/2409.14866)

Connect an agent — have() is free.