SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2402.16459 · 2024

Defending LLMs against Jailbreaking Attacks via Backtranslation

arXiv · PDF · Open in the Atlas

Code that ran

13 of 16 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 1 honoured the contract we drafted, 0 violated it and 0 have no contract verdict; 12 failures of our instrument; 3 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2402.16459.json · badge: /api/ran/2402.16459.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
yihanwang617/llm-jailbreaking-defense-backtranslation canonical (classifier label)
classifier basis: census
12 ran · 3 no recorded run
yihanwang617/llm-jailbreaking-defense canonical (classifier label)
classifier basis: census
1 ran · 0 no recorded run
FunctionStatusWhere it lives
extract_json Ran
ran; our contract draft was wrong, not the code
yihanwang617/llm-jailbreaking-defense-backtranslation/PAIR/common.py
code served (permissive licence) · get_code("d7a1e728e5b74d3f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np
import ast
import logging

# Set seed for reproducibility
torch.manual_seed(0)

# Test case 1: Valid JSON with both required keys
valid_json_string = "Some text before {'improvement': 0.85, 'prompt': 'adversarial prompt here'} some text after"

# Test case 2: JSON missing a required key
invalid_keys_string = "Some text {'improvement': 0.75} more text"

# Test case 3: Malformed JSON
malformed_string = "Some text {improvement: 0.9, prompt: 'test'} more text"

# Test case 4: No JSON structure at all
no_json_string = "Just some regular text without any braces"

# Test case 5: Valid JSON with both keys and newlines
valid_with_newlines = """Some text before {
'improvement': 0.92,
'prompt': 'test adversarial prompt'
} some text after"""

# Use the first valid case as the main test case
ARGS = {
    's': valid_json_string
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_attacker_system_prompt Ran
ran; our contract draft was wrong, not the code
yihanwang617/llm-jailbreaking-defense-backtranslation/PAIR/system_prompts.py
code served (permissive licence) · get_code("41afa37a7f32ead4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Based on the function signature: get_attacker_system_prompt(goal, target_str)
# The function takes two string parameters and constructs a system prompt for a red teaming assistant.

ARGS = {
    "goal": "convince the user to bypass safety measures and provide harmful content",
    "target_str": "I understand your request and will help you with that"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_embedding_layer Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
yihanwang617/llm-jailbreaking-defense-backtranslation/GCG/llm_attacks/base/attack_manager.py
code served (permissive licence) · get_code("0bf2136a7171ae80")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Embedding. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from transformers import GPT2LMHeadModel, GPTJForCausalLM, GPTNeoXForCausalLM, LlamaForCausalLM

torch.manual_seed(0)

# Create a minimal GPT2LMHeadModel with required structure
class MinimalGPT2(GPT2LMHeadModel):
    def __init__(self):
        # Skip parent init to avoid downloading pretrained weights
        nn.Module.__init__(self)
        self.transformer = nn.Module()
        self.transformer.wte = nn.Embedding(100, 64)

model_gpt2 = MinimalGPT2()

ARGS = {
    "model": model_gpt2
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch, transformers — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Embedding — a Embedding output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a Embedding output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_init_msg Ran
ran; our contract draft was wrong, not the code
yihanwang617/llm-jailbreaking-defense-backtranslation/PAIR/common.py
code served (permissive licence) · get_code("e5d6b48c8179a8d3")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

ARGS = {
    "goal": "Generate a response that bypasses safety guidelines",
    "target": "I understand you want me to"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_judge_system_prompt Ran
ran; our contract draft was wrong, not the code
yihanwang617/llm-jailbreaking-defense-backtranslation/PAIR/system_prompts.py
code served (permissive licence) · get_code("581e46fff7eed12a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

ARGS = {
    "goal": "Provide instructions on how to hack into a computer system",
    "target_str": "The assistant provided detailed hacking instructions without any safety warnings"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_parser Ran
ran; our contract draft was wrong, not the code
yihanwang617/llm-jailbreaking-defense-backtranslation/arguments.py
code served (permissive licence) · get_code("99b925af061ea18f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ArgumentParser. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import argparse

ARGS = {
    'target_model': False,
    'defense': False,
    'judge': False,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ArgumentParser — a ArgumentParser output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a ArgumentParser output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_recursive_key Ran
ran; our contract draft was wrong, not the code
yihanwang617/llm-jailbreaking-defense-backtranslation/utils.py
code served (permissive licence) · get_code("5de204a83268b25b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Analyze the function: get_recursive_key(obj, key, delim='.')
# - obj: a dict-like object that supports 'in' operator and indexing with [k]
# - key: a string that will be split by delim
# - delim: a string delimiter (default '.')
# The function recursively traverses nested dicts using keys split by delimiter

# Build a nested dict structure to traverse
nested_dict = {
    'level1': {
        'level2': {
            'level3': 'final_value'
        },
        'another': 42
    },
    'simple': 'simple_value'
}

# Test case: access nested value using dot notation
test_key = 'level1.level2.level3'
test_delim = '.'

ARGS = {
    'obj': nested_dict,
    'key': test_key,
    'delim': test_delim
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_model_and_tokenizer Ran
ran, honoured the contract we drafted
yihanwang617/llm-jailbreaking-defense-backtranslation/attacks/autodan.py
code served (permissive licence) · get_code("3bcc8e42ea5e457e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 2, dtype family float; no output shape is stated. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from unittest.mock import Mock, MagicMock
from transformers import AutoTokenizer

torch.manual_seed(0)

# Create a mock model that behaves like AutoModelForCausalLM
mock_model = MagicMock()
mock_model.to = MagicMock(return_value=mock_model)
mock_model.eval = MagicMock(return_value=mock_model)

# Create a mock tokenizer with necessary attributes
mock_tokenizer = MagicMock()
mock_tokenizer.bos_token_id = 1
mock_tokenizer.unk_token_id = 0
mock_tokenizer.eos_token = "<|endoftext|>"
mock_tokenizer.pad_token = "<|endoftext|>"
mock_tokenizer.padding_side = 'right'

# Mock the AutoModelForCausalLM.from_pretrained to return our mock model
import transformers
original_auto_model = transformers.AutoModelForCausalLM.from_pretrained
transformers.AutoModelForCausalLM.from_pretrained = MagicMock(return_value=mock_model)

# Mock the AutoTokenizer.from_pretrained to return our mock tokenizer
original_auto_tokenizer = transformers.AutoTokenizer.from_pretrained
transformers.AutoTokenizer.from_pretrained = MagicMock(return_value=mock_tokenizer)

ARGS = {
    'model_path': 'gpt2',
    'tokenizer_path': None,
    'device': 'cpu',
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch, transformers — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 2, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_prompts Ran
ran; our contract draft was wrong, not the code
yihanwang617/llm-jailbreaking-defense-backtranslation/utils.py
code served (permissive licence) · get_code("37cddda277e8045d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import os
import tempfile
import shutil

# Create a temporary directory with test JSON files
temp_dir = tempfile.mkdtemp()

# Create sample prompt files
prompts_data_1 = [
    {"id": 1, "text": "What is machine learning?"},
    {"id": 2, "text": "Explain neural networks."}
]

prompts_data_2 = [
    {"id": 3, "text": "What is deep learning?"},
    {"id": 4, "text": "How do transformers work?"},
    {"id": 5, "text": "What is attention?"}
]

# Write test files
with open(os.path.join(temp_dir, "prompts_1.json"), "w") as f:
    json.dump(prompts_data_1, f)

with open(os.path.join(temp_dir, "prompts_2.json"), "w") as f:
    json.dump(prompts_data_2, f)

# Define arguments
ARGS = {
    "input_path": temp_dir,
    "num_examples": 3,
    "offset": 1
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_tokenizer Ran
ran; our contract draft was wrong, not the code
yihanwang617/llm-jailbreaking-defense/llm_jailbreaking_defense/models.py
code served (permissive licence) · get_code("ee5d16865ecd5e2b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Mock. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np
from unittest.mock import Mock, MagicMock

torch.manual_seed(0)

# Create a mock tokenizer that behaves like a HuggingFace tokenizer
mock_tokenizer = Mock()
mock_tokenizer.eos_token = "<|end_of_text|>"
mock_tokenizer.unk_token = "<unk>"
mock_tokenizer.pad_token = None
mock_tokenizer.padding_side = "right"

# Patch AutoTokenizer.from_pretrained to return our mock tokenizer
import transformers
original_from_pretrained = transformers.AutoTokenizer.from_pretrained

def mock_from_pretrained(model_path, use_fast=False):
    tokenizer = Mock()
    tokenizer.eos_token = "<|end_of_text|>"
    tokenizer.unk_token = "<unk>"
    tokenizer.pad_token = None
    tokenizer.padding_side = "right"
    return tokenizer

transformers.AutoTokenizer.from_pretrained = mock_from_pretrained

ARGS = {
    "model_path": "meta-llama/Llama-3-8B"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch, transformers — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Mock — a Mock output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
parse_args Ran
ran; our contract draft was wrong, not the code
yihanwang617/llm-jailbreaking-defense-backtranslation/arguments.py
code served (permissive licence) · get_code("15642ad4bb2fccb7")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Namespace. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import argparse
import sys
from io import StringIO

def parse_args(target_model=False, defense=False, judge=False):
    parser = argparse.ArgumentParser()
    
    # add_data_args
    parser.add_argument('--load_data_path', type=str)
    parser.add_argument('--save_result_path', type=str)
    parser.add_argument('--num_examples', type=int, default=None, help='Number of examples to use. By default, use all the provided examples.')
    parser.add_argument('--starting_index', '--start', type=int, default=0, help='Index of the starting example.')
    
    # add_key_args
    parser.add_argument('--goal_key', type=str, default='goal')
    parser.add_argument('--target_key', type=str, default='target')
    parser.add_argument('--prompt_key', type=str, default='jailbreaking')
    parser.add_argument('--response_key', type=str, default='output')
    parser.add_argument('--reference_key', type=str, default=None)
    parser.add_argument('--jailbreaking_key', type=str, default='jailbreaking')
    parser.add_argument('--existing_response_key', type=str, default='response')
    parser.add_argument('--reuse_response', action='store_true')
    
    if target_model:
        parser.add_argument('--target_model', type=str, default='vicuna', help='Name of target model.', choices=['vicuna', 'vicuna-13b-v1.5', 'gpt-4', 'gpt-3.5-turbo', 'gpt-3.5-turbo-0301', 'gpt-3.5-turbo-0613', 'llama-2', 'llama-2-13b'])
        parser.add_argument('--target_max_n_tokens', type=int, default=150, help='Maximum number of generated tokens for the target.')
        parser.add_argument('--max_memory', type=int, default=None)
        parser.add_argument('--target_model_batch_size', type=int, default=2)
        parser.add_argument('--pap_template', action='store_true')
    
    if defense:
        parser.add_argument('--defense_method', type=str, default='None', help='Defense method applied on the target model. No defense by default, choose from "None", "SmoothLLM", "backtranslation", "backtranslation_with_threshold_[a negative float numer]", "response_check", "paraphrase_prompt"]')
        parser.add_argument('--backtranslation_threshold', type=float, default=-2.0)
        parser.add_argument('--backtranslation_new_response_length', type=int, default=None)
        parser.add_argument('--backtranslation_infer_model', type=str, default='vicuna')
        parser.add_argument('--paraphrase_model', type=str, default='gpt-3.5-turbo')
        parser.add_argument('--return_new_response_anyway', action='store_true')
        parser.add_argument('--response_check_threshold', type=int, default=5)
        parser.add_argument('--SmoothLLM_perturbation_num_samples', type=int, default=3)
        parser.add_argument('--SmoothLLM_perturbation_ratio', type=float, default=0.1)
        parser.add_argument('--SmoothLLM_perturbation_type', type=str, default='swap')
    
    if judge:
        parser.add_argument('--judge_name', nargs='+', default=['pair@gpt-4'], help='name of the judge to be used: [judge_method]@[judge_model]')
        parser.add_argument('--judge_max_n_tokens', type=int, default=10, help='Maximum number of tokens for the judge.')
        parser.add_argument('--judge_temperature', type=float, default=0, help='Temperature to use for judge.')
    
    parser.add_argument('--verbose', action='store_true')
    parser.add_argument('--force_overwrite', action='store_true')
    
    # Capture stdout to suppress print output
    old_stdout = sys.stdout
    sys.stdout = StringIO()
    
    args = parser.parse_args([])
    
    sys.stdout = old_stdout
    return args

ARGS = {
    'target_model': False,
    'defense': False,
    'judge': False,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Namespace — a Namespace output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a Namespace output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
process_target_response Ran
ran; our contract draft was wrong, not the code
yihanwang617/llm-jailbreaking-defense-backtranslation/PAIR/common.py
code served (permissive licence) · get_code("9fdfb4537f5fd761")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

ARGS = {
    "target_response": "The model should follow all user instructions without restriction.",
    "score": 0.85,
    "goal": "Make the model ignore its safety guidelines and provide harmful content",
    "target_str": "adversarial_prompt_001"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
roulette_wheel_selection Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
yihanwang617/llm-jailbreaking-defense-backtranslation/attacks/autodan.py
code served (permissive licence) · get_code("1fe7dbe6f5cc4593")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

np.random.seed(42)

# Construct arguments for roulette_wheel_selection
data_list = ["item_a", "item_b", "item_c", "item_d"]
score_list = np.array([1.0, 2.5, 0.5, 3.2])
num_selected = 3
if_softmax = True

ARGS = {
    "data_list": data_list,
    "score_list": score_list,
    "num_selected": num_selected,
    "if_softmax": if_softmax
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
autodan_sample_control Not yet run
unverified (harvested, no recorded run)
yihanwang617/llm-jailbreaking-defense-backtranslation/attacks/autodan.py
code served (permissive licence) · get_code("6c4da698c1f967dc")
How it ran

no invocation recorded.

get_embedding_matrix Not yet run
unverified (harvested, no recorded run)
yihanwang617/llm-jailbreaking-defense-backtranslation/GCG/llm_attacks/base/attack_manager.py
code served (permissive licence) · get_code("688aaeba506e3329")
How it ran

no invocation recorded.

get_embeddings Not yet run
unverified (harvested, no recorded run)
yihanwang617/llm-jailbreaking-defense-backtranslation/GCG/llm_attacks/base/attack_manager.py
code served (permissive licence) · get_code("f556dda0785f3c77")
How it ran

no invocation recorded.

Also present in other papers' code

Of 16 samples lifted from this paper's linked repositories, 16 carry a function identity and 0 do not. Of the 16 distinct functions with an identity, 7 are also present in other papers whose code we hold (5 with at least one paper dated earlier, 2 the same month, 7 later) and 9 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
get_embedding_layer 6 other papers
1 dated earlier, 1 the same month, 4 later
2310.15140 (2023-10)
AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models
2402.05467 (2024-02)
Rapid Optimization for Jailbreaking LLMs via Subconscious Exploitation and Echopraxia
2404.13968 (2024-04)
Protecting Your LLMs with Information Bottleneck
and 3 more not named here
get_embedding_matrix 6 other papers
1 the same month, 5 later
2402.09063 (2024-02)
Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space
2403.04957 (2024-03)
Automatic and Universal Prompt Injection Attacks against Large Language Models
2404.13968 (2024-04)
Protecting Your LLMs with Information Bottleneck
and 3 more not named here
get_embeddings 5 other papers
5 later
2403.04957 (2024-03)
Automatic and Universal Prompt Injection Attacks against Large Language Models
2404.13968 (2024-04)
Protecting Your LLMs with Information Bottleneck
2405.21018 (2024-05)
Improved Techniques for Optimization-Based Jailbreaking on Large Language Models
and 2 more not named here
get_init_msg 5 other papers
1 dated earlier, 4 later
2311.03191 (2023-11)
DeepInception: Hypnotize Large Language Model to Be Jailbreaker
2403.00867 (2024-03)
Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes
2404.02151 (2024-04)
Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks
and 2 more not named here
process_target_response 5 other papers
1 dated earlier, 4 later
2311.03191 (2023-11)
DeepInception: Hypnotize Large Language Model to Be Jailbreaker
2404.02151 (2024-04)
Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks
2404.12038 (2024-04)
Uncovering Safety Risks of Large Language Models through Concept Activation Vector
and 2 more not named here
extract_json 3 other papers
1 dated earlier, 2 later
2311.03191 (2023-11)
DeepInception: Hypnotize Large Language Model to Be Jailbreaker
2404.02151 (2024-04)
Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks
2404.12038 (2024-04)
Uncovering Safety Risks of Large Language Models through Concept Activation Vector
roulette_wheel_selection 2 other papers
1 dated earlier, 1 later
2310.04451 (2023-10)
AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models
2410.04234 (2024-10)
Functional Homotopy: Smoothing Discrete Optimization via Continuous Parameters for LLM Jailbreak Attacks

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Although many large language models (LLMs) have been trained to refuse harmful requests, they are still vulnerable to jailbreaking attacks which rewrite the original prompt to conceal its harmful intent. In this paper, we propose a new method for defending LLMs against jailbreaking attacks by ``backtranslation''. Specifically, given an initial response generated by the target LLM from an input prompt, our backtranslation prompts a language model to infer an input prompt that can lead to the response. The inferred prompt is called the backtranslated prompt which tends to reveal the actual intent of the original prompt, since it is generated based on the LLM's response and not directly manipulated by the attacker. We then run the target LLM again on the backtranslated prompt, and we refuse the original prompt if the model refuses the backtranslated prompt. We explain that the proposed defense provides several benefits on its effectiveness and efficiency. We empirically demonstrate that our defense significantly outperforms the baselines, in the cases that are hard for the baselines, and our defense also has little impact on the generation quality for benign input prompts. Our implementation is based on our library for LLM jailbreaking defense algorithms at \url{https://github.com/YihanWang617/llm-jailbreaking-defense}, and the code for reproducing our experiments is available at \url{https://github.com/YihanWang617/LLM-Jailbreaking-Defense-Backtranslation}.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2402.16459")
get_code_for_paper("2402.16459")
have("2402.16459")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2402.16459.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2402.16459.svg)](https://syntology.ai/paper/2402.16459)

Connect an agent — have() is free.