SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2405.04346 · ICML · 2024

Revisiting Character-level Adversarial Attacks for Language Models

Fanghui Liu, Volkan Cevher, Grigorios Chrysos, Yongtao Wu, Elias Rocamora

arXiv · PDF · Open in the Atlas

Code that ran

23 of 31 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (3 of them constructed an object rather than computing a result; 3 honoured the contract we drafted, 1 violated it and 5 have no contract verdict; 14 failures of our instrument; 8 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2405.04346.json · badge: /api/ran/2405.04346.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
lions-epfl/charmer canonical (classifier label)
classifier basis: classifier_2025
15 ran · 6 no recorded run
LIONS-EPFL/Charmer canonical (classifier label)
classifier basis: census
8 ran · 2 no recorded run
FunctionStatusWhere it lives
CrossEntropyLoss_negative Ran
ran on a synthesized input
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("6a768c7d89fa45a6")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 0, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

class CrossEntropyLoss_negative:

    def __init__(self):
        pass

    def __call__(self, logits, true_classes):
        return -1 * torch.nn.CrossEntropyLoss(reduce='mean')(logits, true_classes)


INIT_ARGS = {}

ARGS = {
    'logits': torch.randn(2, 4),
    'true_classes': torch.tensor([0, 2])
}

if __name__ == '__main__':
    loss_fn = CrossEntropyLoss_negative(**INIT_ARGS)
    result = loss_fn(**ARGS)
    print(result)
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 0, dtype family float — no output shape is stated: observe_signature.py observed this class with its constructor sizes canonicalised where it could, which the node does not record, so its shape is not stated as this setup block's
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
all_strings_within_edit1 Ran
ran; our contract draft was wrong, not the code
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("b87dfb2a13227d05")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

ARGS = {
    'sequence': 'ATCG',
    'bases': 'ATCG'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
all_strings_within_editx Ran
ran; our contract draft was wrong, not the code
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("232efbd63c8df619")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned set. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

# Setup code for all_strings_within_editx function

# Analyze the function:
# - sequence: str - a DNA sequence string
# - bases: str - alphabet of bases (default 'ATCG')
# - edit_distance: int - number of edits away to generate

# The function generates all strings within a given edit distance
# It calls all_strings_within_edit1 recursively and returns a set of strings

# Valid arguments:
ARGS = {
    'sequence': 'ATG',
    'bases': 'ATCG',
    'edit_distance': 1
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type set — a set output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a set output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
backtrace Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("96a7b58d20a2005f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

def compute_edit_distance_matrix(first, second):
    """Helper to compute the distance matrix for edit distance"""
    f = [char for char in first]
    s = [char for char in second]
    
    rows = len(f) + 1
    cols = len(s) + 1
    matrix = [[0] * cols for _ in range(rows)]
    
    for i in range(rows):
        matrix[i][0] = i
    for j in range(cols):
        matrix[0][j] = j
    
    for i in range(1, rows):
        for j in range(1, cols):
            if f[i - 1] == s[j - 1]:
                cost = 0
            else:
                cost = 1
            matrix[i][j] = min(
                matrix[i - 1][j] + 1,
                matrix[i - 1][j - 1] + cost,
                matrix[i][j - 1] + 1
            )
    
    return matrix

first = "cat"
second = "dog"
matrix = compute_edit_distance_matrix(first, second)

ARGS = {
    "first": first,
    "second": second,
    "matrix": matrix
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
compute_centroid_distances Ran
ran, honoured the contract we drafted
LIONS-EPFL/Charmer/baseline/roben/agglom_clusters.py
code served (permissive licence) · get_code("92dfc8c506947d1c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 2, dtype family float, shape (5, 2) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Set random seed for reproducibility
np.random.seed(0)

# Create small example data
# X: vocabulary words represented as vectors
# Shape: (n_vocab_words, vec_size) = (5, 3)
X = np.random.randn(5, 3).astype(np.float32)

# mu: cluster centroids
# Shape: (n_clusters, vec_size) = (2, 3)
mu = np.random.randn(2, 3).astype(np.float32)

ARGS = {
    "X": X,
    "mu": mu,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 2, dtype family float, shape (5, 2) — shape (5, 2) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
concat_prompt_question_label Ran
ran; our contract draft was wrong, not the code
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("7a4e3ab49ac3c159")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Construct small tensors for the function arguments
# T_promptbegin: dict with 'input_ids' and 'attention_mask', shape [1, seq_len]
T_promptbegin = {
    'input_ids': torch.tensor([[1, 2, 3, 4]], dtype=torch.long),
    'attention_mask': torch.tensor([[1, 1, 1, 1]], dtype=torch.long)
}

# T_promptend: dict with 'input_ids' and 'attention_mask', shape [1, seq_len]
T_promptend = {
    'input_ids': torch.tensor([[5, 6]], dtype=torch.long),
    'attention_mask': torch.tensor([[1, 1]], dtype=torch.long)
}

# T_label: dict with 'input_ids' and 'attention_mask', shape [1, seq_len]
T_label = {
    'input_ids': torch.tensor([[7, 8, 9]], dtype=torch.long),
    'attention_mask': torch.tensor([[1, 1, 1]], dtype=torch.long)
}

# T_questions: dict with 'input_ids' and 'attention_mask'
# input_ids and attention_mask should be 2D tensors where we iterate over first dimension
T_questions = {
    'input_ids': torch.tensor([[10, 11], [12, 13]], dtype=torch.long),
    'attention_mask': torch.tensor([[1, 1], [1, 1]], dtype=torch.long)
}

ARGS = {
    'T_promptbegin': T_promptbegin,
    'T_questions': T_questions,
    'T_promptend': T_promptend,
    'T_label': T_label
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
ctx_noparamgrad Ran
ran on a synthesized input · constructed an object; no result was computed
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("4e36262a542c16fb")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, ctx_noparamgrad: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

class SimpleModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = nn.Linear(4, 3)
        self.linear2 = nn.Linear(3, 2)
    
    def forward(self, x):
        x = self.linear1(x)
        x = self.linear2(x)
        return x

module = SimpleModule()

ARGS = {
    "module": module
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ctx_noparamgrad — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a ctx_noparamgrad output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
generate_all_sentences Ran
ran; our contract draft was wrong, not the code
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("7b06bf5b0955fdac")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Analyzing the function:
# generate_all_sentences_at_z(S, z, V, k=1, alternative=-1)
# - S: a sentence (string) that gets modified
# - z: location id (integer)
# - V: vocabulary, list of UNICODE indices (list of integers)
# - k: number of possible changes (integer, default 1)
# - alternative: alternative character code (integer, default -1)

# The function iterates through range(len(V)) and calls generate_sentence for each
# generate_sentence expects V to contain valid unicode indices or -1 for invalid

# Creating test data:
S = "hello"  # Simple sentence
z = 2  # Position to modify (0-indexed, so 'l' in "hello")
V = [97, 98, 99, -1]  # Unicode indices: a, b, c, and invalid (-1)
k = 1  # Single spacing character
alternative = -1  # Default behavior

ARGS = {
    "S": S,
    "V": V,
    "subset_z": None,
    "k": k,
    "alternative": alternative
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
generate_all_sentences_at_z Ran
ran; our contract draft was wrong, not the code
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("b82a34d25dd75308")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Analyzing the function generate_all_sentences_at_z:
# - S: sentence (string) - needs to be a string
# - z: location id (int) - position in sentence
# - V: vocabulary (list of UNICODE indices) - list of integers
# - k: number of possible changes (default 1) - integer
# - alternative: alternative character (default -1) - integer or None

# The function calls:
# - generate_sentence with S, z, u (ranging from 0 to len(V)-1), V, k, alternative
# - generate_sentence uses: chr(V[c]) where V[c] should be a valid unicode value
# - It checks if V[u] != -1
# - It uses len(V) in the range
# - It uses len(S) to build mask

# Building test data:
S = "hello"  # Simple test sentence
z = 1  # Position to modify (second character)
k = 1  # Number of padding spaces

# V needs to be a list of unicode indices
# Valid unicode values for common characters
V = [ord('a'), ord('b'), ord('c'), ord('d'), -1, ord('e')]  # Include -1 to test that branch

alternative = ord('x')  # Alternative character unicode value

ARGS = {
    "S": S,
    "z": z,
    "V": V,
    "k": k,
    "alternative": alternative
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
generate_sentence Ran
ran; our contract draft was wrong, not the code
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("7f56e8436fe9b621")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

# Analyze the function:
# - S: a sentence (string), iterated over with 'for s in S'
# - z: a position or list of positions (checked with type(z) == list)
# - u: a character id or list of character ids
# - V: vocabulary - a list indexed by character id, contains UNICODE values or -1
# - k: number of possible changes (default 1)
# - alternative: optional character code (can be None, -1, or a valid code)

# Build minimal valid arguments:

# S: a simple sentence string
S = "hello"

# z: a single position (integer)
z = 1

# u: a character id (integer that indexes into V)
u = 2

# V: vocabulary list - maps character id to UNICODE value
# Need enough entries to cover u=2, so at least 3 entries
# Using valid UNICODE values for common characters
V = [ord('a'), ord('b'), ord('c'), ord('d')]

# k: number of changes (default is fine, but let's be explicit)
k = 1

# alternative: None is fine, or we could use a valid UNICODE value or -1
alternative = None

ARGS = {
    'S': S,
    'z': z,
    'u': u,
    'V': V,
    'k': k,
    'alternative': alternative
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_class_num Ran
ran, honoured the contract we drafted
LIONS-EPFL/Charmer/dataloader.py
code served (permissive licence) · get_code("904d63da437332d4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

ARGS = {
    'dataset_name': 'sst'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_goals_and_targets Ran
ran; our contract draft was wrong, not the code
LIONS-EPFL/Charmer/utils_jailbreak.py
code served (permissive licence) · get_code("55554ebb64125749")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 2, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd
import tempfile
import os

# Create temporary CSV files for testing
temp_dir = tempfile.mkdtemp()

# Create train_data.csv with 'target' and 'goal' columns
train_data_path = os.path.join(temp_dir, 'train_data.csv')
train_df = pd.DataFrame({
    'goal': ['goal1', 'goal2', 'goal3', 'goal4'],
    'target': ['target1', 'target2', 'target3', 'target4']
})
train_df.to_csv(train_data_path, index=False)

# Create test_data.csv with 'target' and 'goal' columns
test_data_path = os.path.join(temp_dir, 'test_data.csv')
test_df = pd.DataFrame({
    'goal': ['test_goal1', 'test_goal2'],
    'target': ['test_target1', 'test_target2']
})
test_df.to_csv(test_data_path, index=False)

# Create a params object with all necessary attributes
class Params:
    def __init__(self):
        self.goals = []
        self.targets = []
        self.test_goals = []
        self.test_targets = []
        self.data_offset = 0
        self.train_data = train_data_path
        self.test_data = test_data_path
        self.n_train_data = 2
        self.n_test_data = 2

params = Params()

ARGS = {
    'params': params
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 2, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_module_training_state Ran
ran; our contract draft was wrong, not the code
LIONS-EPFL/Charmer/context.py
code served (permissive licence) · get_code("28413f6ed2f091a9")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Build a small nn.Module with nested submodules
# The function calls module.modules() which iterates through all submodules
class SimpleNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(4, 3)
        self.layer2 = nn.Linear(3, 2)
        self.dropout = nn.Dropout(0.5)
    
    def forward(self, x):
        x = self.layer1(x)
        x = self.dropout(x)
        x = self.layer2(x)
        return x

module = SimpleNet()

ARGS = {
    "module": module
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_param_grad_state Ran
ran; our contract draft was wrong, not the code
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("41eb44fa1a771cd1")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

class SimpleModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = nn.Linear(4, 3)
        self.linear2 = nn.Linear(3, 2)
    
    def forward(self, x):
        x = self.linear1(x)
        x = self.linear2(x)
        return x

module = SimpleModule()

ARGS = {
    "module": module
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_task_type Ran
ran, violated the contract we drafted
LIONS-EPFL/Charmer/dataloader.py
code served (permissive licence) · get_code("75df2d0c0186e517")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

ARGS = {
    'dataset_name': 'sst'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_vocabulary Ran
ran, honoured the contract we drafted
LIONS-EPFL/Charmer/utils.py
code served (permissive licence) · get_code("593614c84d8b37c2")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Create a minimal dataset structure
# The function expects dataset[keyword] to be iterable of strings
dataset = {
    'hypothesis': [
        'this is a test sentence',
        'another example here'
    ],
    'text': [
        'some text content',
        'more text data'
    ],
    'sentence': [
        'default sentence one',
        'default sentence two'
    ]
}

dataset_name = 'mnli'

ARGS = {
    'dataset': dataset,
    'dataset_name': dataset_name
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family int — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
llm_input_processor Ran
ran on a synthesized input · constructed an object; no result was computed
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("456e7d855a79eb1e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, llm_input_processor: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

ARGS = {
    'dataset': 'sst2',
    'premise': 'This movie is absolutely fantastic and entertaining.',
    'model_name': 'llama'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type llm_input_processor — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a llm_input_processor output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
margin_loss_llm Ran
ran on a synthesized input · constructed an object; no result was computed
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("28879c448358c558")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, margin_loss_llm: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Analyze the function:
# - __init__ takes: target_class (int), tau (float)
# - __call__ takes: logits (tensor), true_classes (int or indexable)
# - logits is indexed as logits[:, true_classes] and logits[:, self.target_class]
#   so it's a 2D tensor with shape (batch_size, num_classes)
# - true_classes is used as an index into dimension 1, so it's an int or tensor of indices
# - tau is compared with 0 and used in arithmetic

# Create small tensors
batch_size = 2
num_classes = 4

# logits: shape (2, 4) - batch of 2, with 4 class scores
logits = torch.randn(batch_size, num_classes)

# true_classes: single class index (used as logits[:, true_classes])
true_classes = 1

# target_class: single class index
target_class = 2

# tau: margin parameter (non-zero to test both branches)
tau = 0.5

ARGS = {
    "target_class": target_class,
    "tau": tau,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type margin_loss_llm — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a margin_loss_llm output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
margin_loss_lm Ran
ran; our contract draft was wrong, not the code
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("7eae0eca72c38f98")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 1, dtype family float, shape (2,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function expects:
# - logits: a 2D tensor of shape (batch_size, num_classes)
# - true_class: an integer index representing the true class

# Create small tensors for testing
batch_size = 2
num_classes = 4
true_class = 1

# Create logits tensor with shape (batch_size, num_classes)
logits = torch.randn(batch_size, num_classes)

ARGS = {
    "logits": logits,
    "true_class": true_class,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 1, dtype family float, shape (2,) — shape (2,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
margin_loss_lm_batched Ran
ran on a synthesized input
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("f38056701ce7c0b6")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 0, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

INIT_ARGS = {
    'reduction': 'mean'
}

ARGS = {
    'logits': torch.randn(2, 5),
    'true_classes': torch.tensor([1, 3])
}

METHOD = '__call__'

def margin_loss_lm(logits, true_class):
    """
    Standard margin loss for classification
    """
    max_other, _ = torch.cat((logits[:, :true_class], logits[:, true_class + 1:]), dim=-1).max(dim=-1)
    return max_other - logits[:, true_class]

class margin_loss_lm_batched:

    def __init__(self, reduction='None'):
        self.reduction = reduction

    def __call__(self, logits, true_classes):
        """
        Standard margin loss for classification
        """
        L = torch.cat([margin_loss_lm(l.unsqueeze(0), t) for l, t in zip(logits, true_classes)], dim=0)
        if self.reduction == 'mean':
            return torch.mean(L)
        elif self.reduction == 'sum':
            return torch.sum(L)
        else:
            return L


if __name__ == '__main__':
    instance = margin_loss_lm_batched(**INIT_ARGS)
    result = instance(**ARGS)
    print(f"Result: {result}")
    print(f"Result shape: {result.shape if hasattr(result, 'shape') else 'scalar'}")
Setup block drafted by
a language model, class construct-then-call prompt, model not recorded
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 0, dtype family float — no output shape is stated: observe_signature.py observed this class with its constructor sizes canonicalised where it could, which the node does not record, so its shape is not stated as this setup block's
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
target_loss Ran
ran; our contract draft was wrong, not the code
LIONS-EPFL/Charmer/utils_jailbreak.py
code served (permissive licence) · get_code("11a3348eb0e4ba60")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 1, dtype family float, shape (2,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Analyzing the function:
# - logits: used as logits[:, loss_slice, :].transpose(1, 2)
#   This suggests shape (batch, seq, vocab_size)
# - ids: used as ids[:, target_slice]
#   This suggests shape (batch, seq)
# - target_slice: a slice object with .start and .stop attributes

batch_size = 2
seq_len = 5
vocab_size = 4

logits = torch.randn(batch_size, seq_len, vocab_size)
ids = torch.randint(0, vocab_size, (batch_size, seq_len))
target_slice = slice(2, 4)

ARGS = {
    "logits": logits,
    "ids": ids,
    "target_slice": target_slice,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 1, dtype family float, shape (2,) — shape (2,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
update_mu Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
LIONS-EPFL/Charmer/baseline/roben/agglom_clusters.py
code served (permissive licence) · get_code("12efa3ddd30cf7c2")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 2, dtype family float, shape (3, 2) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# C is expected to be a 2D numpy array (likely cluster assignments or weights)
# f is expected to be a 1D numpy array that gets reshaped to (-1, 1)
# The function performs weighted averaging, so we need compatible shapes

# Let's use:
# - f with shape (2,) - represents 2 samples
# - C with shape (2, 3) - represents 2 samples, 3 clusters

np.random.seed(0)

C = np.array([
    [0.8, 0.2, 0.0],
    [0.1, 0.6, 0.3]
], dtype=np.float32)

f = np.array([2.0, 3.0], dtype=np.float32)

ARGS = {
    "C": C,
    "f": f
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 2, dtype family float, shape (3, 2) — shape (3, 2) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
word_edit_distance Ran
ran; our contract draft was wrong, not the code
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("f0c1289fd2565267")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# The function takes two sequences x and y
# It computes edit distance by comparing elements
# So x and y should be sequences (lists, strings, or arrays) where elements can be compared with ==

# Create two simple sequences to compute edit distance
x = "kitten"
y = "sitting"

ARGS = {
    "x": x,
    "y": y,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
Charmer Not yet run
unverified (harvested, no recorded run)
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("59ab5813bed83cb2")
How it ran

no invocation recorded.

combined_emb_input Not yet run
unverified (harvested, no recorded run)
LIONS-EPFL/Charmer/utils.py
code served (permissive licence) · get_code("a299d58465c7bf59")
How it ran

no invocation recorded.

get_llm_embeddings Not yet run
unverified (harvested, no recorded run)
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("8ae9f9ad369707a9")
How it ran

no invocation recorded.

get_llm_encoder Not yet run
unverified (harvested, no recorded run)
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("6297161e122df004")
How it ran

no invocation recorded.

load_attack_dataset Not yet run
unverified (harvested, no recorded run)
LIONS-EPFL/Charmer/dataloader.py
code served (permissive licence) · get_code("7f504481b40a6171")
How it ran

no invocation recorded.

resize_embeddings Not yet run
unverified (harvested, no recorded run)
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("cb53a78f621babdb")
How it ran

no invocation recorded.

set_param_grad_off Not yet run
unverified (harvested, no recorded run)
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("e75d10fb3cfbb60c")
How it ran

no invocation recorded.

set_param_grad_state Not yet run
unverified (harvested, no recorded run)
lions-epfl/charmer/charmer.py
code served (permissive licence) · get_code("ea9f4cd8cc668205")
How it ran

no invocation recorded.

Also present in other papers' code

Of 31 samples lifted from this paper's linked repositories, 31 carry a function identity and 0 do not. Of the 31 distinct functions with an identity, 2 are also present in other papers whose code we hold (2 with at least one paper dated earlier, 0 the same month, 0 later) and 29 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
get_module_training_state 1 other paper
1 dated earlier
2311.18495 (2023-11)
Improving Adversarial Transferability via Model Alignment
get_param_grad_state
brick of this paper's assembly: helper
1 other paper
1 dated earlier
2311.18495 (2023-11)
Improving Adversarial Transferability via Model Alignment

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Adversarial attacks in Natural Language Processing apply perturbations in the character or token levels. Token-level attacks, gaining prominence for their use of gradient-based methods, are susceptible to altering sentence semantics, leading to invalid adversarial examples. While characterlevel attacks easily maintain semantics, they have received less attention as they cannot easily adopt popular gradient-based methods, and are thought to be easy to defend. Challenging these beliefs, we introduce Charmer, an efficient query-based adversarial attack capable of achieving high attack success rate (ASR) while generating highly similar adversarial examples. Our method successfully targets both small (BERT) and large (Llama 2) models. Specifically, on BERT with SST-2, Charmer improves the ASR in 4.84% points and the USE similarity in 8% points with respect to the previous art. Our implementation is available in github.com/LIONS-EPFL/Charmer.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2405.04346")
get_code_for_paper("2405.04346")
have("2405.04346")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2405.04346.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2405.04346.svg)](https://syntology.ai/paper/2405.04346)

Connect an agent — have() is free.