| CrossEntropyLoss_negative |
Ran ran on a synthesized input |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("6a768c7d89fa45a6") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 0, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
class CrossEntropyLoss_negative:
def __init__(self):
pass
def __call__(self, logits, true_classes):
return -1 * torch.nn.CrossEntropyLoss(reduce='mean')(logits, true_classes)
INIT_ARGS = {}
ARGS = {
'logits': torch.randn(2, 4),
'true_classes': torch.tensor([0, 2])
}
if __name__ == '__main__':
loss_fn = CrossEntropyLoss_negative(**INIT_ARGS)
result = loss_fn(**ARGS)
print(result)
- Setup block drafted by
- a language model, class construct-then-call prompt, model not recorded
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 0, dtype family float — no output shape is stated: observe_signature.py observed this class with its constructor sizes canonicalised where it could, which the node does not record, so its shape is not stated as this setup block's
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
|
| all_strings_within_edit1 |
Ran ran; our contract draft was wrong, not the code |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("b87dfb2a13227d05") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
ARGS = {
'sequence': 'ATCG',
'bases': 'ATCG'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| all_strings_within_editx |
Ran ran; our contract draft was wrong, not the code |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("232efbd63c8df619") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned set. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
# Setup code for all_strings_within_editx function
# Analyze the function:
# - sequence: str - a DNA sequence string
# - bases: str - alphabet of bases (default 'ATCG')
# - edit_distance: int - number of edits away to generate
# The function generates all strings within a given edit distance
# It calls all_strings_within_edit1 recursively and returns a set of strings
# Valid arguments:
ARGS = {
'sequence': 'ATG',
'bases': 'ATCG',
'edit_distance': 1
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type set — a set output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a set output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| backtrace |
Ran ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT) |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("96a7b58d20a2005f") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
def compute_edit_distance_matrix(first, second):
"""Helper to compute the distance matrix for edit distance"""
f = [char for char in first]
s = [char for char in second]
rows = len(f) + 1
cols = len(s) + 1
matrix = [[0] * cols for _ in range(rows)]
for i in range(rows):
matrix[i][0] = i
for j in range(cols):
matrix[0][j] = j
for i in range(1, rows):
for j in range(1, cols):
if f[i - 1] == s[j - 1]:
cost = 0
else:
cost = 1
matrix[i][j] = min(
matrix[i - 1][j] + 1,
matrix[i - 1][j - 1] + cost,
matrix[i][j - 1] + 1
)
return matrix
first = "cat"
second = "dog"
matrix = compute_edit_distance_matrix(first, second)
ARGS = {
"first": first,
"second": second,
"matrix": matrix
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| compute_centroid_distances |
Ran ran, honoured the contract we drafted |
LIONS-EPFL/Charmer/baseline/roben/agglom_clusters.py code served (permissive licence) · get_code("92dfc8c506947d1c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 2, dtype family float, shape (5, 2) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Set random seed for reproducibility
np.random.seed(0)
# Create small example data
# X: vocabulary words represented as vectors
# Shape: (n_vocab_words, vec_size) = (5, 3)
X = np.random.randn(5, 3).astype(np.float32)
# mu: cluster centroids
# Shape: (n_clusters, vec_size) = (2, 3)
mu = np.random.randn(2, 3).astype(np.float32)
ARGS = {
"X": X,
"mu": mu,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 2, dtype family float, shape (5, 2) — shape (5, 2) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| concat_prompt_question_label |
Ran ran; our contract draft was wrong, not the code |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("7a4e3ab49ac3c159") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Construct small tensors for the function arguments
# T_promptbegin: dict with 'input_ids' and 'attention_mask', shape [1, seq_len]
T_promptbegin = {
'input_ids': torch.tensor([[1, 2, 3, 4]], dtype=torch.long),
'attention_mask': torch.tensor([[1, 1, 1, 1]], dtype=torch.long)
}
# T_promptend: dict with 'input_ids' and 'attention_mask', shape [1, seq_len]
T_promptend = {
'input_ids': torch.tensor([[5, 6]], dtype=torch.long),
'attention_mask': torch.tensor([[1, 1]], dtype=torch.long)
}
# T_label: dict with 'input_ids' and 'attention_mask', shape [1, seq_len]
T_label = {
'input_ids': torch.tensor([[7, 8, 9]], dtype=torch.long),
'attention_mask': torch.tensor([[1, 1, 1]], dtype=torch.long)
}
# T_questions: dict with 'input_ids' and 'attention_mask'
# input_ids and attention_mask should be 2D tensors where we iterate over first dimension
T_questions = {
'input_ids': torch.tensor([[10, 11], [12, 13]], dtype=torch.long),
'attention_mask': torch.tensor([[1, 1], [1, 1]], dtype=torch.long)
}
ARGS = {
'T_promptbegin': T_promptbegin,
'T_questions': T_questions,
'T_promptend': T_promptend,
'T_label': T_label
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| ctx_noparamgrad |
Ran ran on a synthesized input · constructed an object; no result was computed |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("4e36262a542c16fb") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, ctx_noparamgrad: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
class SimpleModule(nn.Module):
def __init__(self):
super().__init__()
self.linear1 = nn.Linear(4, 3)
self.linear2 = nn.Linear(3, 2)
def forward(self, x):
x = self.linear1(x)
x = self.linear2(x)
return x
module = SimpleModule()
ARGS = {
"module": module
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ctx_noparamgrad — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a ctx_noparamgrad output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| generate_all_sentences |
Ran ran; our contract draft was wrong, not the code |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("7b06bf5b0955fdac") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# Analyzing the function:
# generate_all_sentences_at_z(S, z, V, k=1, alternative=-1)
# - S: a sentence (string) that gets modified
# - z: location id (integer)
# - V: vocabulary, list of UNICODE indices (list of integers)
# - k: number of possible changes (integer, default 1)
# - alternative: alternative character code (integer, default -1)
# The function iterates through range(len(V)) and calls generate_sentence for each
# generate_sentence expects V to contain valid unicode indices or -1 for invalid
# Creating test data:
S = "hello" # Simple sentence
z = 2 # Position to modify (0-indexed, so 'l' in "hello")
V = [97, 98, 99, -1] # Unicode indices: a, b, c, and invalid (-1)
k = 1 # Single spacing character
alternative = -1 # Default behavior
ARGS = {
"S": S,
"V": V,
"subset_z": None,
"k": k,
"alternative": alternative
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| generate_all_sentences_at_z |
Ran ran; our contract draft was wrong, not the code |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("b82a34d25dd75308") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
# Analyzing the function generate_all_sentences_at_z:
# - S: sentence (string) - needs to be a string
# - z: location id (int) - position in sentence
# - V: vocabulary (list of UNICODE indices) - list of integers
# - k: number of possible changes (default 1) - integer
# - alternative: alternative character (default -1) - integer or None
# The function calls:
# - generate_sentence with S, z, u (ranging from 0 to len(V)-1), V, k, alternative
# - generate_sentence uses: chr(V[c]) where V[c] should be a valid unicode value
# - It checks if V[u] != -1
# - It uses len(V) in the range
# - It uses len(S) to build mask
# Building test data:
S = "hello" # Simple test sentence
z = 1 # Position to modify (second character)
k = 1 # Number of padding spaces
# V needs to be a list of unicode indices
# Valid unicode values for common characters
V = [ord('a'), ord('b'), ord('c'), ord('d'), -1, ord('e')] # Include -1 to test that branch
alternative = ord('x') # Alternative character unicode value
ARGS = {
"S": S,
"z": z,
"V": V,
"k": k,
"alternative": alternative
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| generate_sentence |
Ran ran; our contract draft was wrong, not the code |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("7f56e8436fe9b621") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
# Analyze the function:
# - S: a sentence (string), iterated over with 'for s in S'
# - z: a position or list of positions (checked with type(z) == list)
# - u: a character id or list of character ids
# - V: vocabulary - a list indexed by character id, contains UNICODE values or -1
# - k: number of possible changes (default 1)
# - alternative: optional character code (can be None, -1, or a valid code)
# Build minimal valid arguments:
# S: a simple sentence string
S = "hello"
# z: a single position (integer)
z = 1
# u: a character id (integer that indexes into V)
u = 2
# V: vocabulary list - maps character id to UNICODE value
# Need enough entries to cover u=2, so at least 3 entries
# Using valid UNICODE values for common characters
V = [ord('a'), ord('b'), ord('c'), ord('d')]
# k: number of changes (default is fine, but let's be explicit)
k = 1
# alternative: None is fine, or we could use a valid UNICODE value or -1
alternative = None
ARGS = {
'S': S,
'z': z,
'u': u,
'V': V,
'k': k,
'alternative': alternative
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_class_num |
Ran ran, honoured the contract we drafted |
LIONS-EPFL/Charmer/dataloader.py code served (permissive licence) · get_code("904d63da437332d4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
ARGS = {
'dataset_name': 'sst'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_goals_and_targets |
Ran ran; our contract draft was wrong, not the code |
LIONS-EPFL/Charmer/utils_jailbreak.py code served (permissive licence) · get_code("55554ebb64125749") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 2, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import pandas as pd
import tempfile
import os
# Create temporary CSV files for testing
temp_dir = tempfile.mkdtemp()
# Create train_data.csv with 'target' and 'goal' columns
train_data_path = os.path.join(temp_dir, 'train_data.csv')
train_df = pd.DataFrame({
'goal': ['goal1', 'goal2', 'goal3', 'goal4'],
'target': ['target1', 'target2', 'target3', 'target4']
})
train_df.to_csv(train_data_path, index=False)
# Create test_data.csv with 'target' and 'goal' columns
test_data_path = os.path.join(temp_dir, 'test_data.csv')
test_df = pd.DataFrame({
'goal': ['test_goal1', 'test_goal2'],
'target': ['test_target1', 'test_target2']
})
test_df.to_csv(test_data_path, index=False)
# Create a params object with all necessary attributes
class Params:
def __init__(self):
self.goals = []
self.targets = []
self.test_goals = []
self.test_targets = []
self.data_offset = 0
self.train_data = train_data_path
self.test_data = test_data_path
self.n_train_data = 2
self.n_test_data = 2
params = Params()
ARGS = {
'params': params
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 2, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_module_training_state |
Ran ran; our contract draft was wrong, not the code |
LIONS-EPFL/Charmer/context.py code served (permissive licence) · get_code("28413f6ed2f091a9") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
# Build a small nn.Module with nested submodules
# The function calls module.modules() which iterates through all submodules
class SimpleNet(nn.Module):
def __init__(self):
super().__init__()
self.layer1 = nn.Linear(4, 3)
self.layer2 = nn.Linear(3, 2)
self.dropout = nn.Dropout(0.5)
def forward(self, x):
x = self.layer1(x)
x = self.dropout(x)
x = self.layer2(x)
return x
module = SimpleNet()
ARGS = {
"module": module
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_param_grad_state |
Ran ran; our contract draft was wrong, not the code |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("41eb44fa1a771cd1") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
class SimpleModule(nn.Module):
def __init__(self):
super().__init__()
self.linear1 = nn.Linear(4, 3)
self.linear2 = nn.Linear(3, 2)
def forward(self, x):
x = self.linear1(x)
x = self.linear2(x)
return x
module = SimpleModule()
ARGS = {
"module": module
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_task_type |
Ran ran, violated the contract we drafted |
LIONS-EPFL/Charmer/dataloader.py code served (permissive licence) · get_code("75df2d0c0186e517") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
ARGS = {
'dataset_name': 'sst'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_vocabulary |
Ran ran, honoured the contract we drafted |
LIONS-EPFL/Charmer/utils.py code served (permissive licence) · get_code("593614c84d8b37c2") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
# Create a minimal dataset structure
# The function expects dataset[keyword] to be iterable of strings
dataset = {
'hypothesis': [
'this is a test sentence',
'another example here'
],
'text': [
'some text content',
'more text data'
],
'sentence': [
'default sentence one',
'default sentence two'
]
}
dataset_name = 'mnli'
ARGS = {
'dataset': dataset,
'dataset_name': dataset_name
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family int — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| llm_input_processor |
Ran ran on a synthesized input · constructed an object; no result was computed |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("456e7d855a79eb1e") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, llm_input_processor: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
ARGS = {
'dataset': 'sst2',
'premise': 'This movie is absolutely fantastic and entertaining.',
'model_name': 'llama'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type llm_input_processor — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a llm_input_processor output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| margin_loss_llm |
Ran ran on a synthesized input · constructed an object; no result was computed |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("28879c448358c558") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned an instance of its own class, margin_loss_llm: the run constructed the object and did not call it, so no result was computed. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
# Analyze the function:
# - __init__ takes: target_class (int), tau (float)
# - __call__ takes: logits (tensor), true_classes (int or indexable)
# - logits is indexed as logits[:, true_classes] and logits[:, self.target_class]
# so it's a 2D tensor with shape (batch_size, num_classes)
# - true_classes is used as an index into dimension 1, so it's an int or tensor of indices
# - tau is compared with 0 and used in arithmetic
# Create small tensors
batch_size = 2
num_classes = 4
# logits: shape (2, 4) - batch of 2, with 4 class scores
logits = torch.randn(batch_size, num_classes)
# true_classes: single class index (used as logits[:, true_classes])
true_classes = 1
# target_class: single class index
target_class = 2
# tau: margin parameter (non-zero to test both branches)
tau = 0.5
ARGS = {
"target_class": target_class,
"tau": tau,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type margin_loss_llm — the run constructed an instance of its own class and did not call it; no shape is stated — the instrument coerces any output to an array; for a margin_loss_llm output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| margin_loss_lm |
Ran ran; our contract draft was wrong, not the code |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("7eae0eca72c38f98") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 1, dtype family float, shape (2,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# The function expects:
# - logits: a 2D tensor of shape (batch_size, num_classes)
# - true_class: an integer index representing the true class
# Create small tensors for testing
batch_size = 2
num_classes = 4
true_class = 1
# Create logits tensor with shape (batch_size, num_classes)
logits = torch.randn(batch_size, num_classes)
ARGS = {
"logits": logits,
"true_class": true_class,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 1, dtype family float, shape (2,) — shape (2,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| margin_loss_lm_batched |
Ran ran on a synthesized input |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("f38056701ce7c0b6") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, class construct-then-call prompt, model not recorded) and returned Tensor, ndim 0, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
INIT_ARGS = {
'reduction': 'mean'
}
ARGS = {
'logits': torch.randn(2, 5),
'true_classes': torch.tensor([1, 3])
}
METHOD = '__call__'
def margin_loss_lm(logits, true_class):
"""
Standard margin loss for classification
"""
max_other, _ = torch.cat((logits[:, :true_class], logits[:, true_class + 1:]), dim=-1).max(dim=-1)
return max_other - logits[:, true_class]
class margin_loss_lm_batched:
def __init__(self, reduction='None'):
self.reduction = reduction
def __call__(self, logits, true_classes):
"""
Standard margin loss for classification
"""
L = torch.cat([margin_loss_lm(l.unsqueeze(0), t) for l, t in zip(logits, true_classes)], dim=0)
if self.reduction == 'mean':
return torch.mean(L)
elif self.reduction == 'sum':
return torch.sum(L)
else:
return L
if __name__ == '__main__':
instance = margin_loss_lm_batched(**INIT_ARGS)
result = instance(**ARGS)
print(f"Result: {result}")
print(f"Result shape: {result.shape if hasattr(result, 'shape') else 'scalar'}")
- Setup block drafted by
- a language model, class construct-then-call prompt, model not recorded
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 0, dtype family float — no output shape is stated: observe_signature.py observed this class with its constructor sizes canonicalised where it could, which the node does not record, so its shape is not stated as this setup block's
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: torch and numpy seeded to 0 by the harness immediately before the class was constructed from INIT_ARGS (fixture_check.py)
|
| target_loss |
Ran ran; our contract draft was wrong, not the code |
LIONS-EPFL/Charmer/utils_jailbreak.py code served (permissive licence) · get_code("11a3348eb0e4ba60") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 1, dtype family float, shape (2,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Analyzing the function:
# - logits: used as logits[:, loss_slice, :].transpose(1, 2)
# This suggests shape (batch, seq, vocab_size)
# - ids: used as ids[:, target_slice]
# This suggests shape (batch, seq)
# - target_slice: a slice object with .start and .stop attributes
batch_size = 2
seq_len = 5
vocab_size = 4
logits = torch.randn(batch_size, seq_len, vocab_size)
ids = torch.randint(0, vocab_size, (batch_size, seq_len))
target_slice = slice(2, 4)
ARGS = {
"logits": logits,
"ids": ids,
"target_slice": target_slice,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 1, dtype family float, shape (2,) — shape (2,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| update_mu |
Ran ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT) |
LIONS-EPFL/Charmer/baseline/roben/agglom_clusters.py code served (permissive licence) · get_code("12efa3ddd30cf7c2") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 2, dtype family float, shape (3, 2) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# C is expected to be a 2D numpy array (likely cluster assignments or weights)
# f is expected to be a 1D numpy array that gets reshaped to (-1, 1)
# The function performs weighted averaging, so we need compatible shapes
# Let's use:
# - f with shape (2,) - represents 2 samples
# - C with shape (2, 3) - represents 2 samples, 3 clusters
np.random.seed(0)
C = np.array([
[0.8, 0.2, 0.0],
[0.1, 0.6, 0.3]
], dtype=np.float32)
f = np.array([2.0, 3.0], dtype=np.float32)
ARGS = {
"C": C,
"f": f
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 2, dtype family float, shape (3, 2) — shape (3, 2) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| word_edit_distance |
Ran ran; our contract draft was wrong, not the code |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("f0c1289fd2565267") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# The function takes two sequences x and y
# It computes edit distance by comparing elements
# So x and y should be sequences (lists, strings, or arrays) where elements can be compared with ==
# Create two simple sequences to compute edit distance
x = "kitten"
y = "sitting"
ARGS = {
"x": x,
"y": y,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| Charmer |
Not yet run unverified (harvested, no recorded run) |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("59ab5813bed83cb2") How it ran |
| combined_emb_input |
Not yet run unverified (harvested, no recorded run) |
LIONS-EPFL/Charmer/utils.py code served (permissive licence) · get_code("a299d58465c7bf59") How it ran |
| get_llm_embeddings |
Not yet run unverified (harvested, no recorded run) |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("8ae9f9ad369707a9") How it ran |
| get_llm_encoder |
Not yet run unverified (harvested, no recorded run) |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("6297161e122df004") How it ran |
| load_attack_dataset |
Not yet run unverified (harvested, no recorded run) |
LIONS-EPFL/Charmer/dataloader.py code served (permissive licence) · get_code("7f504481b40a6171") How it ran |
| resize_embeddings |
Not yet run unverified (harvested, no recorded run) |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("cb53a78f621babdb") How it ran |
| set_param_grad_off |
Not yet run unverified (harvested, no recorded run) |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("e75d10fb3cfbb60c") How it ran |
| set_param_grad_state |
Not yet run unverified (harvested, no recorded run) |
lions-epfl/charmer/charmer.py code served (permissive licence) · get_code("ea9f4cd8cc668205") How it ran |