SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2402.09497 · 2024

Instruction Tuning for Secure Code Generation

arXiv · PDF · Open in the Atlas

Code that ran

10 of 11 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 2 honoured the contract we drafted, 0 violated it and 0 have no contract verdict; 8 failures of our instrument; 1 no recorded run), measured 2026-11-21; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2402.09497.json · badge: /api/ran/2402.09497.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
eth-sri/safecoder canonical (classifier label)
classifier basis: census
10 ran · 1 no recorded run
FunctionStatusWhere it lives
eval_script_python Ran
ran; our contract draft was wrong, not the code
eth-sri/safecoder/safecoder/human_eval/containerized_eval.py
code served (permissive licence) · get_code("247d5669b5fd3163")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import subprocess
from pathlib import Path
import tempfile
import os

# Create a temporary Python script to evaluate
temp_dir = tempfile.mkdtemp()
script_path = Path(temp_dir) / "test_script.py"

# Write a simple valid Python script
script_content = """
print("Hello, World!")
x = 1 + 1
print(f"Result: {x}")
"""

with open(script_path, 'w') as f:
    f.write(script_content)

ARGS = {
    'path': script_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
eval_string_script Ran
ran; our contract draft was wrong, not the code
eth-sri/safecoder/safecoder/human_eval/containerized_eval.py
code served (permissive licence) · get_code("b1eca2a5e7fedc04")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import tempfile
from pathlib import Path

# Simple test program that will execute successfully
test_program = """
print("Hello, World!")
x = 1 + 1
print(f"Result: {x}")
"""

ARGS = {
    'language': 'python',
    'program': test_program
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_cp_args Ran
ran; our contract draft was wrong, not the code
eth-sri/safecoder/safecoder/utils.py
code served (permissive licence) · get_code("22127156c00606cd")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile

# Create temporary directories to use as class paths
temp_dir1 = tempfile.gettempdir()
temp_dir2 = tempfile.gettempdir()

# Test case 1: empty class_path
info_empty = {
    'class_path': ''
}

# Test case 2: single path in class_path
info_single = {
    'class_path': temp_dir1
}

# Test case 3: multiple paths in class_path (colon-separated)
info_multiple = {
    'class_path': f'{temp_dir1}:{temp_dir2}'
}

# Use the multiple paths case as the main test
ARGS = {
    'info': info_multiple
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_logits_from_lm Ran
ran; our contract draft was wrong, not the code
eth-sri/safecoder/safecoder/trainer.py
code served (permissive licence) · get_code("ad7d9a005c32f9ae")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)

class MockLM(nn.Module):
    def __init__(self, vocab_size=10, hidden_size=4):
        super().__init__()
        self.vocab_size = vocab_size
        self.hidden_size = hidden_size
        self.embedding = nn.Embedding(vocab_size, hidden_size)
        self.lm_head = nn.Linear(hidden_size, vocab_size)
    
    def get_past_from_prefix(self, control_ids):
        # Returns a mock past_key_values structure
        # For a transformer-like model, this would be a tuple of tuples
        # We'll return a simple tensor that can be passed as past_key_values
        batch_size = control_ids.shape[0]
        # Return a mock past structure: tuple of (key, value) pairs for each layer
        return (
            (torch.randn(batch_size, 2, 4, 4), torch.randn(batch_size, 2, 4, 4)),
        )
    
    def forward(self, inputs, past_key_values=None):
        # inputs shape: (batch_size, seq_len)
        x = self.embedding(inputs)  # (batch_size, seq_len, hidden_size)
        logits = self.lm_head(x)  # (batch_size, seq_len, vocab_size)
        
        class Outputs:
            pass
        
        outputs = Outputs()
        outputs.logits = logits
        return outputs

# Create mock LM
lm = MockLM(vocab_size=10, hidden_size=4)

# Create inputs: (batch_size=1, seq_len=5)
inputs = torch.randint(0, 10, (1, 5))

# Create control_ids: (batch_size=1, prefix_len=3)
control_ids = torch.randint(0, 10, (1, 3))

ARGS = {
    "lm": lm,
    "inputs": inputs,
    "control_ids": control_ids,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
pass_at_k Ran
ran, honoured the contract we drafted
eth-sri/safecoder/safecoder/metric.py
code served (permissive licence) · get_code("3a7c7ec16769928a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned float64, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Analyzing the function:
# - n: appears to be an integer (used in np.arange and arithmetic)
# - c: appears to be an integer (used in arithmetic)
# - k: appears to be an integer or array-like (used in division with np.arange result)
# The function computes pass@k metric, commonly used in code generation evaluation
# where n is total samples, c is correct samples, k is the cutoff

# Valid test case: n=10 total samples, c=3 correct, k=2 cutoff
ARGS = {
    "n": 10,
    "c": 3,
    "k": 2
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float64, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
token_weighted_loss Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
eth-sri/safecoder/safecoder/trainer.py
code served (permissive licence) · get_code("7ba8f0a6db856c7d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn.functional as F

torch.manual_seed(0)

# Test setup for token_weighted_loss function
# We need: loss_type, inputs, targets, weights

# For 'ce' and 'nll' loss types:
# inputs: shape (batch, seq_len, num_classes)
# targets: shape (batch, seq_len) with class indices
# weights: shape (batch, seq_len)

# For 'ul' loss type:
# inputs: shape (batch, seq_len, num_classes)
# targets: shape (batch, seq_len, 1) with class indices
# weights: shape (batch, seq_len)

# For 'kl' loss type:
# inputs: shape (batch*seq_len, num_classes) - log probabilities
# targets: shape (batch*seq_len, num_classes) - log probabilities
# weights: shape (batch*seq_len)

batch_size = 2
seq_len = 3
num_classes = 4

# Create test tensors for 'ce' loss type
inputs = torch.randn(batch_size, seq_len, num_classes)
targets = torch.randint(0, num_classes, (batch_size, seq_len))
weights = torch.ones(batch_size, seq_len)
weights[0, 0] = 0  # Set one weight to 0 to test filtering

ARGS = {
    'loss_type': 'ce',
    'inputs': inputs,
    'targets': targets,
    'weights': weights
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
truncate_after Ran
ran; our contract draft was wrong, not the code
eth-sri/safecoder/safecoder/evaler.py
code served (permissive licence) · get_code("780f982b0a1c9b3a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

ARGS = {
    "completion": "Hello world, this is a test string.",
    "trunc_str": "test"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
truncate_after_last Ran
ran; our contract draft was wrong, not the code
eth-sri/safecoder/safecoder/evaler.py
code served (permissive licence) · get_code("f58ac813b10f0912")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function truncate_after_last takes a completion string and a truncation string
# It finds the last occurrence of trunc_str in completion and returns everything up to and including that occurrence

completion = "The quick brown fox jumps over the lazy dog. The end. The end."
trunc_str = "The end"

ARGS = {
    "completion": completion,
    "trunc_str": trunc_str
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
truncate_before Ran
ran; our contract draft was wrong, not the code
eth-sri/safecoder/safecoder/evaler.py
code served (permissive licence) · get_code("99c28062b34f83e2")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function truncate_before takes a completion string and a truncation string
# It finds the truncation string within the completion and returns everything before it
# Both parameters are strings

completion = "Hello world this is a test string"
trunc_str = "test"

ARGS = {
    "completion": completion,
    "trunc_str": trunc_str
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
try_parse Ran
ran, honoured the contract we drafted
eth-sri/safecoder/safecoder/utils.py
code served (permissive licence) · get_code("dcbf19c0a49959fb")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import ast
import tempfile
import os

# Test with Python code (simplest case)
python_code = "x = 1 + 2\nprint(x)"

# Test with Java code (requires class_path)
java_code = """
public class MyTestClass {
    public static void main(String[] args) {
        System.out.println("Hello");
    }
}
"""

# Test with Go code
go_code = """
package main

import "fmt"

func main() {
    fmt.Println("Hello")
}
"""

# Test with JavaScript code
js_code = "const x = 1 + 2; console.log(x);"

# Test with Ruby code
rb_code = "puts 1 + 2"

# Test with JSX code
jsx_code = "const App = () => <div>Hello</div>; export default App;"

# Test with C code
c_code = """
#include <stdio.h>
int main() {
    printf("Hello\\n");
    return 0;
}
"""

ARGS = {
    'code': python_code,
    'info': {
        'language': 'py',
        'class_path': ''
    }
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_model Not yet run
unverified (harvested, no recorded run)
eth-sri/safecoder/safecoder/utils.py
code served (permissive licence) · get_code("a477d8eb91a8ac1b")
How it ran

no invocation recorded.

Also present in other papers' code

Of 11 samples lifted from this paper's linked repositories, 11 carry a function identity and 0 do not. Of the 11 distinct functions with an identity, 1 are also present in other papers whose code we hold (1 with at least one paper dated earlier, 0 the same month, 1 later) and 10 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
pass_at_k 6 other papers
2 dated earlier, 4 later
2206.03865 (2022-06)
Fault-Aware Neural Code Rankers
2306.10763 (2023-06)
Guiding Language Models of Code with Global Context using Monitors
2503.05592 (2025-03)
R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning
and 3 more not named here

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-21.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Modern language models (LMs) have gained widespread acceptance in everyday and professional contexts, particularly in programming. An essential procedure enabling this adoption is instruction tuning, which substantially enhances LMs' practical utility by training them to follow user instructions and human preferences. However, existing instruction tuning schemes overlook a crucial aspect: the security of generated code. As a result, even the state-of-the-art instruction-tuned LMs frequently produce unsafe code, posing significant security risks. In this work, we introduce SafeCoder to address this gap. SafeCoder performs security-centric fine-tuning using a diverse and high-quality dataset that we collected using an automated pipeline. We integrate the security fine-tuning with standard instruction tuning, to facilitate a joint optimization of both security and utility. Despite its simplicity, we show that SafeCoder is effective across a variety of popular LMs and datasets. It is able to drastically improve security (by about 30%), while preserving utility.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2402.09497")
get_code_for_paper("2402.09497")
have("2402.09497")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2402.09497.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2402.09497.svg)](https://syntology.ai/paper/2402.09497)

Connect an agent — have() is free.