SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2406.08464 · 2024

Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing

arXiv · PDF · Open in the Atlas

Code that ran

6 of 7 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 0 honoured the contract we drafted, 0 violated it and 0 have no contract verdict; 6 failures of our instrument; 1 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2406.08464.json · badge: /api/ran/2406.08464.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
magpie-align/magpie canonical (classifier label)
classifier basis: census
6 ran · 1 no recorded run
FunctionStatusWhere it lives
input_classification Ran
ran; our contract draft was wrong, not the code
magpie-align/magpie/exp/str_utils.py
code served (permissive licence) · get_code("d3cf2c66839647d5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

ARGS = {
    "input": "How do I write a Python function to sort a list?"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
input_difficulty_rating Ran
ran; our contract draft was wrong, not the code
magpie-align/magpie/exp/str_utils.py
code served (permissive licence) · get_code("45fc1188241aeb24")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

ARGS = {
    "input": "How do I implement a neural network for image classification?"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
input_quality_rating Ran
ran; our contract draft was wrong, not the code
magpie-align/magpie/exp/str_utils.py
code served (permissive licence) · get_code("509f95bf20449536")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function input_quality_rating takes a single parameter 'input' which is a string
# It constructs a user message by formatting the input into a prompt template
# No complex objects needed - just a string input

ARGS = {
    "input": "How do I improve my programming skills?"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_dataset_from_file Ran
ran; our contract draft was wrong, not the code
magpie-align/magpie/exp/utils.py
code served (permissive licence) · get_code("99433f4e746df6bf")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
import os

# Create temporary files for testing
temp_dir = tempfile.mkdtemp()

# Create a test .json file
json_file_path = os.path.join(temp_dir, "test.json")
test_json_data = [
    {"id": 1, "text": "hello"},
    {"id": 2, "text": "world"}
]
with open(json_file_path, 'w') as f:
    json.dump(test_json_data, f)

# Create a test .jsonl file
jsonl_file_path = os.path.join(temp_dir, "test.jsonl")
with open(jsonl_file_path, 'w') as f:
    f.write(json.dumps({"id": 1, "text": "hello"}) + "\n")
    f.write(json.dumps({"id": 2, "text": "world"}) + "\n")

# Test with .json file
ARGS = {
    "filename": json_file_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_jsonl_to_list Ran
ran; our contract draft was wrong, not the code
magpie-align/magpie/exp/utils.py
code served (permissive licence) · get_code("f3dd95b90f7ac7d4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import tempfile
import os

# Create a temporary JSONL file with sample data
temp_dir = tempfile.mkdtemp()
jsonl_file_path = os.path.join(temp_dir, "test_data.jsonl")

# Write sample JSONL data
with open(jsonl_file_path, 'w') as f:
    f.write(json.dumps({"id": 1, "text": "hello"}) + "\n")
    f.write(json.dumps({"id": 2, "text": "world"}) + "\n")
    f.write(json.dumps({"id": 3, "value": 42}) + "\n")

ARGS = {
    "jsonl_file_path": jsonl_file_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
process_engine_responses Ran
ran; our contract draft was wrong, not the code
magpie-align/magpie/exp/unitag.py
code served (permissive licence) · get_code("73b99e147986cc52")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import argparse

def get_args():
    parser = argparse.ArgumentParser(description='Unified Tagging Manager.')
    parser.add_argument('--tag_mission', type=str, default='quality', help='The tagging mission.', choices=['difficulty', 'quality', 'classification', 'safety', 'reward', 'language'])
    parser.add_argument('--model_path', type=str, default='meta-llama/Meta-Llama-3-8B-Instruct', help='Tag Model.')
    parser.add_argument('--guard_model_path', type=str, default='meta-llama/Meta-Llama-Guard-2-8B', help='Guard Model.')
    parser.add_argument('--reward_model_path', type=str, default='sfairXC/FsfairX-LLaMA3-RM-v0.1', help='Reward Model.')
    parser.add_argument('--input_file', type=str, default=None, help='Input dataset file name')
    parser.add_argument('--batch_size', type=int, default=1000, help='Number of samples per batch. Online <100, Offline <200.')
    parser.add_argument('--checkpoint_every', type=int, default=40, help='Save checkpoint every n batches')
    parser.add_argument('--api', type=bool, default=False, help='Use API to generate responses')
    parser.add_argument('--offline', action='store_false', dest='api', help='Use local vllm engine')
    parser.add_argument('--online', action='store_true', dest='api', help='Use Together API engine')
    parser.add_argument('--api_url', type=str, default='https://api.together.xyz/v1/chat/completions', help='API URL')
    parser.add_argument('--api_key', type=str, default=None, help='Together API Key')
    parser.add_argument('--debug', type=bool, default=False, help='Debug mode. Only process the first 100 samples.')
    parser.add_argument('--save_as', type=str, default='json', choices=['json', 'jsonl'], help='Save the generated responses as a what kind of file')
    parser.add_argument('--device', type=str, default='0')
    parser.add_argument('--tensor_parallel_size', type=int, default=1)
    parser.add_argument('--dtype', type=str, default='bfloat16', choices=['float16', 'bfloat16'])
    parser.add_argument('--quantization', type=str, default='fp8', choices=['fp8', 'awq', 'gptq', 'None'])
    parser.add_argument('--kv_cache_dtype', type=str, default='auto', choices=['auto', 'fp8'])
    parser.add_argument('--max_model_len', type=int, default=4096)
    parser.add_argument('--gpu_memory_utilization', type=float, default=0.95)
    parser.add_argument('--max_tokens', type=int, default=1024)
    parser.add_argument('--temperature', type=float, default=0)
    parser.add_argument('--repetition_penalty', type=float, default=1.0)
    return parser.parse_args()

args = get_args()

MODEL_NAME = args.model_path

mission = args.tag_mission

def process_engine_responses(response, item, mission):
    try:
        tags_json = json.loads(response)
        if mission == 'difficulty':
            item['intent'] = tags_json['intent']
            item['knowledge'] = tags_json['knowledge']
            item['difficulty'] = tags_json['difficulty']
            item['difficulty_generator'] = MODEL_NAME
        elif mission == 'quality':
            item['input_quality'] = tags_json['input_quality']
            item['quality_explanation'] = tags_json['explanation']
            item['quality_generator'] = MODEL_NAME
        elif mission == 'classification':
            item['task_category'] = tags_json['primary_tag']
            item['other_task_category'] = tags_json['other_tags']
            item['task_category_generator'] = MODEL_NAME
    except Exception as e:
        print(f'[unitag.py] Failed to process item with error: {str(e)}')
        print(f'[unitag.py] Raw response from LLM tagger: {response}')
        if mission == 'difficulty':
            item['intent'] = None
            item['knowledge'] = None
            item['difficulty'] = None
            item['difficulty_generator'] = None
        elif mission == 'quality':
            item['input_quality'] = None
            item['quality_explanation'] = None
            item['quality_generator'] = None
        elif mission == 'classification':
            item['task_category'] = None
            item['other_task_category'] = None
            item['task_category_generator'] = None
    return item

ARGS = {
    'response': json.dumps({
        'input_quality': 'high',
        'explanation': 'This is a well-formed question.',
        'primary_tag': 'math',
        'other_tags': ['algebra'],
        'intent': 'learn',
        'knowledge': 'intermediate',
        'difficulty': 'medium'
    }),
    'item': {
        'id': 1,
        'text': 'What is 2 + 2?',
    },
    'mission': 'quality'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
make_api_request_with_retry Not yet run
unverified (harvested, no recorded run)
magpie-align/magpie/exp/utils.py
code served (permissive licence) · get_code("ec2e2e4d69080db8")
How it ran

no invocation recorded.

Also present in other papers' code

Of 7 samples lifted from this paper's linked repositories, 7 carry a function identity and 0 do not. Of the 7 distinct functions with an identity, 2 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 2 later) and 5 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
load_dataset_from_file 1 other paper
1 later
2505.14625 (2025-05)
TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning
make_api_request_with_retry 1 other paper
1 later
2505.14625 (2025-05)
TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human labor costs and a limited, predefined scope for prompting prevent existing open-source data creation methods from scaling effectively, potentially limiting the diversity and quality of public alignment datasets. Is it possible to synthesize high-quality instruction data at scale by extracting it directly from an aligned LLM? We present a self-synthesis method for generating large-scale alignment data named Magpie. Our key observation is that aligned LLMs like Llama-3-Instruct can generate a user query when we input only the left-side templates up to the position reserved for user messages, thanks to their auto-regressive nature. We use this method to prompt Llama-3-Instruct and generate 4 million instructions along with their corresponding responses. We perform a comprehensive analysis of the extracted data and select 300K high-quality instances. To compare Magpie data with other public instruction datasets, we fine-tune Llama-3-8B-Base with each dataset and evaluate the performance of the fine-tuned models. Our results indicate that in some tasks, models fine-tuned with Magpie perform comparably to the official Llama-3-8B-Instruct, despite the latter being enhanced with 10 million data points through supervised fine-tuning (SFT) and subsequent feedback learning. We also show that using Magpie solely for SFT can surpass the performance of previous public datasets utilized for both SFT and preference optimization, such as direct preference optimization with UltraFeedback. This advantage is evident on alignment benchmarks such as AlpacaEval, ArenaHard, and WildBench.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2406.08464")
get_code_for_paper("2406.08464")
have("2406.08464")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2406.08464.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2406.08464.svg)](https://syntology.ai/paper/2406.08464)

Connect an agent — have() is free.