6 of 7 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 0 honoured the contract we drafted, 0 violated it and 0 have no contract verdict; 6 failures of our instrument; 1 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.
"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2406.08464.json · badge: /api/ran/2406.08464.svg
Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.
| Repository | Role | Ran |
|---|---|---|
| magpie-align/magpie | canonical (classifier label) classifier basis: census |
6 ran · 1 no recorded run |
| Function | Status | Where it lives |
|---|---|---|
| input_classification | Ran ran; our contract draft was wrong, not the code |
magpie-align/magpie/exp/str_utils.py code served (permissive licence) · get_code("d3cf2c66839647d5") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
|
| input_difficulty_rating | Ran ran; our contract draft was wrong, not the code |
magpie-align/magpie/exp/str_utils.py code served (permissive licence) · get_code("45fc1188241aeb24") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
|
| input_quality_rating | Ran ran; our contract draft was wrong, not the code |
magpie-align/magpie/exp/str_utils.py code served (permissive licence) · get_code("509f95bf20449536") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
|
| load_dataset_from_file | Ran ran; our contract draft was wrong, not the code |
magpie-align/magpie/exp/utils.py code served (permissive licence) · get_code("99433f4e746df6bf") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
|
| load_jsonl_to_list | Ran ran; our contract draft was wrong, not the code |
magpie-align/magpie/exp/utils.py code served (permissive licence) · get_code("f3dd95b90f7ac7d4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
|
| process_engine_responses | Ran ran; our contract draft was wrong, not the code |
magpie-align/magpie/exp/unitag.py code served (permissive licence) · get_code("73b99e147986cc52") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
|
| make_api_request_with_retry | Not yet run unverified (harvested, no recorded run) |
magpie-align/magpie/exp/utils.py code served (permissive licence) · get_code("ec2e2e4d69080db8") How it ranno invocation recorded. |
Of 7 samples lifted from this paper's linked repositories, 7 carry a function identity and 0 do not. Of the 7 distinct functions with an identity, 2 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 2 later) and 5 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).
Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.
| Function | Also present in | Earliest-dated | Others named |
|---|---|---|---|
| load_dataset_from_file | 1 other paper 1 later |
2505.14625 (2025-05) TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning |
|
| make_api_request_with_retry | 1 other paper 1 later |
2505.14625 (2025-05) TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning |
Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human labor costs and a limited, predefined scope for prompting prevent existing open-source data creation methods from scaling effectively, potentially limiting the diversity and quality of public alignment datasets. Is it possible to synthesize high-quality instruction data at scale by extracting it directly from an aligned LLM? We present a self-synthesis method for generating large-scale alignment data named Magpie. Our key observation is that aligned LLMs like Llama-3-Instruct can generate a user query when we input only the left-side templates up to the position reserved for user messages, thanks to their auto-regressive nature. We use this method to prompt Llama-3-Instruct and generate 4 million instructions along with their corresponding responses. We perform a comprehensive analysis of the extracted data and select 300K high-quality instances. To compare Magpie data with other public instruction datasets, we fine-tune Llama-3-8B-Base with each dataset and evaluate the performance of the fine-tuned models. Our results indicate that in some tasks, models fine-tuned with Magpie perform comparably to the official Llama-3-8B-Instruct, despite the latter being enhanced with 10 million data points through supervised fine-tuning (SFT) and subsequent feedback learning. We also show that using Magpie solely for SFT can surpass the performance of previous public datasets utilized for both SFT and preference optimization, such as direct preference optimization with UltraFeedback. This advantage is evident on alignment benchmarks such as AlpacaEval, ArenaHard, and WildBench.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2406.08464")
get_code_for_paper("2406.08464")
have("2406.08464")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2406.08464.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2406.08464)
Connect an agent — have() is free.