SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2406.19371 · 2024

Suri: Multi-constraint Instruction Following for Long-form Text Generation

arXiv · PDF · Open in the Atlas

Code that ran

12 of 15 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 1 honoured the contract we drafted, 2 violated it and 1 has no contract verdict; 8 failures of our instrument; 3 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2406.19371.json · badge: /api/ran/2406.19371.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
chtmp223/suri canonical (classifier label)
classifier basis: census
12 ran · 3 no recorded run
FunctionStatusWhere it lives
apply_chat_template Ran
ran; our contract draft was wrong, not the code
chtmp223/suri/ft/lib/alignment_mod/data.py
pointer only (licence: NONE) · get_code("ed74b164b2fe6a65")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, chtmp223/suri when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

build_generator Ran
ran; our contract draft was wrong, not the code
chtmp223/suri/eval/automatic/ranking.py
pointer only (licence: NONE) · get_code("67cf0babead8c2de")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, chtmp223/suri when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

check_bib Ran
ran, violated the contract we drafted
chtmp223/suri/utils.py
pointer only (licence: NONE) · get_code("cc64fc0500364fe4")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, chtmp223/suri when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

convert_to_scalar Ran
ran; our contract draft was wrong, not the code
chtmp223/suri/ft/lib/trl_mod/core.py
pointer only (licence: NONE) · get_code("c32b636f13ed9ee5")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, chtmp223/suri when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

extract_docstring Ran
ran; our contract draft was wrong, not the code
chtmp223/suri/ft/lib/alignment_mod/decontaminate.py
pointer only (licence: NONE) · get_code("0edeacd7a9034f53")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, chtmp223/suri when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

is_openai_format Ran
ran, violated the contract we drafted
chtmp223/suri/ft/lib/alignment_mod/data.py
pointer only (licence: NONE); the stored copy get_code serves, from tengxiao1/cal-dpo, is pointer only (licence: NONE) · get_code("4253aac010c4504f")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, chtmp223/suri when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

load_dataset_column Ran
ran on a synthesized input
chtmp223/suri/ft/lib/alignment_mod/decontaminate.py
pointer only (licence: NONE) · get_code("a3791c5c2185e013")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, chtmp223/suri when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

normalize_whitespace Ran
ran; our contract draft was wrong, not the code
chtmp223/suri/ft/lib/alignment_mod/decontaminate.py
pointer only (licence: NONE) · get_code("f43e59d239019e86")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, chtmp223/suri when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

sequence_prob Ran
ran, honoured the contract we drafted
chtmp223/suri/eval/automatic/ranking.py
pointer only (licence: NONE) · get_code("5bffca7d5514764e")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, chtmp223/suri when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

strip_trailing_text Ran
ran; our contract draft was wrong, not the code
chtmp223/suri/utils.py
pointer only (licence: NONE) · get_code("1db337e94d1e9417")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, chtmp223/suri when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

top_k_top_p_filtering Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
chtmp223/suri/ft/lib/trl_mod/core.py
pointer only (licence: NONE) · get_code("5bcf8e086423cf6b")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, chtmp223/suri when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

truncating_words Ran
ran; our contract draft was wrong, not the code
chtmp223/suri/utils.py
pointer only (licence: NONE) · get_code("9a2951468cd4ee8b")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, chtmp223/suri when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

flatten_dict Not yet run
unverified (harvested, no recorded run)
chtmp223/suri/ft/lib/trl_mod/core.py
pointer only (licence: NONE) · get_code("a51ae45a1f6c0b5b")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, chtmp223/suri when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

is_transformers_greater_than Not yet run
unverified (harvested, no recorded run)
chtmp223/suri/ft/lib/trl_mod/import_utils.py
pointer only (licence: NONE) · get_code("2a9ca02cb6e196ff")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, chtmp223/suri when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

read_file Not yet run
unverified (harvested, no recorded run)
chtmp223/suri/eval/automatic/ranking.py
pointer only (licence: NONE) · get_code("8f7c20be180ad75b")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, chtmp223/suri when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

Also present in other papers' code

Of 15 samples lifted from this paper's linked repositories, 15 carry a function identity and 0 do not. Of the 15 distinct functions with an identity, 8 are also present in other papers whose code we hold (8 with at least one paper dated earlier, 6 the same month, 8 later) and 7 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
top_k_top_p_filtering 21 other papers
5 dated earlier, 16 later
2305.12191 (2023-05)
Pointwise Mutual Information Based Metric and Decoding Strategy for Faithful Generation in Document Grounded Dialogs
2305.18290 (2023-05)
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
2311.17911 (2023-11)
OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
and 18 more not named here
convert_to_scalar 18 other papers
1 dated earlier, 1 the same month, 16 later
2305.18290 (2023-05)
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
2406.11280 (2024-06)
ISR-DPO: Aligning Large Multimodal Models for Videos by Iterative Self-Retrospective DPO
2410.06101 (2024-10)
Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning
and 15 more not named here
flatten_dict 18 other papers
1 dated earlier, 1 the same month, 16 later
2305.18290 (2023-05)
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
2406.11280 (2024-06)
ISR-DPO: Aligning Large Multimodal Models for Videos by Iterative Self-Retrospective DPO
2410.06101 (2024-10)
Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning
and 15 more not named here
is_openai_format 12 other papers
1 dated earlier, 1 the same month, 10 later
2405.14734 (2024-05)
SimPO: Simple Preference Optimization with a Reference-Free Reward
2406.08414 (2024-06)
Discovering Preference Optimization Algorithms with and for Large Language Models
2407.09014 (2024-07)
COMPACT: Compressing Retrieved Documents Actively for Question Answering
and 9 more not named here
extract_docstring 10 other papers
1 dated earlier, 1 the same month, 8 later
2405.14734 (2024-05)
SimPO: Simple Preference Optimization with a Reference-Free Reward
2406.08414 (2024-06)
Discovering Preference Optimization Algorithms with and for Large Language Models
2407.09014 (2024-07)
COMPACT: Compressing Retrieved Documents Actively for Question Answering
and 7 more not named here
normalize_whitespace 9 other papers
1 dated earlier, 1 the same month, 7 later
2405.14734 (2024-05)
SimPO: Simple Preference Optimization with a Reference-Free Reward
2406.08414 (2024-06)
Discovering Preference Optimization Algorithms with and for Large Language Models
2407.09014 (2024-07)
COMPACT: Compressing Retrieved Documents Actively for Question Answering
and 6 more not named here
load_dataset_column 8 other papers
1 dated earlier, 1 the same month, 6 later
2405.14734 (2024-05)
SimPO: Simple Preference Optimization with a Reference-Free Reward
2406.08414 (2024-06)
Discovering Preference Optimization Algorithms with and for Large Language Models
2407.09014 (2024-07)
COMPACT: Compressing Retrieved Documents Actively for Question Answering
and 5 more not named here
is_transformers_greater_than 3 other papers
1 dated earlier, 2 later
2305.18290 (2023-05)
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
2507.04976 (2025-07)
Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Models
2509.12521 (2025-09)
Phi: Preference Hijacking in Multi-modal Large Language Models at Inference Time

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Existing research on instruction following largely focuses on tasks with simple instructions and short responses. In this work, we explore multi-constraint instruction following for generating long-form text. We create Suri, a dataset with 20K human-written long-form texts paired with LLM-generated backtranslated instructions that contain multiple complex constraints. Because of prohibitive challenges associated with collecting human preference judgments on long-form texts, preference-tuning algorithms such as DPO are infeasible in our setting; thus, we propose Instructional ORPO (I-ORPO), an alignment method based on the ORPO algorithm. Instead of receiving negative feedback from dispreferred responses, I-ORPO obtains negative feedback from synthetically corrupted instructions generated by an LLM. Using Suri, we perform supervised and I-ORPO fine-tuning on Mistral-7b-Instruct-v0.2. The resulting models, Suri-SFT and Suri-I-ORPO, generate significantly longer texts (~5K tokens) than base models without significant quality deterioration. Our human evaluation shows that while both SFT and I-ORPO models satisfy most constraints, Suri-I-ORPO generations are generally preferred for their coherent and informative incorporation of the constraints. We release our code at https://github.com/chtmp223/suri.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2406.19371")
get_code_for_paper("2406.19371")
have("2406.19371")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2406.19371.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2406.19371.svg)](https://syntology.ai/paper/2406.19371)

Connect an agent — have() is free.