SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2606.09124 · ICML · 2026

A Regret Minimization Framework on Preference Learning in Large Language Models

Geon-Hyeong Kim, Youngsoo Jang, Moontae Lee, Jungwoo Lee, Taehyun Cho, Suhwan Kim, Yu Kim

arXiv · PDF · Open in the Atlas

Code that ran

15 of 18 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 1 honoured the contract we drafted, 1 violated it and 0 have no contract verdict; 13 failures of our instrument; 3 no recorded run), measured 2026-11-22; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2606.09124.json · badge: /api/ran/2606.09124.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
YSLIU627/Regularized-Preference-Optimization canonical (classifier label)
classifier basis: census
9 ran · 1 no recorded run
OpenRLHF/OpenRLHF canonical (classifier label)
classifier basis: census
5 ran · 2 no recorded run
Vance0124/Token-level-Direct-Preference-Optimization community
Papers with Code archive
1 ran · 0 no recorded run
FunctionStatusWhere it lives
aggregate_loss Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
OpenRLHF/OpenRLHF/openrlhf/models/loss.py
code served (permissive licence) · get_code("0edeb138e07fbecc")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import math

torch.manual_seed(0)

def masked_mean(tensor, mask, dim=None):
    """Helper function that the aggregate_loss function calls"""
    if dim is None:
        return (tensor * mask).sum() / (mask.sum() + 1e-08)
    return (tensor * mask).sum(dim=dim) / (mask.sum(dim=dim) + 1e-08)

# Create small test tensors
batch_size = 2
seq_length = 3
loss = torch.randn(batch_size, seq_length)
loss_mask = torch.ones(batch_size, seq_length)
loss_mask[0, 2] = 0  # mask out one token

# Calculate batch_num_tokens (global token count)
batch_num_tokens = loss_mask.sum().item()

# Calculate global_batch_size (global sample count)
global_batch_size = 2.0

# Build ARGS dict with all parameters
ARGS = {
    "loss": loss,
    "loss_mask": loss_mask,
    "token_level_loss": True,
    "dp_size": 1,
    "batch_num_tokens": batch_num_tokens,
    "global_batch_size": global_batch_size,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
apply_chat_template Ran
ran; our contract draft was wrong, not the code
YSLIU627/Regularized-Preference-Optimization/alignment-handbook/src/alignment/data.py
code served (permissive licence) · get_code("5a84f9e5640fe497")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np
from unittest.mock import Mock

torch.manual_seed(0)

# Create a minimal tokenizer mock that supports the required methods
class MinimalTokenizer:
    def __init__(self):
        self.chat_template = "{% for message in messages %}{{ message['role'] }}: {{ message['content'] }}\n{% endfor %}"
    
    def get_chat_template(self):
        return self.chat_template
    
    def apply_chat_template(self, messages, tokenize=False, add_generation_prompt=False):
        """Convert messages list to a formatted string"""
        result = ""
        for msg in messages:
            result += f"{msg['role']}: {msg['content']}\n"
        if add_generation_prompt:
            result += "assistant: "
        return result

# Create tokenizer instance
tokenizer = MinimalTokenizer()

# Create example for 'sft' task
example_sft = {
    'messages': [
        {'role': 'user', 'content': 'Hello'},
        {'role': 'assistant', 'content': 'Hi there!'}
    ]
}

# Create example for 'generation' task
example_generation = {
    'messages': [
        {'role': 'user', 'content': 'What is 2+2?'},
        {'role': 'assistant', 'content': '4'}
    ]
}

# Create example for 'rm' task
example_rm = {
    'chosen': [
        {'role': 'user', 'content': 'Question'},
        {'role': 'assistant', 'content': 'Good answer'}
    ],
    'rejected': [
        {'role': 'user', 'content': 'Question'},
        {'role': 'assistant', 'content': 'Bad answer'}
    ]
}

# Create example for 'dpo' task
example_dpo = {
    'prompt': [
        {'role': 'system', 'content': ''},
        {'role': 'user', 'content': 'Prompt'}
    ],
    'chosen': [
        {'role': 'assistant', 'content': 'Better response'}
    ],
    'rejected': [
        {'role': 'assistant', 'content': 'Worse response'}
    ]
}

# Test with 'sft' task
ARGS = {
    'example': example_sft,
    'tokenizer': tokenizer,
    'task': 'sft',
    'auto_insert_empty_system_msg': True
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
compute_approx_kl Ran
ran; our contract draft was wrong, not the code
OpenRLHF/OpenRLHF/openrlhf/models/utils.py
code served (permissive licence) · get_code("c146f3cb0e5259a0")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Create small log probability tensors
batch_size = 2
vocab_size = 4

log_probs = torch.randn(batch_size, vocab_size)
log_probs_base = torch.randn(batch_size, vocab_size)
log_probs_old = torch.randn(batch_size, vocab_size)

ARGS = {
    'log_probs': log_probs,
    'log_probs_base': log_probs_base,
    'log_probs_old': log_probs_old,
    'kl_estimator': 'k1',
    'unbiased_gradient': False,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
compute_approx_kl Ran
ran, honoured the contract we drafted
YSLIU627/Regularized-Preference-Optimization/OpenRLHF/openrlhf/models/utils.py
code served (permissive licence) · get_code("60d73ddf6afc1bc3")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Create small tensors for log probabilities
# Shape: (batch_size=2, num_actions=4)
log_probs = torch.randn(2, 4, dtype=torch.float32)
log_probs_base = torch.randn(2, 4, dtype=torch.float32)

# Create action mask with same shape
# Using values of 1.0 to indicate valid actions, 0.0 for masked
action_mask = torch.ones(2, 4, dtype=torch.float32)

ARGS = {
    "log_probs": log_probs,
    "log_probs_base": log_probs_base,
    "action_mask": action_mask,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
compute_reward Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
OpenRLHF/OpenRLHF/openrlhf/models/utils.py
code served (permissive licence) · get_code("812f2e3f5cda7c93")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Analyzing the function:
# - r: Union[torch.Tensor, float] - used with .clamp() and .unsqueeze(1), so torch.Tensor
# - kl_coef: float - compared and used in multiplication
# - kl: Union[torch.Tensor, list[torch.Tensor]] - used with operations like size(), long(), fliplr(), argmax(), scatter_
# - action_mask: Optional[torch.Tensor] - used with .size(1), .long(), .fliplr(), .argmax()
# - reward_clip_range: Tuple[float, float] - checked in if statement

# The function operates on 2D tensors with shape [batch, seq_len]
batch_size = 2
seq_len = 4

# r: reward tensor, shape [batch]
r = torch.randn(batch_size)

# kl_coef: coefficient for KL divergence penalty
kl_coef = 0.5

# kl: KL divergence tensor, shape [batch, seq_len]
kl = torch.randn(batch_size, seq_len)

# action_mask: binary mask indicating valid positions, shape [batch, seq_len]
# Values should be mostly 1s with 0s at the end to mark sequence boundaries
action_mask = torch.ones(batch_size, seq_len)
action_mask[0, 2:] = 0  # First sequence ends at index 1
action_mask[1, 3:] = 0  # Second sequence ends at index 2

# reward_clip_range: tuple for clipping rewards
reward_clip_range = (-1.0, 1.0)

ARGS = {
    "r": r,
    "kl_coef": kl_coef,
    "kl": kl,
    "action_mask": action_mask,
    "reward_clip_range": reward_clip_range,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_docstring Ran
ran; our contract draft was wrong, not the code
YSLIU627/Regularized-Preference-Optimization/alignment-handbook/src/alignment/decontaminate.py
code served (permissive licence) · get_code("0edeacd7a9034f53")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

# This function extracts docstrings from Python code prompts
# It looks for triple-quoted strings and returns the content between them

# Test case 1: docstring with """ delimiters (2 occurrences)
prompt_1 = '''def foo():
    """This is a docstring"""
    pass
'''

# Test case 2: docstring with """ delimiters (4 occurrences - nested or multiple)
prompt_2 = '''def bar():
    """First docstring"""
    x = """some string literal"""
    """Second docstring"""
    pass
'''

# Test case 3: docstring with ''' delimiters
prompt_3 = """def baz():
    '''This is another docstring'''
    pass
"""

# Use the first prompt as the main test case
ARGS = {
    "prompt": prompt_1
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_loss_batch_info Ran
ran; our contract draft was wrong, not the code
OpenRLHF/OpenRLHF/openrlhf/utils/loss_utils.py
code served (permissive licence) · get_code("557aef158c28bb84")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Build a minimal strategy object that the function may query for ds_device_mesh
class MinimalStrategy:
    def __init__(self):
        self.ds_device_mesh = None

strategy = MinimalStrategy()

# Build a minimal replay_buffer with the required attributes
class MinimalReplayBuffer:
    def __init__(self):
        self.dynamic_batch_num_tokens = {0: 8.0, 1: 10.0}
        self.dynamic_global_batch_size = {0: 2.0, 1: 3.0}

replay_buffer = MinimalReplayBuffer()

# Create a loss_mask tensor: shape (2, 3) representing a batch of 2 samples with 3 tokens each
loss_mask = torch.ones(2, 3, dtype=torch.float32)

ARGS = {
    'strategy': strategy,
    'loss_mask': loss_mask,
    'replay_buffer': replay_buffer,
    'step': 0,
    'dynamic_batch': False,
    'batch_num_tokens': None,
    'global_batch_size': None,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_seqlens_in_batch Ran
ran; our contract draft was wrong, not the code
YSLIU627/Regularized-Preference-Optimization/OpenRLHF/openrlhf/models/packing_utils.py
code served (permissive licence) · get_code("a9fb683d7652195d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 1, dtype family int, shape (5,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# The function takes an attention_mask tensor that contains sequence group IDs
# (0 for padding, 1+ for actual sequence groups)
# and returns the lengths of each sequence group
attention_mask = torch.tensor([
    [1, 1, 2, 2, 2, 0],
    [1, 2, 2, 3, 3, 3],
], dtype=torch.long)

ARGS = {
    "attention_mask": attention_mask,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 1, dtype family int, shape (5,) — shape (5,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_unpad_data Ran
ran; our contract draft was wrong, not the code
YSLIU627/Regularized-Preference-Optimization/OpenRLHF/openrlhf/models/packing_utils.py
code served (permissive licence) · get_code("08e261d2bbc45dbc")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn.functional as F

torch.manual_seed(0)

# Create a small attention mask with batch size 2 and sequence length 6
# The mask uses integers where 0 means padding, and 1,2,3 represent different sequence IDs
# Example from docstring:
# [
#     [1, 1, 2, 2, 2, 0],
#     [1, 2, 2, 3, 3, 3],
# ]
attention_mask = torch.tensor(
    [
        [1, 1, 2, 2, 2, 0],
        [1, 2, 2, 3, 3, 3],
    ],
    dtype=torch.int64
)

ARGS = {
    "attention_mask": attention_mask,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
is_openai_format Ran
ran, violated the contract we drafted
YSLIU627/Regularized-Preference-Optimization/alignment-handbook/src/alignment/data.py
pointer only (licence: Apache-2.0); the stored copy get_code serves, from tengxiao1/cal-dpo, is pointer only (licence: NONE) · get_code("4253aac010c4504f")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (Apache-2.0), but get_code serves the stored copy, first harvested from tengxiao1/cal-dpo, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

log_probs_from_logits Ran
ran; our contract draft was wrong, not the code
YSLIU627/Regularized-Preference-Optimization/OpenRLHF/openrlhf/models/utils.py
code served (permissive licence) · get_code("959df7824bf09cc1")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned Tensor, ndim 1, dtype family float, shape (2,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn.functional as F

torch.manual_seed(0)

# Create small tensors for testing
# logits: shape (batch_size, num_classes) = (2, 4)
logits = torch.randn(2, 4)

# labels: shape (batch_size,) with class indices in range [0, num_classes)
labels = torch.tensor([0, 2])

ARGS = {
    "logits": logits,
    "labels": labels,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 1, dtype family float, shape (2,) — shape (2,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
log_probs_from_logits Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
OpenRLHF/OpenRLHF/openrlhf/models/utils.py
code served (permissive licence) · get_code("75afa2affd2a8e36")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family float, shape (2, 3) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Create small tensors for testing
# logits: shape (batch_size, seq_len, vocab_size) = (2, 3, 4)
# labels: shape (batch_size, seq_len) = (2, 3)
batch_size = 2
seq_len = 3
vocab_size = 4

logits = torch.randn(batch_size, seq_len, vocab_size, dtype=torch.float32)
labels = torch.randint(0, vocab_size, (batch_size, seq_len), dtype=torch.long)
temperature = 1.0

ARGS = {
    "logits": logits,
    "labels": labels,
    "temperature": temperature,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
flash_attn, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 2, dtype family float, shape (2, 3) — shape (2, 3) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
normalize_whitespace Ran
ran; our contract draft was wrong, not the code
YSLIU627/Regularized-Preference-Optimization/alignment-handbook/src/alignment/decontaminate.py
code served (permissive licence) · get_code("f43e59d239019e86")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

text_input = "hello    world  \n  foo   bar"

ARGS = {
    "text": text_input
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
preprocess_data Ran
ran; our contract draft was wrong, not the code
YSLIU627/Regularized-Preference-Optimization/OpenRLHF/openrlhf/datasets/unpaired_preference_dataset.py
code served (permissive licence) · get_code("f4db4c93fe22b070")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Stand-in class for chat template function
class ChatTemplateFunc:
    def __call__(self, messages, tokenize=False, add_generation_prompt=False):
        """
        Simple mock chat template that concatenates messages and adds prompt marker if needed.
        """
        if isinstance(messages, list):
            result = "".join(str(m) for m in messages)
        else:
            result = str(messages)
        
        if add_generation_prompt:
            result = result + "[PROMPT]"
        
        return result

# Create sample data dictionary
data = {
    "input": [{"role": "user", "content": "Hello"}],
    "output": [{"role": "assistant", "content": "Hi there"}],
    "label": 1
}

# Alternative data for the else branch (when output_key is None)
data_alt = {
    "input": [{"role": "user", "content": "Hello"}, {"role": "assistant", "content": "Hi"}],
    "label": 0
}

# String template for input
input_template_str = "Question: {}"

# Chat template function instance
chat_template_func = ChatTemplateFunc()

ARGS = {
    "data": data,
    "input_template": input_template_str,
    "input_key": "input",
    "output_key": "output",
    "label_key": "label",
    "apply_chat_template": chat_template_func
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
tdpo_loss Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
Vance0124/Token-level-Direct-Preference-Optimization/trainers.py
code served (permissive licence) · get_code("cbd5b92bbd58d7d3")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 2, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn.functional as F

torch.manual_seed(0)

batch_size = 2

# Create small tensors for the function arguments
chosen_logps_margin = torch.randn(batch_size, dtype=torch.float32)
rejected_logps_margin = torch.randn(batch_size, dtype=torch.float32)
chosen_position_kl = torch.randn(batch_size, dtype=torch.float32)
rejected_position_kl = torch.randn(batch_size, dtype=torch.float32)

# Define scalar parameters
beta = 0.3
alpha = 0.5
if_tdpo2 = True

ARGS = {
    "chosen_logps_margin": chosen_logps_margin,
    "rejected_logps_margin": rejected_logps_margin,
    "chosen_position_kl": chosen_position_kl,
    "rejected_position_kl": rejected_position_kl,
    "beta": beta,
    "alpha": alpha,
    "if_tdpo2": if_tdpo2,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 2, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
compute_reward Not yet run
unverified (harvested, no recorded run)
YSLIU627/Regularized-Preference-Optimization/OpenRLHF/openrlhf/models/utils.py
code served (permissive licence) · get_code("99a2d91ea5aed978")
How it ran

no invocation recorded.

get_tensor_in_current_ring_attn_rank Not yet run
unverified (harvested, no recorded run)
OpenRLHF/OpenRLHF/openrlhf/models/ring_attn_utils.py
code served (permissive licence) · get_code("5110d3bb195169a8")
How it ran

no invocation recorded.

reset_ring_attn_position_ids Not yet run
unverified (harvested, no recorded run)
OpenRLHF/OpenRLHF/openrlhf/models/ring_attn_utils.py
code served (permissive licence) · get_code("e93117d99aa6b210")
How it ran

no invocation recorded.

Also present in other papers' code

Of 18 samples lifted from this paper's linked repositories, 18 carry a function identity and 0 do not. Of the 18 distinct functions with an identity, 16 are also present in other papers whose code we hold (16 with at least one paper dated earlier, 0 the same month, 0 later) and 2 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
reset_ring_attn_position_ids 15 other papers
15 dated earlier
2306.00398 (2023-06)
Preference-grounded Token-level Guidance for Language Model Fine-tuning
2402.09345 (2024-02)
InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling
2404.18922 (2024-04)
DPO Meets PPO: Reinforced Token Optimization for RLHF
and 12 more not named here
is_openai_format 12 other papers
12 dated earlier
2405.14734 (2024-05)
SimPO: Simple Preference Optimization with a Reference-Free Reward
2406.08414 (2024-06)
Discovering Preference Optimization Algorithms with and for Large Language Models
2406.19371 (2024-06)
Suri: Multi-constraint Instruction Following for Long-form Text Generation
and 9 more not named here
extract_docstring 10 other papers
10 dated earlier
2405.14734 (2024-05)
SimPO: Simple Preference Optimization with a Reference-Free Reward
2406.08414 (2024-06)
Discovering Preference Optimization Algorithms with and for Large Language Models
2406.19371 (2024-06)
Suri: Multi-constraint Instruction Following for Long-form Text Generation
and 7 more not named here
normalize_whitespace 9 other papers
9 dated earlier
2405.14734 (2024-05)
SimPO: Simple Preference Optimization with a Reference-Free Reward
2406.08414 (2024-06)
Discovering Preference Optimization Algorithms with and for Large Language Models
2406.19371 (2024-06)
Suri: Multi-constraint Instruction Following for Long-form Text Generation
and 6 more not named here
log_probs_from_logits 8 other papers
8 dated earlier
2306.00398 (2023-06)
Preference-grounded Token-level Guidance for Language Model Fine-tuning
2404.18922 (2024-04)
DPO Meets PPO: Reinforced Token Optimization for RLHF
2410.20749 (2024-10)
Matryoshka Pilot: Learning to Drive Black-Box LLMs with LLMs
and 5 more not named here
log_probs_from_logits 4 other papers
4 dated earlier
2502.14768 (2025-02)
Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning
2504.07912 (2025-04)
Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
2505.15456 (2025-05)
Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment
and 1 more not named here
compute_approx_kl 3 other papers
3 dated earlier
2410.20749 (2024-10)
Matryoshka Pilot: Learning to Drive Black-Box LLMs with LLMs
2412.16145 (2024-12)
Offline Reinforcement Learning for LLM Multi-Step Reasoning
2503.10990 (2025-03)
Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium
compute_reward 3 other papers
3 dated earlier
2410.20749 (2024-10)
Matryoshka Pilot: Learning to Drive Black-Box LLMs with LLMs
2412.16145 (2024-12)
Offline Reinforcement Learning for LLM Multi-Step Reasoning
2503.10990 (2025-03)
Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium
compute_reward 2 other papers
2 dated earlier
2502.14768 (2025-02)
Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning
2603.22352 (2026-03)
WIST: Web-Grounded Iterative Self-Play Tree for Domain-Targeted Reasoning Improvement
get_tensor_in_current_ring_attn_rank 2 other papers
2 dated earlier
2502.14768 (2025-02)
Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning
2603.22352 (2026-03)
WIST: Web-Grounded Iterative Self-Play Tree for Domain-Targeted Reasoning Improvement
aggregate_loss 1 other paper
1 dated earlier
2502.14768 (2025-02)
Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning
compute_approx_kl 1 other paper
1 dated earlier
2502.14768 (2025-02)
Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning
get_loss_batch_info 1 other paper
1 dated earlier
2502.14768 (2025-02)
Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning
get_seqlens_in_batch 1 other paper
1 dated earlier
2503.10990 (2025-03)
Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium
get_unpad_data 1 other paper
1 dated earlier
2503.10990 (2025-03)
Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium
tdpo_loss
brick of this paper's assembly: entry
1 other paper
1 dated earlier
2404.11999 (2024-04)
Token-level Direct Preference Optimization

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has enabled progress on reasoningintensive tasks by relying on task-specific verifiers that provide automated correctness signals. However, many realistic language tasks are difficult to equip with reliable verifiers, motivating a growing reliance on reinforcement learning from human feedback (RLHF). In this setting, we argue that a closer examination of how human feedback should be interpreted is essential. We introduce Regret-based Preference Optimization (RePO), which reframes RLHF through regret minimization rather than reward maximization. Human preferences are often shaped by prospective anticipation of outcomes and counterfactual comparisons to alternative behaviors, rather than by immediate, outcome-independent utility. RePO captures this structure by modeling preferences as behavior-conditioned assessments of relative suboptimality. Experiments on mathematical reasoning benchmarks and human preference datasets demonstrate consistent performance gains, indicating that RePO is an effective and human-aligned approach for training large language models.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2606.09124")
get_code_for_paper("2606.09124")
have("2606.09124")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2606.09124.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2606.09124.svg)](https://syntology.ai/paper/2606.09124)

Connect an agent — have() is free.