SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2501.16629 · ICLR · 2025

CHiP: CROSS-MODAL HIERARCHICAL DIRECT PREF-ERENCE OPTIMIZATION FOR MULTIMODAL LLMS

Xiaoyu Shen, Bryan Hooi, Xipeng Qiu, Hao Fei, See-Kiong Ng, Jinlan Fu, Shenzhen Huangfu

arXiv · PDF · Open in the Atlas

Code that ran

5 of 6 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 1 honoured the contract we drafted, 0 violated it and 0 have no contract verdict; 4 failures of our instrument; 1 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2501.16629.json · badge: /api/ran/2501.16629.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
lvugai/chip canonical (classifier label)
classifier basis: classifier_2025
5 ran · 1 no recorded run
FunctionStatusWhere it lives
combine_coco_captions Ran
ran; our contract draft was wrong, not the code
lvugai/chip/eval/eval_gpt_obj_halbench.py
code served (permissive licence) · get_code("25585704d77cafb4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import json
import tempfile
import shutil

# Create a temporary directory structure with mock COCO caption files
temp_dir = tempfile.mkdtemp()

# Create mock val captions file
val_captions = {
    'info': {'description': 'COCO 2014 Dataset', 'version': '1.0'},
    'licenses': [{'id': 1, 'name': 'Attribution License'}],
    'images': [
        {'id': 1, 'file_name': 'val_image_1.jpg'},
        {'id': 2, 'file_name': 'val_image_2.jpg'}
    ],
    'annotations': [
        {'id': 1, 'image_id': 1, 'caption': 'A cat on a mat'},
        {'id': 2, 'image_id': 2, 'caption': 'A dog playing'}
    ]
}

# Create mock train captions file
train_captions = {
    'info': {'description': 'COCO 2014 Dataset', 'version': '1.0'},
    'licenses': [{'id': 1, 'name': 'Attribution License'}],
    'images': [
        {'id': 3, 'file_name': 'train_image_1.jpg'},
        {'id': 4, 'file_name': 'train_image_2.jpg'}
    ],
    'annotations': [
        {'id': 3, 'image_id': 3, 'caption': 'A bird flying'},
        {'id': 4, 'image_id': 4, 'caption': 'A fish swimming'}
    ]
}

# Write the mock files
with open(os.path.join(temp_dir, 'captions_val2014.json'), 'w') as f:
    json.dump(val_captions, f)

with open(os.path.join(temp_dir, 'captions_train2014.json'), 'w') as f:
    json.dump(train_captions, f)

ARGS = {
    'annotation_path': temp_dir
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
expand_question_into_multimodal Ran
ran; our contract draft was wrong, not the code
lvugai/chip/muffin/muffin/eval/inference.py
code served (permissive licence) · get_code("adb407bf7ff1f083")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

ARGS = {
    "question_text": "What is in the <image>?",
    "image_token_len": 5,
    "im_st_token": "[IMG_START]",
    "im_ed_token": "[IMG_END]",
    "im_patch_token": "[PATCH]"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
parse_object_list Ran
ran; our contract draft was wrong, not the code
lvugai/chip/eval/eval_gpt_obj_halbench.py
code served (permissive licence) · get_code("887addb3dd50b07e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json

def parse_object_list(content):
    try:
        content = json.loads(content)
    except:
        if '```json' in content:
            try:
                content = json.loads(content.replace('```json', '').replace('```', '').strip())
            except:
                raise ValueError('Content is not json interpretable')
        elif '["' in content:
            try:
                content = json.loads(content.strip().split('\n')[-1])
            except:
                raise ValueError('Content is not json interpretable')
        else:
            raise ValueError('Content is not json interpretable')
    return content


# Test case 1: Valid JSON string
content_valid_json = '{"key": "value", "number": 42}'

# Test case 2: JSON wrapped in markdown code block
content_markdown_json = '```json\n{"items": [1, 2, 3]}\n```'

# Test case 3: JSON as last line with markdown wrapper
content_last_line_json = 'Some text\nMore text\n["item1", "item2"]'

ARGS = {
    'content': content_valid_json
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
preprocess_coh_results Ran
ran; our contract draft was wrong, not the code
lvugai/chip/eval/eval_gpt_obj_halbench.py
code served (permissive licence) · get_code("f7745bc5cafc18b6")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Construct test data for preprocess_coh_results
# The function expects a list of dicts, each with a 'caption' key

caps = [
    {
        'caption': 'This is a test. The following is a response without hallucination. This is the actual response.',
        'image_id': 1
    },
    {
        'caption': 'This is the response with hallucination. The following is a response with hallucination. Extra text.',
        'image_id': 2
    },
    {
        'caption': 'Some response here. Generate a response without errors. The cleaned response.',
        'image_id': 3
    },
    {
        'caption': 'The cleaned response. Generate a response with errors. Some error text.',
        'image_id': 4
    },
    {
        'caption': 'Just a normal caption with no special markers.',
        'image_id': 5
    }
]

ARGS = {
    'caps': caps
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
torch_pad_sequence Ran
ran, honoured the contract we drafted
lvugai/chip/muffin/muffin/eval/inference.py
code served (permissive licence) · get_code("86450c9db0fb7ad9")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned Tensor, ndim 3, dtype family float, shape (3, 7, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Create a sequence of tensors with different lengths
# This is what pad_sequence expects - a list of 1D or 2D tensors
sequence = [
    torch.randn(5, 4),  # First tensor: length 5, feature dim 4
    torch.randn(3, 4),  # Second tensor: length 3, feature dim 4
    torch.randn(7, 4),  # Third tensor: length 7, feature dim 4
]

ARGS = {
    'sequence': sequence,
    'padding_value': 0.0,
    'batch_first': True,
    'padding_side': 'right'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 3, dtype family float, shape (3, 7, 4) — shape (3, 7, 4) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
chip_get_batch_logps Not yet run
unverified (harvested, no recorded run)
lvugai/chip/llava/train/llava_trainer.py
code served (permissive licence) · get_code("6c5838ce1b68d726")
How it ran

no invocation recorded.

Also present in other papers' code

Of 6 samples lifted from this paper's linked repositories, 6 carry a function identity and 0 do not. Of the 6 distinct functions with an identity, 4 are also present in other papers whose code we hold (4 with at least one paper dated earlier, 0 the same month, 1 later) and 2 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
combine_coco_captions 10 other papers
6 dated earlier, 3 later, 1 same year, order not determinable
2110.01705 (2021-10)
Let there be a clock on the beach: Reducing Object Hallucination in Image Captioning
Huang_OPERA_Alleviating_Hallucination_in_Multi-Modal_Large_Language_Models_via_Over-Trust_CVPR_2024_paper (2024 (year only))
OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
2402.14545 (2024-02)
Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspective
and 7 more not named here
expand_question_into_multimodal 1 other paper
1 dated earlier
2310.00653 (2023-10)
Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants
preprocess_coh_results 1 other paper
1 dated earlier
2405.17220 (2024-05)
RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness
torch_pad_sequence 1 other paper
1 dated earlier
2310.00653 (2023-10)
Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Multimodal Large Language Models (MLLMs) still struggle with hallucinations despite their impressive capabilities. Recent studies have attempted to mitigate this by applying Direct Preference Optimization (DPO) to multimodal scenarios using preference pairs from text-based responses. However, our analysis of representation distributions reveals that multimodal DPO struggles to align image and text representations and to distinguish between hallucinated and non-hallucinated descriptions. To address these challenges, In this work, we propose a Cross-modal Hierarchical Direct Preference Optimization (CHiP) to address these limitations. We introduce a visual preference optimization module within the DPO framework, enabling MLLMs to learn from both textual and visual preferences simultaneously. Furthermore, we propose a hierarchical textual preference optimization module that allows the model to capture preferences at multiple granular levels, including response, segment, and token levels. We evaluate CHiP through both quantitative and qualitative analyses, with results across multiple benchmarks demonstrating its effectiveness in reducing hallucinations. On the Object Hal-Bench dataset, CHiP outperforms DPO in hallucination reduction, achieving improvements of 52.7% and 55.5% relative points based on the base model Muffin and LLaVA models, respectively. We make all our datasets and code publicly available. 1

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2501.16629")
get_code_for_paper("2501.16629")
have("2501.16629")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2501.16629.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2501.16629.svg)](https://syntology.ai/paper/2501.16629)

Connect an agent — have() is free.