SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2305.15852 · 2023

Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation

arXiv · PDF · Open in the Atlas

Code that ran

5 of 9 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 0 honoured the contract we drafted, 0 violated it and 5 have no contract verdict; 0 failures of our instrument; 4 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2305.15852.json · badge: /api/ran/2305.15852.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
eth-sri/chatprotect canonical (classifier label)
classifier basis: census
5 ran · 4 no recorded run
FunctionStatusWhere it lives
question_generation_sampling Ran
ran on a synthesized input
eth-sri/chatprotect/chatprotect/selfcheckgpt/modeling_mqag.py
code served (permissive licence) · get_code("2ff8cbbaf8e530db")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Stand-in tokenizer class with required methods and attributes
class StandInTokenizer:
    def __init__(self):
        self.pad_token = "[PAD]"
        self.eos_token = "[EOS]"
        self.sep_token = "[SEP]"
    
    def encode(self, text, return_tensors=None):
        # Simple mock encoding: return small tensor
        token_ids = [101, 2054, 2003, 102]  # Mock token IDs
        if return_tensors == "pt":
            return torch.tensor([token_ids], dtype=torch.long)
        return token_ids
    
    def decode(self, token_ids, skip_special_tokens=False):
        # Mock decoding: return a string with sep token
        if isinstance(token_ids, torch.Tensor):
            token_ids = token_ids.tolist()
        if isinstance(token_ids, list) and len(token_ids) > 0 and isinstance(token_ids[0], list):
            token_ids = token_ids[0]
        return "What is AI? [SEP] Artificial Intelligence [SEP] Machine Learning [SEP] Deep Learning"

# Stand-in model class with generate method
class StandInModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 4)
    
    def generate(self, input_ids, max_new_tokens=128, do_sample=False):
        # Mock generation: return tensor of token IDs
        batch_size = input_ids.shape[0]
        # Return mock token sequence
        return torch.randint(0, 1000, (batch_size, 20), dtype=torch.long)

# Mock prepare_qa_input function
def prepare_qa_input(tokenizer, context, device):
    # Mock implementation
    return torch.randint(0, 100, (1, 8), dtype=torch.long).to(device)

# Mock prepare_distractor_input function
def prepare_distractor_input(tokenizer, context, question, answer, device, separator):
    # Mock implementation
    return torch.randint(0, 100, (1, 8), dtype=torch.long).to(device)

# Inject mocked functions into the module namespace where they'll be called
import sys
current_module = sys.modules[__name__]
setattr(current_module, 'prepare_qa_input', prepare_qa_input)
setattr(current_module, 'prepare_distractor_input', prepare_distractor_input)

# Construct ARGS dictionary
ARGS = {
    "g1_model": StandInModel(),
    "g1_tokenizer": StandInTokenizer(),
    "g2_model": StandInModel(),
    "g2_tokenizer": StandInTokenizer(),
    "context": "What is artificial intelligence and how does it work?",
    "num_questions": 2,
    "device": torch.device("cpu"),
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
read_entities Ran
ran on a synthesized input
eth-sri/chatprotect/chatprotect/util.py
code served (permissive licence) · get_code("10de4d9e39ba3715")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pathlib
import tempfile
import os

# Create a temporary file with test entity data
with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.txt') as f:
    temp_path = f.name
    f.write("[Section Header]\n")
    f.write("# Comment line\n")
    f.write("entity1\n")
    f.write("\n")
    f.write("entity2\n")
    f.write("entity3\n")

try:
    ARGS = {
        'path': temp_path
    }
finally:
    # Clean up will happen after the function is called
    pass
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
read_sent_file Ran
ran on a synthesized input
eth-sri/chatprotect/chatprotect/util.py
code served (permissive licence) · get_code("c260837e886b1746")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ExtSentence. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import pathlib
import tempfile
from dataclasses import dataclass
import typing

@dataclass(unsafe_hash=True)
class ExtSentence:
    target: str
    orig: str
    alt: str
    prefix: str
    triple: typing.Tuple[str, str, str]
    tag: typing.Union[typing.Literal['ok'], typing.Literal['weak'], typing.Literal['strong']]
    wrong: str = 'unk'
    orig_tag: typing.Optional[typing.Union[typing.Literal['ok'], typing.Literal['weak'], typing.Literal['strong']]] = None
    explanation: typing.Optional[str] = None

def read_sent_file(target: typing.Optional[str], path: typing.Union[pathlib.Path, str]):
    if '_cost' in str(path):
        raise ValueError('trying to read cost file')
    with pathlib.Path(path).open('r') as f:
        d = json.load(f)
    ext = ExtSentence(**d)
    if target is not None and ext.target != target:
        raise ValueError('Invalid sents file for given target')
    return ext

# Create a temporary JSON file with valid ExtSentence data
temp_dir = tempfile.TemporaryDirectory()
temp_path = pathlib.Path(temp_dir.name) / "test_sent.json"

sentence_data = {
    "target": "test_target",
    "orig": "The original sentence",
    "alt": "The alternative sentence",
    "prefix": "prefix_text",
    "triple": ["subject", "predicate", "object"],
    "tag": "ok",
    "wrong": "unk",
    "orig_tag": "weak",
    "explanation": "This is an explanation"
}

with open(temp_path, 'w') as f:
    json.dump(sentence_data, f)

ARGS = {
    "target": "test_target",
    "path": temp_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ExtSentence — a ExtSentence output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a ExtSentence output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
rephrase_sent_remove_conflict Ran
ran on a synthesized input
eth-sri/chatprotect/chatprotect/rephrase.py
code served (permissive licence) · get_code("20bd07858a5fc478")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

class MockBot:
    """Mock bot object that implements context manager and question-answering"""
    
    def __init__(self):
        self.num_answers = 1
        self.deterministic = False
        self.session = None
    
    def __enter__(self):
        self.session = self
        return self
    
    def __exit__(self, exc_type, exc_val, exc_tb):
        self.session = None
        return False
    
    def set_num_answers(self, n):
        self.num_answers = n
    
    def set_deterministic(self, val):
        self.deterministic = val
    
    def ask(self, prompt):
        # Return a simple rephrased sentence with conflicting info removed
        return "The product is durable."

# Construct arguments
bot = MockBot()
sent = "The product is fragile and breaks easily."
alt_sent = "The product is very durable and lasts for years."
target = "What are the product qualities?"
prefix = "This is an excellent product that performs well."

ARGS = {
    "bot": bot,
    "sent": sent,
    "alt_sent": alt_sent,
    "target": target,
    "prefix": prefix
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
to_hour Ran
ran on a synthesized input
eth-sri/chatprotect/chatprotect/bots/openai_chat_stream_api.py
code served (permissive licence) · get_code("20c4db5099969535")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import datetime

dt = datetime.datetime(2024, 1, 15, 10, 30, 45)

ARGS = {
    "dt": dt
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
llama_v2_prompt Not yet run
unverified (harvested, no recorded run)
eth-sri/chatprotect/chatprotect/bots/together_ai_api.py
code served (permissive licence) · get_code("07b62c885f7ea17e")
How it ran

no invocation recorded.

question_generation_beamsearch Not yet run
unverified (harvested, no recorded run)
eth-sri/chatprotect/chatprotect/selfcheckgpt/modeling_mqag.py
code served (permissive licence) · get_code("ec7bbc872d252c0f")
How it ran

no invocation recorded.

question_generation_sentence_level Not yet run
unverified (harvested, no recorded run)
eth-sri/chatprotect/chatprotect/selfcheckgpt/modeling_mqag.py
code served (permissive licence) · get_code("d0dc64dd7b3c3112")
How it ran

no invocation recorded.

split_sentences Not yet run
unverified (harvested, no recorded run)
eth-sri/chatprotect/chatprotect/util.py
code served (permissive licence) · get_code("00998fc8648726d1")
How it ran

no invocation recorded.

Also present in other papers' code

Of 9 samples lifted from this paper's linked repositories, 9 carry a function identity and 0 do not. Of the 9 distinct functions with an identity, 1 are also present in other papers whose code we hold (1 with at least one paper dated earlier, 0 the same month, 0 later) and 8 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
question_generation_sentence_level 1 other paper
1 dated earlier
2303.08896 (2023-03)
SELFCHECKGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Large language models (large LMs) are susceptible to producing text that contains hallucinated content. An important instance of this problem is self-contradiction, where the LM generates two contradictory sentences within the same context. In this work, we present a comprehensive investigation into self-contradiction for various instruction-tuned LMs, covering evaluation, detection, and mitigation. Our primary evaluation task is open-domain text generation, but we also demonstrate the applicability of our approach to shorter question answering. Our analysis reveals the prevalence of self-contradictions, e.g., in 17.7% of all sentences produced by ChatGPT. We then propose a novel prompting-based framework designed to effectively detect and mitigate self-contradictions. Our detector achieves high accuracy, e.g., around 80% F1 score when prompting ChatGPT. The mitigation algorithm iteratively refines the generated text to remove contradictory information while preserving text fluency and informativeness. Importantly, our entire framework is applicable to black-box LMs and does not require retrieval of external knowledge. Rather, our method complements retrieval-based methods, as a large portion of self-contradictions (e.g., 35.2% for ChatGPT) cannot be verified using online text. Our approach is practically effective and has been released as a push-button tool to benefit the public at https://chatprotect.ai/.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2305.15852")
get_code_for_paper("2305.15852")
have("2305.15852")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2305.15852.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2305.15852.svg)](https://syntology.ai/paper/2305.15852)

Connect an agent — have() is free.