SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2501.09993 · EMNLP · 2025

Agent-as-Judge for Factual Summarization of Long Narratives

Minsoo Kim, Seung-Won Hwang, Yeonseok Jeong, Byung-Hak Kim

arXiv · PDF · Open in the Atlas

Code that ran

10 of 13 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 0 honoured the contract we drafted, 0 violated it and 10 have no contract verdict; 0 failures of our instrument; 3 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2501.09993.json · badge: /api/ran/2501.09993.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
YeonseokJeong/NarrativeFactScore canonical (classifier label)
classifier basis not recorded on the link
10 ran · 3 no recorded run
FunctionStatusWhere it lives
break_down2scenes Ran
ran on a synthesized input
YeonseokJeong/NarrativeFactScore/src/fact/utils.py
code served (permissive licence) · get_code("27358d1c0a6c53b6")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def break_down2scenes(text: str):
    scenes = re.split('(s#\\d+)', text)
    scenes = [scene for scene in scenes if scene.strip()]
    scenes_list = []
    current_scene_number = None
    for i in range(0, len(scenes), 2):
        scene_marker = scenes[i].strip()
        try:
            scene_number = int(scene_marker.split('#')[1])
        except:
            if len(scenes) % 2 == 1:
                return [scenes[0]]
        scene_text = scenes[i + 1].strip() if i + 1 < len(scenes) else ''
        if current_scene_number is not None:
            expected_scene_number = current_scene_number + 1
            if scene_number != expected_scene_number:
                raise ValueError(f'Unexpected scene number: {scene_number}, expected {expected_scene_number}')
        scenes_list.append({'detected_scene_number': scene_number, 'text': f'{scene_marker}\n{scene_text}'.strip()})
    filtered_scene_list = []
    scene_number = 0
    for scene_dict in scenes_list:
        detected_scene_number = int(scene_dict['detected_scene_number'])
        filtered_scene_list.append(scene_dict['text'])
        scene_number = detected_scene_number
    return filtered_scene_list


# Test text with properly formatted scenes
text_with_scenes = """s#1
This is the first scene with some content and dialogue.
The characters are introduced here.
s#2
This is the second scene where the plot develops.
More action happens in this scene.
s#3
The final scene wraps up the story."""

ARGS = {
    'text': text_with_scenes
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
build_fact_prompt Ran
ran on a synthesized input
YeonseokJeong/NarrativeFactScore/src/fact/prompt.py
code served (permissive licence) · get_code("27c939504cb861df")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile

# Create a temporary prompt template file
temp_dir = tempfile.gettempdir()
prompt_template_path = os.path.join(temp_dir, "test_prompt_template.txt")
with open(prompt_template_path, 'w') as f:
    f.write("Fact: {0}\nContext: {1}")

# Define mock template functions that would be imported
def _get_hf_chat_template():
    return "[INST] {} [/INST]"

def _get_kullm_template():
    return "<s>[INST] {} [/INST]</s>"

# Inject these into the module namespace for the function to use
import sys
current_module = sys.modules[__name__]
setattr(current_module, '_get_hf_chat_template', _get_hf_chat_template)
setattr(current_module, '_get_kullm_template', _get_kullm_template)

# Construct the arguments
ARGS = {
    'prompt_template': prompt_template_path,
    'input_text_list': ["The Earth orbits the Sun", "This is a scientific fact"],
    'chat_mode': None
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
calculate_bert_score Ran
ran on a synthesized input
YeonseokJeong/NarrativeFactScore/calculate_metrics.py
code served (permissive licence) · get_code("a995b01f880808ea")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

class MockBertScorer:
    """Mock BertScore object that implements the score method."""
    
    def score(self, generated_summaries, reference_summaries):
        """
        Mock implementation of bert_scorer.score().
        Returns three tensors (precision, recall, f1) with batch size matching inputs.
        """
        batch_size = len(generated_summaries)
        # Return tensors with shape (batch_size,) as expected by the function
        p_scores = torch.tensor([0.85] * batch_size)
        r_scores = torch.tensor([0.90] * batch_size)
        f1_scores = torch.tensor([0.87] * batch_size)
        return p_scores, r_scores, f1_scores


bert_scorer = MockBertScorer()
reference_summary = "This is a reference summary about the document."
generated_summary = "This is a generated summary about the document."

ARGS = {
    "bert_scorer": bert_scorer,
    "reference_summary": reference_summary,
    "generated_summary": generated_summary,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
calculate_rouge Ran
ran on a synthesized input
YeonseokJeong/NarrativeFactScore/calculate_metrics.py
code served (permissive licence) · get_code("1ac357bf2eed797a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

from rouge_score import rouge_scorer

# The function takes two string parameters: reference_summary and generated_summary
# These are used with rouge_scorer.RougeScorer.score() which expects strings

ARGS = {
    "reference_summary": "The quick brown fox jumps over the lazy dog. This is a reference summary.",
    "generated_summary": "A quick brown fox jumped over a lazy dog. This is a generated summary."
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
rouge_score — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
delete_special Ran
ran on a synthesized input
YeonseokJeong/NarrativeFactScore/src/fact/utils.py
code served (permissive licence) · get_code("2a13a9eae8815574")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Construct arguments for delete_special(pre_text, character_list)
pre_text = "Hello, World! This is a test."
character_list = [',', '!', '.']

ARGS = {
    'pre_text': pre_text,
    'character_list': character_list
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
generate_names_graph Ran
ran on a synthesized input
YeonseokJeong/NarrativeFactScore/src/kg/knowledge_graph.py
code served (permissive licence) · get_code("f50f1c74536e69c4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Graph. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import networkx as nx

# The function takes a 'names' parameter which should be an iterable of name groups
# Each name group is an iterable of names (strings)
# The function uses combinations() to create pairs within each group
# and adds nodes and edges to a networkx.Graph()

names = [
    ["Alice", "A", "Alice Smith"],
    ["Bob", "Robert", "Bob Jones"],
    ["Charlie"]
]

ARGS = {
    "names": names
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
networkx — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Graph — a Graph output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a Graph output the recorded ndim 1 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
gpt_inference Ran
ran on a synthesized input
YeonseokJeong/NarrativeFactScore/src/kg/generate_kg.py
code served (permissive licence) · get_code("00ab0223fa638a2f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import openai
import time

# Mock the openai module to avoid actual API calls
class MockResponse:
    def __getitem__(self, key):
        if key == 'choices':
            return [{'message': {'content': 'Mock response from GPT'}}]
        return None

class MockChatCompletion:
    @staticmethod
    def create(**kwargs):
        return MockResponse()

class MockOpenAI:
    api_key = None
    error = type('error', (), {'APIError': Exception})()
    ChatCompletion = MockChatCompletion()

# Set up mock openai module
openai.api_key = None
openai.ChatCompletion = MockChatCompletion()
openai.error = type('error', (), {'APIError': Exception})()

# Set environment variable for API key (not actually used in mock)
os.environ['OPENAI_API_KEY'] = 'mock-key-12345'

# Define arguments for gpt_inference
ARGS = {
    'system_instruction': 'You are a helpful assistant.',
    'prompt': 'What is 2+2?',
    'retries': 3,
    'delay': 1
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
openai — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_knowledge_graph Ran
ran on a synthesized input
YeonseokJeong/NarrativeFactScore/src/kg/generate_kg.py
code served (permissive licence) · get_code("c8df89ad35127fe7")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pickle
import tempfile
from pathlib import Path

# Create a temporary directory and a valid pickle file
temp_dir = tempfile.mkdtemp()
save_path = Path(temp_dir)

# Create a simple knowledge graph object (can be a dict or any picklable object)
knowledge_graph = {
    'nodes': ['node1', 'node2', 'node3'],
    'edges': [('node1', 'node2'), ('node2', 'node3')],
    'properties': {'name': 'test_kg', 'version': 1}
}

# Save it to the expected location
kg_file_path = save_path / 'kg.pkl'
with open(kg_file_path, 'wb') as f:
    pickle.dump(knowledge_graph, f)

# Define arguments for the function
ARGS = {
    'project_gutenberg_id': 12345,
    'save_path': save_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
remove_number_prefix Ran
ran on a synthesized input
YeonseokJeong/NarrativeFactScore/src/kg/knowledge_graph.py
code served (permissive licence) · get_code("8936fbeddd8acbb0")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def remove_number_prefix(text):
    clean_text = re.sub(r'^\d+\.\s*', '', text)
    return clean_text

ARGS = {
    'text': '1. This is a sample text'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
split_and_clean_script Ran
ran on a synthesized input
YeonseokJeong/NarrativeFactScore/down_data.py
code served (permissive licence) · get_code("69d99f943fa8a011")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def split_and_clean_script(script):
    scenes = re.split('<scene>', script)
    cleaned_scenes = []
    for scene in scenes:
        cleaned_scene = re.sub('<.*?>', '', scene)
        cleaned_scene = cleaned_scene.strip()
        if cleaned_scene:
            cleaned_scenes.append(cleaned_scene)
    return cleaned_scenes

ARGS = {
    'script': '<scene>  <title>Act One</title>\n  The hero enters the stage.\n</scene><scene>  <action>He looks around.</action>\n  The room is empty.\n</scene><scene>  <dialogue>Hello?</dialogue>\n</scene>'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
calculate_bleu Not yet run
unverified (harvested, no recorded run)
YeonseokJeong/NarrativeFactScore/calculate_metrics.py
code served (permissive licence) · get_code("50254ec8b6e5f2d1")
How it ran

no invocation recorded.

generate_knowledge_graph_for_scripts Not yet run
unverified (harvested, no recorded run)
YeonseokJeong/NarrativeFactScore/src/kg/generate_kg.py
code served (permissive licence) · get_code("d24be1d4cdb47e5f")
How it ran

no invocation recorded.

parse_response_text Not yet run
unverified (harvested, no recorded run)
YeonseokJeong/NarrativeFactScore/src/kg/knowledge_graph.py
code served (permissive licence) · get_code("4308a1d6439f3b09")
How it ran

no invocation recorded.

Also present in other papers' code

Of 13 samples lifted from this paper's linked repositories, 13 carry a function identity and 0 do not. Of the 13 distinct functions with an identity, 0 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 0 later) and 13 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Large Language Models (LLMs) have demonstrated near-human performance in summarization tasks based on traditional metrics such as ROUGE and BERTScore. However, these metrics do not adequately capture critical aspects of summarization quality, such as factual accuracy, particularly for long narratives (>100K tokens). Recent advances, such as LLM-as-a-Judge, address the limitations of metrics based on lexical similarity but still exhibit factual inconsistencies, especially in understanding character relationships and states. In this work, we introduce NARRATIVEFACTSCORE (NFS), the first "Agent-as-a-Judge" framework that evaluates and refines factuality in narrative summarization. By leveraging a Character Knowledge Graph (CKG) extracted from input narrative, NARRATIVEFACTSCORE evaluates the factuality and provides actionable guidance for refinement, such as identifying missing or erroneous facts. Our experimental results demonstrate that constructing the CKG enables reasoning with 1/3 of the factuality computation used in the prior approach, and achieve three times higher correlation with human judgments. Furthermore, refinement with actionable guidance improves the quality of the summary. 1 * Corresponding Author 1 https://github.com/YeonseokJeong/NarrativeFactScore # 14. BAG END LIVING ROOM … Bilbo: It's mine, my own. my precious (Frodo rushes into Bag End. He stops and picks up the ring at his feet.) … # 25. BAG END KITCHEN … Gandalf: Sauron needs only this ring to cover all the lands in the second darkness. He is seeking it, seeking it, all his thought is bent on it. … Frodo: Alright! ... Gandalf warned Frodo, who carries the Ring, that its master is Sauron. Sauron is searching for the Ring and is pursuing Gandalf.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2501.09993")
get_code_for_paper("2501.09993")
have("2501.09993")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2501.09993.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2501.09993.svg)](https://syntology.ai/paper/2501.09993)

Connect an agent — have() is free.