SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2403.10446 · 2024

Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Bases

arXiv · PDF · Open in the Atlas

Code that ran

12 of 13 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 1 honoured the contract we drafted, 0 violated it and 0 have no contract verdict; 11 failures of our instrument; 1 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2403.10446.json · badge: /api/ran/2403.10446.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
anlp-team/LTI_Neural_Navigator canonical (classifier label)
classifier basis: census
12 ran · 1 no recorded run
FunctionStatusWhere it lives
filter Ran
ran, honoured the contract we drafted
anlp-team/LTI_Neural_Navigator/src/data_prep/get_paper.py
code served (permissive licence) · get_code("16f4131c96c45844")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import os
import tempfile
import shutil

# Create temporary directories for testing
temp_dir = tempfile.mkdtemp()
data_dir = os.path.join(temp_dir, "data.json")
save_dir = os.path.join(temp_dir, "output")

# Create sample JSON data matching the structure expected by the filter function
sample_data = [
    {
        "name": "Professor_Smith",
        "hIndex": 45,
        "citationCount": 5000,
        "paperCount": 50,
        "papers": [
            {
                "title": "Machine_Learning_Paper_1",
                "year": 2023,
                "isOpenAccess": True,
                "abstract": "This is an abstract about machine learning.",
                "authors": [
                    {"name": "John Doe"},
                    {"name": "Jane Smith"}
                ],
                "publicationVenue": {
                    "name": "Conference on Machine Learning",
                    "alternate_names": ["CML", "ML Conference"]
                },
                "openAccessPdf": None
            },
            {
                "title": "Data_Science_Paper",
                "year": 2022,
                "isOpenAccess": True,
                "abstract": "This is about data science.",
                "authors": [
                    {"name": "Alice Johnson"}
                ],
                "publicationVenue": {
                    "name": "Data Science Conference",
                    "alternate_names": None
                },
                "openAccessPdf": None
            }
        ]
    },
    {
        "name": "Professor_Johnson",
        "hIndex": 38,
        "citationCount": 3200,
        "paperCount": 35,
        "papers": [
            {
                "title": "AI_Research_Paper",
                "year": 2023,
                "isOpenAccess": True,
                "abstract": "Research in artificial intelligence.",
                "authors": [
                    {"name": "Bob Wilson"}
                ],
                "publicationVenue": "Journal of AI",
                "openAccessPdf": None
            }
        ]
    }
]

# Write the sample data to a JSON file
with open(data_dir, 'w') as f:
    json.dump(sample_data, f)

# Ensure save directory exists
if not os.path.exists(save_dir):
    os.makedirs(save_dir)

ARGS = {
    "data_dir": data_dir,
    "save_dir": save_dir
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
requests — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
find_pdf_files Ran
ran; our contract draft was wrong, not the code
anlp-team/LTI_Neural_Navigator/src/data_prep/get_paper.py
code served (permissive licence) · get_code("6b05e4043da9fa41")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile
import shutil

# Create a temporary directory structure with some PDF files
temp_dir = tempfile.mkdtemp()

# Create some subdirectories
subdir1 = os.path.join(temp_dir, "subdir1")
subdir2 = os.path.join(temp_dir, "subdir2")
os.makedirs(subdir1, exist_ok=True)
os.makedirs(subdir2, exist_ok=True)

# Create some dummy PDF files
pdf_file1 = os.path.join(temp_dir, "document1.pdf")
pdf_file2 = os.path.join(subdir1, "document2.pdf")
pdf_file3 = os.path.join(subdir2, "document3.PDF")
txt_file = os.path.join(temp_dir, "readme.txt")

open(pdf_file1, 'w').close()
open(pdf_file2, 'w').close()
open(pdf_file3, 'w').close()
open(txt_file, 'w').close()

ARGS = {
    "directory": temp_dir
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
fix_json_string Ran
ran; our contract draft was wrong, not the code
anlp-team/LTI_Neural_Navigator/src/annotation/fix_json_str.py
code served (permissive licence) · get_code("6c210b1b9a24dc88")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import re

def fix_json_string(json_str):
    json_str = json_str.strip()
    json_str = re.sub(',\\s*]', ']', json_str)
    fixed_json_str = re.sub(',\\s*}', '}', json_str)
    fixed_json_str = fixed_json_str.strip()
    if not fixed_json_str.startswith('['):
        fixed_json_str = '[' + fixed_json_str
    if not fixed_json_str.endswith(']'):
        fixed_json_str += ']'
    try:
        json_obj = json.loads(fixed_json_str)
        return json.dumps(json_obj)
    except json.JSONDecodeError as e:
        print(f'Error fixing JSON: {e}')
        try:
            last_valid_index = fixed_json_str.rfind('}')
            valid_json_part = fixed_json_str[:last_valid_index + 1] + ']'
            json_obj = json.loads(valid_json_part)
            return json.dumps(json_obj)
        except json.JSONDecodeError as e:
            print(f'Failed to fix JSON: {e}')
            return None


# Test case 1: Valid JSON array with trailing commas
json_str_1 = '{"name": "test", "value": 42,}'

# Test case 2: Valid JSON object without array wrapper
json_str_2 = '{"id": 1, "data": [1, 2, 3,]}'

# Test case 3: Partially malformed JSON
json_str_3 = '{"key": "value",'

ARGS = {
    "json_str": json_str_1
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_dst_filename Ran
ran; our contract draft was wrong, not the code
anlp-team/LTI_Neural_Navigator/src/annotation/anno_reference.py
code served (permissive licence) · get_code("09838de41af0c00e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os

# The function get_dst_filename takes three string parameters representing file paths
# It uses os.path.relpath and os.path.join which operate on path strings
# We need to provide valid path strings for src_file, src_path, and dst_path

ARGS = {
    "src_file": "/home/user/project/src/module/file.py",
    "src_path": "/home/user/project/src",
    "dst_path": "/home/user/project/dst"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_incremental_list Ran
ran; our contract draft was wrong, not the code
anlp-team/LTI_Neural_Navigator/src/annotation/anno_reference.py
code served (permissive licence) · get_code("32d7d414e146c489")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile
import shutil

# Create temporary directories for testing
temp_src = tempfile.mkdtemp()
temp_dst = tempfile.mkdtemp()

# Create some test .json files in src_path
os.makedirs(os.path.join(temp_src, "subdir1"), exist_ok=True)
os.makedirs(os.path.join(temp_src, "subdir2"), exist_ok=True)

with open(os.path.join(temp_src, "file1.json"), "w") as f:
    f.write("{}")
with open(os.path.join(temp_src, "subdir1", "file2.json"), "w") as f:
    f.write("{}")
with open(os.path.join(temp_src, "subdir2", "file3.json"), "w") as f:
    f.write("{}")
with open(os.path.join(temp_src, "other.txt"), "w") as f:
    f.write("not json")

# Create some processed files in dst_path (subset of src files)
os.makedirs(os.path.join(temp_dst, "subdir1"), exist_ok=True)
with open(os.path.join(temp_dst, "file1.json"), "w") as f:
    f.write("{}")
with open(os.path.join(temp_dst, "subdir1", "file2.json"), "w") as f:
    f.write("{}")

# Create file_list with all source .json files
file_list = [
    os.path.join(temp_src, "file1.json"),
    os.path.join(temp_src, "subdir1", "file2.json"),
    os.path.join(temp_src, "subdir2", "file3.json"),
]

# Create args object
class Args:
    def __init__(self, src_path, dst_path):
        self.src_path = src_path
        self.dst_path = dst_path

args = Args(src_path=temp_src, dst_path=temp_dst)

ARGS = {
    "file_list": file_list,
    "args": args,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_incremental_list Ran
ran; our contract draft was wrong, not the code
anlp-team/LTI_Neural_Navigator/src/annotation/plan_annotation.py
code served (permissive licence) · get_code("f8e9d133543a2771")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile
from argparse import Namespace

# Create temporary directories for testing
with tempfile.TemporaryDirectory() as temp_dir:
    input_dir = os.path.join(temp_dir, 'input')
    output_dir = os.path.join(temp_dir, 'output')
    
    # Create directories
    os.makedirs(input_dir, exist_ok=True)
    os.makedirs(output_dir, exist_ok=True)
    
    # Create some sample input files
    sample_input_files = [
        os.path.join(input_dir, 'file1.txt'),
        os.path.join(input_dir, 'file2.txt'),
        os.path.join(input_dir, 'file3.txt'),
    ]
    for file_path in sample_input_files:
        with open(file_path, 'w') as f:
            f.write('sample content')
    
    # Create some processed output files (as .json files)
    processed_files = [
        os.path.join(output_dir, 'file1.json'),
        os.path.join(output_dir, 'file2.json'),
    ]
    for file_path in processed_files:
        with open(file_path, 'w') as f:
            f.write('processed content')
    
    # Create file_list with all input files
    file_list = sample_input_files
    
    # Create args object with in_dir and out_dir attributes
    args = Namespace(
        in_dir=input_dir,
        out_dir=output_dir
    )
    
    ARGS = {
        'file_list': file_list,
        'args': args
    }
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_msg_func Ran
ran; our contract draft was wrong, not the code
anlp-team/LTI_Neural_Navigator/src/annotation/local_auto.py
code served (permissive licence) · get_code("a1bec14819bec9eb")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

class Message:
    def __init__(self, content):
        self.content = content

ARGS = {
    'data': {
        'messages': [
            Message('first message'),
            Message('second message'),
            Message('final message content')
        ]
    }
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
json_repair_api Ran
ran; our contract draft was wrong, not the code
anlp-team/LTI_Neural_Navigator/src/annotation/fix_json_str.py
code served (permissive licence) · get_code("786dc72edd4e7742")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

from json_repair import repair_json

# Analyzing the function:
# json_repair_api takes json_str parameter
# It calls normalize_dirty_str(json_str) which expects a string
# The string should either:
#   1. Contain '```' with json content inside
#   2. Start with 'text=' format
# Then it calls repair_json() on the normalized string

# Test cases for json_str parameter:
# Option 1: markdown code block format
test_json_1 = """```json
{"name": "test", "value": 123}
"""

# Option 2: text= format  
test_json_2 = """text='{"name": "test", "value": 123}'"""

# Option 3: markdown with extra whitespace
test_json_3 = """```json
{
  "key": "value",
  "incomplete": 
}
"""

# Use the first valid test case
ARGS = {
    "json_str": test_json_1
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
json_repair — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
list_all_files Ran
ran; our contract draft was wrong, not the code
anlp-team/LTI_Neural_Navigator/src/annotation/anno_reference.py
code served (permissive licence) · get_code("43ff90719e493366")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile
import shutil

# Create a temporary directory structure with some .json files and other files
temp_dir = tempfile.mkdtemp()

# Create subdirectories
os.makedirs(os.path.join(temp_dir, "subdir1"), exist_ok=True)
os.makedirs(os.path.join(temp_dir, "subdir2"), exist_ok=True)

# Create some .json files
with open(os.path.join(temp_dir, "file1.json"), "w") as f:
    f.write("{}")
with open(os.path.join(temp_dir, "subdir1", "file2.json"), "w") as f:
    f.write("{}")
with open(os.path.join(temp_dir, "subdir2", "file3.json"), "w") as f:
    f.write("{}")

# Create some non-.json files (these should be ignored)
with open(os.path.join(temp_dir, "file.txt"), "w") as f:
    f.write("text")
with open(os.path.join(temp_dir, "subdir1", "file.csv"), "w") as f:
    f.write("data")

ARGS = {
    "root_path": temp_dir
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
list_all_files Ran
ran; our contract draft was wrong, not the code
anlp-team/LTI_Neural_Navigator/src/annotation/local_auto.py
code served (permissive licence) · get_code("8976b470bd9dbdb3")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile
import shutil

# Create a temporary directory structure for testing
temp_dir = tempfile.mkdtemp()

# Create some nested directories and files
os.makedirs(os.path.join(temp_dir, "subdir1"), exist_ok=True)
os.makedirs(os.path.join(temp_dir, "subdir2"), exist_ok=True)
os.makedirs(os.path.join(temp_dir, "subdir1", "nested"), exist_ok=True)

# Create some test files
open(os.path.join(temp_dir, "file1.txt"), "w").close()
open(os.path.join(temp_dir, "file2.txt"), "w").close()
open(os.path.join(temp_dir, "subdir1", "file3.txt"), "w").close()
open(os.path.join(temp_dir, "subdir2", "file4.txt"), "w").close()
open(os.path.join(temp_dir, "subdir1", "nested", "file5.txt"), "w").close()

ARGS = {
    "root_path": temp_dir
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
normalize_dirty_str Ran
ran; our contract draft was wrong, not the code
anlp-team/LTI_Neural_Navigator/src/annotation/fix_json_str.py
code served (permissive licence) · get_code("58068d0689548837")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

# Analyze the function: normalize_dirty_str takes a single parameter 'dirty_str' of type str
# The function processes strings containing code blocks (marked by ```) or text (prefixed with 'text=')
# It extracts and cleans the content

# Test case 1: String with triple backticks and json content
dirty_str_1 = "```json\n{'key': 'value'}\n```"

# Test case 2: String with text= prefix
dirty_str_2 = "text='hello\\nworld\\'s test'"

# Test case 3: String with triple backticks without json prefix
dirty_str_3 = "```\nsome code\n```"

# We'll use the first test case as it covers the main code path
ARGS = {
    "dirty_str": "```json\n{'key': 'value'}\n```"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
split_file Ran
ran; our contract draft was wrong, not the code
anlp-team/LTI_Neural_Navigator/src/annotation/local_auto.py
code served (permissive licence) · get_code("e1f6085807a868da")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile

# Create a temporary file with some content
temp_dir = tempfile.mkdtemp()
test_file = os.path.join(temp_dir, "test_file.bin")

# Write some content to the test file (larger than max_size to trigger splitting)
with open(test_file, 'wb') as f:
    f.write(b'x' * 50000)  # Write 50KB of data, larger than default max_size of 25600

ARGS = {
    'file_path': test_file,
    'max_size': 25600
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
download_pdf_file Not yet run
unverified (harvested, no recorded run)
anlp-team/LTI_Neural_Navigator/src/data_prep/get_paper.py
code served (permissive licence) · get_code("766cd5d654d83f7a")
How it ran

no invocation recorded.

Also present in other papers' code

Of 13 samples lifted from this paper's linked repositories, 13 carry a function identity and 0 do not. Of the 13 distinct functions with an identity, 0 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 0 later) and 13 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

We proposed an end-to-end system design towards utilizing Retrieval Augmented Generation (RAG) to improve the factual accuracy of Large Language Models (LLMs) for domain-specific and time-sensitive queries related to private knowledge-bases. Our system integrates RAG pipeline with upstream datasets processing and downstream performance evaluation. Addressing the challenge of LLM hallucinations, we finetune models with a curated dataset which originates from CMU's extensive resources and annotated with the teacher model. Our experiments demonstrate the system's effectiveness in generating more accurate answers to domain-specific and time-sensitive inquiries. The results also revealed the limitations of fine-tuning LLMs with small-scale and skewed datasets. This research highlights the potential of RAG systems in augmenting LLMs with external datasets for improved performance in knowledge-intensive tasks. Our code and models are available on Github.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2403.10446")
get_code_for_paper("2403.10446")
have("2403.10446")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2403.10446.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2403.10446.svg)](https://syntology.ai/paper/2403.10446)

Connect an agent — have() is free.