SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2312.14890 · 2023

NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes

arXiv · PDF · Open in the Atlas

Code that ran

13 of 13 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 2 honoured the contract we drafted, 0 violated it and 0 have no contract verdict; 11 failures of our instrument; 0 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2312.14890.json · badge: /api/ran/2312.14890.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
casmlab/nphardeval canonical (classifier label)
classifier basis: census
13 ran · 0 no recorded run
FunctionStatusWhere it lives
gcp_decision_check Ran
ran; our contract draft was wrong, not the code
casmlab/nphardeval/Close/check/check_cmp_GCP_D.py
code served (permissive licence) · get_code("cef31f606aaf119f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import networkx as nx

# Create a valid DIMACS format string for a simple graph coloring problem
# This represents a graph with 4 vertices and 3 edges
dimacs_str = """c Simple graph coloring instance
c
p edge 4 3
e 1 2
e 2 3
e 3 4
"""

# Create an answer dictionary indicating the problem is feasible
answer = {
    'Feasible': 'yes'
}

# Set k_colors to a value that allows the coloring
# For a simple path graph of 4 vertices, 2 colors are sufficient
k_colors = 2

ARGS = {
    'dimacs_str': dimacs_str,
    'answer': answer,
    'k_colors': k_colors
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
networkx — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
gcp_greedy_solution Ran
ran; our contract draft was wrong, not the code
casmlab/nphardeval/Close/check/check_cmp_GCP_D.py
code served (permissive licence) · get_code("9b9c556b14ca91e2")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import networkx as nx

# Create a simple adjacency list for the graph coloring problem
# This represents a small graph with 4 nodes
adjacency_list = {
    0: [1, 2],
    1: [0, 2, 3],
    2: [0, 1, 3],
    3: [1, 2]
}

ARGS = {
    "adjacency_list": adjacency_list
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
networkx — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
greedy_tsp Ran
ran; our contract draft was wrong, not the code
casmlab/nphardeval/Close/check/check_hard_TSP.py
code served (permissive licence) · get_code("1d9385153fc562b0")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Set seed for reproducibility
np.random.seed(42)

# Create a small distance matrix for the TSP
# 5 cities with distances between them
distance_matrix = np.array([
    [0.0, 2.5, 4.1, 3.2, 5.0],
    [2.5, 0.0, 1.8, 3.5, 2.1],
    [4.1, 1.8, 0.0, 2.9, 1.5],
    [3.2, 3.5, 2.9, 0.0, 4.2],
    [5.0, 2.1, 1.5, 4.2, 0.0]
], dtype=np.float64)

ARGS = {
    "distance_matrix": distance_matrix
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
kspCheck Ran
ran; our contract draft was wrong, not the code
casmlab/nphardeval/Close/check/check_cmp_KSP.py
code served (permissive licence) · get_code("f70922a9b0c477bc")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

# Build a valid KSP instance with items and capacity
instance = {
    'items': [
        {'id': 1, 'weight': 2, 'value': 3},
        {'id': 2, 'weight': 3, 'value': 4},
        {'id': 3, 'weight': 4, 'value': 5},
    ],
    'knapsack_capacity': 5
}

# Build a valid solution that matches the optimal solution
# For capacity 5: we can take item 1 (w=2, v=3) and item 2 (w=3, v=4) = total w=5, v=7
# Or item 3 (w=4, v=5) and item 1 (w=2, v=3) = total w=6 (exceeds)
# Or item 3 alone (w=4, v=5)
# Or item 1+2 (w=5, v=7) is optimal
solution = {
    'Feasible': 'yes',
    'TotalValue': 7,
    'SelectedItemIds': [1, 2]
}

ARGS = {
    'instance': instance,
    'solution': solution
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
ksp_optimal_solution Ran
ran, honoured the contract we drafted
casmlab/nphardeval/Close/check/check_cmp_KSP.py
code served (permissive licence) · get_code("65c7255263fe6367")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np

torch.manual_seed(0)

# Analyze the function:
# - knapsacks: A dictionary where keys are itemIds and values are tuples of (weight, value)
# - capacity: An integer representing the knapsack capacity
# The function iterates over knapsacks.items() and accesses weight and value from tuples
# It uses standard Python dict operations and range iteration

# Create a small knapsack problem instance
knapsacks = {
    0: (2, 3),      # item 0: weight=2, value=3
    1: (3, 4),      # item 1: weight=3, value=4
    2: (4, 5),      # item 2: weight=4, value=5
    3: (1, 2),      # item 3: weight=1, value=2
}

capacity = 5

ARGS = {
    "knapsacks": knapsacks,
    "capacity": capacity,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_data Ran
ran; our contract draft was wrong, not the code
casmlab/nphardeval/Open/run/finetune.py
code served (permissive licence) · get_code("aad4839260ed97a9")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import tempfile
import json
import pandas as pd
import os

# Create a temporary directory to serve as data_dir
temp_dir = tempfile.mkdtemp()

# Create mock data files for each task type

# BSP instances file
bsp_data = [{"problem": "test_bsp_1"}, {"problem": "test_bsp_2"}]
with open(os.path.join(temp_dir, 'bsp_instances.json'), 'w') as f:
    json.dump(bsp_data, f)

# EDP instances file
edp_data = [{"problem": "test_edp_1"}]
with open(os.path.join(temp_dir, 'edp_instances.json'), 'w') as f:
    json.dump(edp_data, f)

# SPP instances file
spp_data = [{"problem": "test_spp_1"}]
with open(os.path.join(temp_dir, 'spp_instances.json'), 'w') as f:
    json.dump(spp_data, f)

# KSP instances file
ksp_data = [{"problem": "test_ksp_1"}]
with open(os.path.join(temp_dir, 'ksp_instances.json'), 'w') as f:
    json.dump(ksp_data, f)

# GCP_Decision files
for file_num in range(1, 11):
    with open(os.path.join(temp_dir, f'decision_data_GCP_{file_num}.txt'), 'w') as f:
        f.write("graph 1\n\n")

# TSP_Decision files
for level in range(1, 11):
    for file_num in range(1, 11):
        df = pd.DataFrame([[1, 2], [3, 4]])
        df.to_csv(os.path.join(temp_dir, f'decision_data_TSP_level_{level}_instance_{file_num}.csv'), header=False, index=False)

# GCP files
for file_num in range(1, 11):
    with open(os.path.join(temp_dir, f'synthesized_data_GCP_{file_num}.txt'), 'w') as f:
        f.write("graph 1\n\n")

# TSP files
for level in range(1, 11):
    for file_num in range(1, 11):
        df = pd.DataFrame([[1, 2], [3, 4]])
        df.to_csv(os.path.join(temp_dir, f'synthesized_data_TSP_level_{level}_instance_{file_num}.csv'), header=False, index=False)

ARGS = {
    'data_dir': temp_dir,
    'task': 'BSP'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
mspCheck Ran
ran; our contract draft was wrong, not the code
casmlab/nphardeval/Close/check/check_hard_MSP.py
code served (permissive licence) · get_code("94299eebcf1e9f7b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import ast
import xml.etree.ElementTree as ET

# Construct a valid MSP instance
instance = {
    'time_slots': 10,
    'meetings': [
        {
            'id': 1,
            'duration': 2
        },
        {
            'id': 2,
            'duration': 3
        }
    ],
    'participants': {
        'p1': {
            'meetings': [1, 2],
            'available_slots': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
        },
        'p2': {
            'meetings': [1],
            'available_slots': [0, 1, 2, 3, 4, 5]
        },
        'p3': {
            'meetings': [2],
            'available_slots': [2, 3, 4, 5, 6, 7, 8]
        }
    }
}

# Construct a valid XML string with a solution
solution_dict = {
    '1': [0, 1],
    '2': [3, 4, 5]
}

xml_string = f"""<?xml version="1.0" encoding="UTF-8"?>
<root>
    <final_answer>{solution_dict}</final_answer>
    <reasoning>This is a valid solution</reasoning>
</root>"""

ARGS = {
    'instance': instance,
    'llm_string': xml_string
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
parse_answer Ran
ran; our contract draft was wrong, not the code
casmlab/nphardeval/Close/check/check_hard_GCP.py
code served (permissive licence) · get_code("78998436efd67437")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import xml.etree.ElementTree as ET
import ast

# Create valid XML string with final_answer and reasoning elements
xml_string = """<?xml version="1.0"?>
<root>
    <final_answer>{"0": "answer_one", "1": "answer_two", "2": "answer_three"}</final_answer>
    <reasoning>This is the reasoning</reasoning>
</root>
"""

ARGS = {
    "llm_string": xml_string
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
parse_xml_to_dict Ran
ran, honoured the contract we drafted
casmlab/nphardeval/Close/check/check_hard_GCP.py
code served (permissive licence) · get_code("0a4eb48c9fdfc64a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 2, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import xml.etree.ElementTree as ET

# Create a valid XML string with final_answer and reasoning elements
xml_string = """<?xml version="1.0"?>
<root>
    <final_answer>42</final_answer>
    <reasoning>This is the answer to everything</reasoning>
</root>
"""

ARGS = {
    "xml_string": xml_string
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 2, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
read_dimacs_format Ran
ran; our contract draft was wrong, not the code
casmlab/nphardeval/Close/check/check_cmp_GCP_D.py
code served (permissive licence) · get_code("6c71a84120d287d5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

# Construct valid arguments for read_dimacs_format

# The function expects a DIMACS format string
# DIMACS format is a text format for graph specifications
# Format:
# - c: comment lines (optional)
# - p: problem line with format "p edge num_vertices num_edges"
# - e: edge lines with format "e vertex1 vertex2"

dimacs_str = """c Sample graph
c This is a comment
p edge 4 5
e 1 2
e 1 3
e 2 3
e 2 4
e 3 4
"""

ARGS = {
    "dimacs_str": dimacs_str
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
tspCheck Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
casmlab/nphardeval/Close/check/check_hard_TSP.py
code served (permissive licence) · get_code("a2816f94b482351b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np
import ast
import xml.etree.ElementTree as ET

# Create a small distance matrix for TSP (4 cities)
# Using a symmetric distance matrix
distance_matrix = np.array([
    [0, 10, 15, 20],
    [10, 0, 35, 25],
    [15, 35, 0, 30],
    [20, 25, 30, 0]
], dtype=float)

# Create a valid LLM response string in XML format
# The greedy TSP solution starting from city 0 would be: 0->1->3->2->0
# Let's calculate that distance: 10 + 25 + 30 + 15 = 80
llm_response = """<?xml version="1.0"?>
<response>
    <final_answer>{'Path': '0->1->3->2->0'}</final_answer>
    <reasoning>This is the greedy TSP solution.</reasoning>
</response>"""

ARGS = {
    'distance_matrix': distance_matrix,
    'llm_string': llm_response
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
tsp_approx Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
casmlab/nphardeval/Close/check/check_cmp_TSP_D.py
code served (permissive licence) · get_code("9702d36b3d6750fa")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np
import networkx as nx

# Create a small distance matrix for the TSP problem
# Using a 4x4 matrix representing distances between 4 cities
distance_matrix = np.array([
    [0.0, 1.5, 2.3, 3.1],
    [1.5, 0.0, 1.2, 2.8],
    [2.3, 1.2, 0.0, 1.9],
    [3.1, 2.8, 1.9, 0.0]
], dtype=np.float64)

ARGS = {
    "distance_matrix": distance_matrix
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
networkx, numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family int — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
tsp_decision_check Ran
ran; our contract draft was wrong, not the code
casmlab/nphardeval/Close/check/check_cmp_TSP_D.py
code served (permissive licence) · get_code("0e92242e90ab63bd")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np
import networkx as nx

# Create a small distance matrix for TSP
# Using a 4x4 symmetric distance matrix (valid for TSP)
distance_matrix = np.array([
    [0.0, 1.5, 2.0, 3.5],
    [1.5, 0.0, 2.5, 1.0],
    [2.0, 2.5, 0.0, 1.5],
    [3.5, 1.0, 1.5, 0.0]
], dtype=np.float64)

# Calculate a reasonable threshold
# First compute an approximate tour distance
G = nx.from_numpy_array(distance_matrix)
approx_tour = nx.approximation.traveling_salesman_problem(G)
approx_distance = sum(distance_matrix[approx_tour[i], approx_tour[i + 1]] 
                      for i in range(len(approx_tour) - 1)) + distance_matrix[approx_tour[-1], approx_tour[0]]

# Set threshold to be feasible (equal to or greater than tour distance)
threshold = approx_distance + 0.1

# Create a tour dict that matches the feasibility
tour = {'Feasible': 'yes'}

ARGS = {
    'distance_matrix': distance_matrix,
    'threshold': threshold,
    'tour': tour
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
networkx, numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block

Also present in other papers' code

Of 13 samples lifted from this paper's linked repositories, 13 carry a function identity and 0 do not. Of the 13 distinct functions with an identity, 13 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 13 later) and 0 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
gcp_decision_check 1 other paper
1 later
2024.acl-long.225 (2024 (year only))
NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes
gcp_greedy_solution 1 other paper
1 later
2024.acl-long.225 (2024 (year only))
NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes
greedy_tsp 1 other paper
1 later
2024.acl-long.225 (2024 (year only))
NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes
kspCheck 1 other paper
1 later
2024.acl-long.225 (2024 (year only))
NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes
ksp_optimal_solution 1 other paper
1 later
2024.acl-long.225 (2024 (year only))
NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes
load_data 1 other paper
1 later
2024.acl-long.225 (2024 (year only))
NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes
mspCheck 1 other paper
1 later
2024.acl-long.225 (2024 (year only))
NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes
parse_answer 1 other paper
1 later
2024.acl-long.225 (2024 (year only))
NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes
parse_xml_to_dict 1 other paper
1 later
2024.acl-long.225 (2024 (year only))
NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes
read_dimacs_format 1 other paper
1 later
2024.acl-long.225 (2024 (year only))
NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes
tspCheck 1 other paper
1 later
2024.acl-long.225 (2024 (year only))
NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes
tsp_approx 1 other paper
1 later
2024.acl-long.225 (2024 (year only))
NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes
tsp_decision_check 1 other paper
1 later
2024.acl-long.225 (2024 (year only))
NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Complex reasoning ability is one of the most important features of current LLMs, which has also been leveraged to play an integral role in complex decision-making tasks. Therefore, the investigation into the reasoning capabilities of Large Language Models (LLMs) is critical: numerous benchmarks have been established to assess the reasoning abilities of LLMs. However, current benchmarks are inadequate in offering a rigorous evaluation of the full extent of reasoning abilities that LLMs are capable of achieving. They are also prone to the risk of overfitting, as these benchmarks, being publicly accessible and static, allow models to potentially tailor their responses to specific benchmark metrics, thereby inflating their performance. Addressing these limitations, our research introduces a new benchmark, named NPHardEval. This benchmark is designed to evaluate the reasoning abilities of LLMs across a broad spectrum of 900 algorithmic questions, extending up to the NP-Hard complexity class. These questions are meticulously chosen to represent a wide range of complexity class below the NP-hard complexity class, offering a rigorous measure of the reasoning ability of LLMs. Through this study, we shed light on the current state of reasoning in LLMs, providing an objective and rigorous perspective through the comparison of LLMs' performance across complex classes. Moreover, this benchmark is designed with a dynamic update mechanism, where the datapoints are refreshed on a monthly basis. Such regular updates play a crucial role in mitigating the risk of LLMs overfitting to the benchmark, promoting a more accurate and reliable assessment of their reasoning capabilities. The benchmark dataset and code of NPHardEval are available at https://github.com/casmlab/NPHardEval.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2312.14890")
get_code_for_paper("2312.14890")
have("2312.14890")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2312.14890.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2312.14890.svg)](https://syntology.ai/paper/2312.14890)

Connect an agent — have() is free.