SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2310.12931 · ICLR · 2024

Eureka: Human-Level Reward Design via Coding Large Language Models

Anima Anandkumar, Yuke Zhu, Caltech, Dinesh Jayaraman, Osbert Bastani, Guanzhi Wang, De-An Huang, Yecheng Jason Ma, William Liang, Linxi, Jim Fan, Upenn

arXiv · PDF · Open in the Atlas

Code that ran

11 of 12 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 0 honoured the contract we drafted, 1 violated it and 0 have no contract verdict; 10 failures of our instrument; 1 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2310.12931.json · badge: /api/ran/2310.12931.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
eureka-research/Eureka canonical (classifier label)
classifier basis: census
8 ran · 1 no recorded run
ModalMinds/MM-EUREKA extension (classifier label)
classifier basis: classifier_2025
3 ran · 0 no recorded run
FunctionStatusWhere it lives
extract_answer_with_tags Ran
ran; our contract draft was wrong, not the code
ModalMinds/MM-EUREKA/examples/scripts/reward_func_qwen_instruct.py
code served (permissive licence) · get_code("f7bb7fab758e88c6")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def extract_answer_with_tags(text):
    match = re.search('(<answer>.*?</answer>)', text)
    if match:
        return match.group(1)
    return None

# Setup code
ARGS = {
    'text': 'Some prefix text <answer>This is the answer content</answer> some suffix text'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_class_info Ran
ran; our contract draft was wrong, not the code
eureka-research/Eureka/eureka/utils/prune_env.py
code served (permissive licence) · get_code("b2430468f620777e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import ast
import tempfile
import os

# Create a temporary Python file with a class definition
temp_dir = tempfile.mkdtemp()
temp_file = os.path.join(temp_dir, "test_module.py")

class_content = '''class ExampleClass:
    """This is the class docstring.
    
    This is the second paragraph that should not be included.
    """
    
    def __init__(self):
        pass
    
    def method(self):
        return "example"
'''

with open(temp_file, 'w') as f:
    f.write(class_content)

ARGS = {
    'file_path': temp_file
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_observation_code Ran
ran; our contract draft was wrong, not the code
eureka-research/Eureka/eureka/utils/extract_task_code.py
code served (permissive licence) · get_code("ace4148116adb7ef")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import tempfile
import os

# Create a temporary file with a function that matches the regex pattern
temp_dir = tempfile.mkdtemp()
temp_file = os.path.join(temp_dir, "test_file.py")

with open(temp_file, 'w') as f:
    f.write("""def calculate_observations(data):
    result = data * 2
    return result

def extract_observations(x, y):
    print("extracting observations")
    processed = x + y
    
    return processed

def process_observations_data(batch):
    return batch

""")

ARGS = {
    "filename": temp_file
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
extract_task_code Ran
ran; our contract draft was wrong, not the code
eureka-research/Eureka/eureka/utils/extract_task_code.py
code served (permissive licence) · get_code("7edd71cf38a4ac85")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import tempfile
import os

# Create a temporary file with sample content
temp_file = tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.py')
temp_filename = temp_file.name

# Write sample content that matches the function's parsing logic
sample_content = """# This is a comment
# Another comment
def task_function():
    return 42
## Task ends here, reward begins
def reward_function():
    return 10
"""

temp_file.write(sample_content)
temp_file.close()

ARGS = {
    'filename': temp_filename
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
file_to_string Ran
ran; our contract draft was wrong, not the code
eureka-research/Eureka/eureka/utils/extract_task_code.py
pointer only (licence: MIT); the stored copy get_code serves, from labicon/curricullm, is pointer only (licence: NONE) · get_code("ab6795497be8743a")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from labicon/curricullm, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

filter_traceback Ran
ran; our contract draft was wrong, not the code
eureka-research/Eureka/eureka/utils/misc.py
code served (permissive licence) · get_code("16b43ab11204e083")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# This function takes a string parameter `s` and filters a traceback from it.
# It looks for lines starting with 'Traceback' and stops when it finds 
# 'Set the environment variable HYDRA_FULL_ERROR=1'

# Create a sample traceback string to test the function
traceback_string = """Traceback (most recent call last):
  File "script.py", line 10, in <module>
    result = some_function()
  File "module.py", line 5, in some_function
    return bad_operation()
  File "other.py", line 2, in bad_operation
    raise ValueError("Something went wrong")
ValueError: Something went wrong
Set the environment variable HYDRA_FULL_ERROR=1 for a full error trace."""

ARGS = {
    's': traceback_string
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
find_files_with_substring Ran
ran; our contract draft was wrong, not the code
eureka-research/Eureka/eureka/utils/file_utils.py
code served (permissive licence) · get_code("7e3ca15f72ec5048")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile
import shutil

# Create a temporary directory structure with test files
temp_dir = tempfile.mkdtemp()

# Create some subdirectories and files
os.makedirs(os.path.join(temp_dir, "subdir1"), exist_ok=True)
os.makedirs(os.path.join(temp_dir, "subdir2"), exist_ok=True)

# Create files with and without the substring
open(os.path.join(temp_dir, "test_file.txt"), "w").close()
open(os.path.join(temp_dir, "data_file.txt"), "w").close()
open(os.path.join(temp_dir, "subdir1", "test_document.pdf"), "w").close()
open(os.path.join(temp_dir, "subdir1", "report.pdf"), "w").close()
open(os.path.join(temp_dir, "subdir2", "test_image.png"), "w").close()
open(os.path.join(temp_dir, "subdir2", "photo.png"), "w").close()

ARGS = {
    "directory": temp_dir,
    "substring": "test"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_query_from_query Ran
ran, violated the contract we drafted
ModalMinds/MM-EUREKA/examples/scripts/reward_func_qwen_instruct.py
code served (permissive licence) · get_code("4dbb9ce8ca054e5a")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

problem_pattern = '<\\|im_start\\|>user\\n(.*?)<\\|im_end\\|>'

def get_query_from_query(q: str):
    try:
        matches = re.findall(problem_pattern, q, re.DOTALL)
        return matches[0]
    except:
        return q

# Test case 1: Valid query with the expected pattern
test_query_1 = '<|im_start|>user\nWhat is the capital of France?\n<|im_end|>'

# Test case 2: Query without the pattern (should return the original string)
test_query_2 = 'This is just a plain query'

# Test case 3: Query with the pattern and multiline content
test_query_3 = '<|im_start|>user\nMultiline\nquery\ncontent\n<|im_end|>'

ARGS = {
    'q': test_query_1
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_response_from_query Ran
ran; our contract draft was wrong, not the code
ModalMinds/MM-EUREKA/examples/scripts/reward_func_qwen_instruct.py
code served (permissive licence) · get_code("621fb76355aa5e4b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

response_prefix = '<\\|im_start\\|>assistant\\n'

def get_response_from_query(q: str):
    ends_of_sentence = ['<|im_end|>', '<|end▁of▁sentence|>', '<|endoftext|>']
    pos = re.search(response_prefix, q)
    if pos is None:
        return ''
    response = q[pos.end():]
    for e in ends_of_sentence:
        response = response.replace(e, '')
    return response.strip()

# Setup code to construct valid arguments
ARGS = {
    'q': '<|im_start|>assistant\nHello, this is a test response<|im_end|>'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
import_class_from_file Ran
ran; our contract draft was wrong, not the code
eureka-research/Eureka/eureka/utils/file_utils.py
code served (permissive licence) · get_code("f409fbe9be3b6b98")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned type. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import importlib.util
import tempfile
import os

# Create a temporary Python file with a test class/function
temp_dir = tempfile.mkdtemp()
test_file = os.path.join(temp_dir, "test_module.py")

# Write a simple test class to the file
with open(test_file, "w") as f:
    f.write("""
class TestClass:
    def __init__(self):
        self.value = 42
    
    def method(self):
        return self.value

def test_function():
    return "Hello, World!"
""")

# Define arguments for the function
ARGS = {
    "file_path": test_file,
    "function_name": "TestClass"
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type type — a type output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a type output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_tensorboard_logs Ran
ran; our contract draft was wrong, not the code
eureka-research/Eureka/eureka/utils/file_utils.py
code served (permissive licence) · get_code("dbeca344f569e3c3")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned defaultdict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import tempfile
import os
from tensorboard.plugins.scalar import metadata as scalar_metadata
from tensorboard.compat.proto import summary_pb2, event_pb2
import time

# Create a temporary directory with valid tensorboard event files
tmpdir = tempfile.mkdtemp()

# Write some minimal event files to the temporary directory
def write_scalar_event(logdir, tag, value, step):
    """Helper to write a scalar event to tensorboard logs"""
    event_file = os.path.join(logdir, f'events.out.tfevents.{int(time.time())}')
    
    # Create summary with scalar value
    summary = summary_pb2.Summary()
    summary_value = summary.value.add()
    summary_value.tag = tag
    summary_value.simple_value = value
    
    # Create event
    event = event_pb2.Event()
    event.wall_time = time.time()
    event.step = step
    event.summary.CopyFrom(summary)
    
    # Write to file
    with open(event_file, 'ab') as f:
        f.write(event.SerializeToString())

# Write some test events
write_scalar_event(tmpdir, 'loss', 0.5, 0)
write_scalar_event(tmpdir, 'loss', 0.3, 1)
write_scalar_event(tmpdir, 'accuracy', 0.7, 0)
write_scalar_event(tmpdir, 'accuracy', 0.85, 1)

ARGS = {
    'path': tmpdir
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
tensorboard, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type defaultdict — a defaultdict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a defaultdict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
depth_image_to_point_cloud_GPU Not yet run
unverified (harvested, no recorded run)
eureka-research/Eureka/eureka/envs/bidex/shadow_hand_block_stack.py
code served (permissive licence) · get_code("a03f028b2da7c641")
How it ran

no invocation recorded.

Also present in other papers' code

Of 12 samples lifted from this paper's linked repositories, 12 carry a function identity and 0 do not. Of the 12 distinct functions with an identity, 2 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 2 later) and 10 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
file_to_string 3 other papers
3 later
2409.18382 (2024-09)
CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Models
2412.14995 (2024-12)
HSEvo: Elevating Automatic Heuristic Design with Diversity-Driven Harmony Search and Genetic Algorithm Using LLMs
2501.08603 (2025-01)
Monte Carlo Tree Search for Comprehensive Exploration in LLM-Based Automatic Heuristic Design
filter_traceback 2 other papers
2 later
2412.14995 (2024-12)
HSEvo: Elevating Automatic Heuristic Design with Diversity-Driven Harmony Search and Genetic Algorithm Using LLMs
2501.08603 (2025-01)
Monte Carlo Tree Search for Comprehensive Exploration in LLM-Based Automatic Heuristic Design

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Large Language Models (LLMs) have excelled as high-level semantic planners for sequential decision-making tasks. However, harnessing them to learn complex lowlevel manipulation tasks, such as dexterous pen spinning, remains an open problem. We bridge this fundamental gap and present EUREKA, a human-level reward design algorithm powered by LLMs. EUREKA exploits the remarkable zero-shot generation, code-writing, and in-context improvement capabilities of state-of-theart LLMs, such as GPT-4, to perform evolutionary optimization over reward code. The resulting rewards can then be used to acquire complex skills via reinforcement learning. Without any task-specific prompting or pre-defined reward templates, EUREKA generates reward functions that outperform expert human-engineered rewards. In a diverse suite of 29 open-source RL environments that include 10 distinct robot morphologies, EUREKA outperforms human experts on 83% of the tasks, leading to an average normalized improvement of 52%. The generality of EUREKA also enables a new gradient-free in-context learning approach to reinforcement learning from human feedback (RLHF), readily incorporating human inputs to improve the quality and the safety of the generated rewards without model updating. Finally, using EUREKA rewards in a curriculum learning setting, we demonstrate for the first time, a simulated Shadow Hand capable of performing pen spinning tricks, adeptly manipulating a pen in circles at rapid speed.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2310.12931")
get_code_for_paper("2310.12931")
have("2310.12931")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2310.12931.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2310.12931.svg)](https://syntology.ai/paper/2310.12931)

Connect an agent — have() is free.