SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2210.07573 · NeurIPS · 2022

Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm

Ashish Kumar, Shalabh Bhatnagar

arXiv · PDF · Open in the Atlas

Code that ran

9 of 13 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 2 honoured the contract we drafted, 0 violated it and 4 have no contract verdict; 3 failures of our instrument; 4 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2210.07573.json · badge: /api/ran/2210.07573.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
akjayant/mbppol canonical (classifier label)
classifier basis not recorded on the link
9 ran · 4 no recorded run
FunctionStatusWhere it lives
colorize Ran
ran; our contract draft was wrong, not the code
akjayant/mbppol/src/utils/logx.py
pointer only (licence: MIT); the stored copy get_code serves, from milanganai/milanganai.github.io, is pointer only (licence: NONE) · get_code("7e36f1c357c4618d")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from milanganai/milanganai.github.io, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

combined_shape Ran
ran; our contract draft was wrong, not the code
akjayant/mbppol/src/core.py
pointer only (licence: MIT); the stored copy get_code serves, from yinglunxu/reward_poisoning_attack_drl, is pointer only (licence: NONE) · get_code("57542f87e248db45")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from yinglunxu/reward_poisoning_attack_drl, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

count_vars Ran
ran, honoured the contract we drafted
akjayant/mbppol/src/core.py
pointer only (licence: MIT); the stored copy get_code serves, from milanganai/milanganai.github.io, is pointer only (licence: NONE) · get_code("4cca5e460067b900")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from milanganai/milanganai.github.io, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

dist_xy Ran
ran on a synthesized input
akjayant/mbppol/src/aux.py
code served (permissive licence) · get_code("8a69cfc50e0f440c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float64, ndim 0, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# The function dist_xy takes two position arguments (pos1 and pos2)
# and computes the Euclidean distance between them in 2D space.
# It handles both 2D positions (x, y) and 3D positions (x, y, z) by extracting [:2]

# Create valid arguments
pos1_2d = np.array([1.0, 2.0])
pos2_2d = np.array([4.0, 6.0])

# Also test with 3D positions to verify the slicing works
pos1_3d = np.array([1.0, 2.0, 5.0])
pos2_3d = np.array([4.0, 6.0, 10.0])

ARGS = {
    "pos1": pos1_2d,
    "pos2": pos2_2d
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float64, ndim 0, dtype family float — no output shape is stated: observe_signature.py recorded float ndim 0, which disagrees with the fixture run
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
flat_concat Ran
ran, honoured the contract we drafted
akjayant/mbppol/src/utils/mpi_tf.py
pointer only (licence: MIT); the stored copy get_code serves, from milanganai/milanganai.github.io, is pointer only (licence: NONE) · get_code("f29202d8108bccd3")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from milanganai/milanganai.github.io, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

get_data Ran
ran on a synthesized input
akjayant/mbppol/src/model.py
code served (permissive licence) · get_code("37d85d44187b2915")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np
import gzip
import io
import struct

# Create temporary gzip files with MNIST-like data
def create_test_data_files(num_examples=2):
    # Create inputs file (28x28 images as bytes)
    inputs_buffer = io.BytesIO()
    with gzip.GzipFile(fileobj=inputs_buffer, mode='wb') as gz:
        # MNIST magic number and dimensions
        gz.write(struct.pack('>IIII', 2051, num_examples, 28, 28))
        # Write pixel data (28*28*num_examples bytes, values 0-255)
        pixel_data = np.random.randint(0, 256, size=28*28*num_examples, dtype=np.uint8).tobytes()
        gz.write(pixel_data)
    inputs_buffer.seek(0)
    
    # Create labels file
    labels_buffer = io.BytesIO()
    with gzip.GzipFile(fileobj=labels_buffer, mode='wb') as gz:
        # MNIST magic number and count
        gz.write(struct.pack('>II', 2049, num_examples))
        # Write label data (num_examples bytes, values 0-9)
        label_data = np.random.randint(0, 10, size=num_examples, dtype=np.uint8).tobytes()
        gz.write(label_data)
    labels_buffer.seek(0)
    
    return inputs_buffer, labels_buffer

# Create temporary files
inputs_file = io.BytesIO()
labels_file = io.BytesIO()

num_ex = 2

# Write inputs file
with gzip.GzipFile(fileobj=inputs_file, mode='wb') as gz:
    gz.write(struct.pack('>IIII', 2051, num_ex, 28, 28))
    pixel_data = np.random.randint(0, 256, size=28*28*num_ex, dtype=np.uint8).tobytes()
    gz.write(pixel_data)

# Write labels file
with gzip.GzipFile(fileobj=labels_file, mode='wb') as gz:
    gz.write(struct.pack('>II', 2049, num_ex))
    label_data = np.random.randint(0, 10, size=num_ex, dtype=np.uint8).tobytes()
    gz.write(label_data)

# Get the byte data
inputs_data = inputs_file.getvalue()
labels_data = labels_file.getvalue()

# Write to actual temporary files
import tempfile
import os

temp_dir = tempfile.gettempdir()
inputs_file_path = os.path.join(temp_dir, 'test_inputs.gz')
labels_file_path = os.path.join(temp_dir, 'test_labels.gz')

with open(inputs_file_path, 'wb') as f:
    f.write(inputs_data)

with open(labels_file_path, 'wb') as f:
    f.write(labels_data)

ARGS = {
    'inputs_file_path': inputs_file_path,
    'labels_file_path': labels_file_path,
    'num_examples': num_ex
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_goal_flag Ran
ran on a synthesized input
akjayant/mbppol/src/aux.py
code served (permissive licence) · get_code("6bd1e1acc559a3c0")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# The function get_goal_flag takes two parameters:
# - robot_pos: a position (can be 2D or 3D array-like)
# - goal_pos: a position (can be 2D or 3D array-like)
# Both are converted to numpy arrays and their XY components are used

# Create sample positions
robot_position = np.array([1.0, 2.0, 0.5])  # 3D position
goal_position = np.array([1.1, 2.1])  # 2D position

ARGS = {
    "robot_pos": robot_position,
    "goal_pos": goal_position,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_reward_cost Ran
ran on a synthesized input
akjayant/mbppol/src/aux.py
code served (permissive licence) · get_code("f09be98e3ec66db5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Analyze the function signature and body:
# - last_dist_goal: a scalar float (distance value)
# - robot_pos: array-like, XY or XYZ position
# - hazards_pos: iterable of positions (each position is array-like)
# - goal_pos: array-like, XY or XYZ position

# Create valid arguments
last_dist_goal = 5.0

robot_pos = np.array([1.0, 2.0])

hazards_pos = [
    np.array([1.1, 2.1]),
    np.array([5.0, 6.0])
]

goal_pos = np.array([3.0, 4.0])

ARGS = {
    'last_dist_goal': last_dist_goal,
    'robot_pos': robot_pos,
    'hazards_pos': hazards_pos,
    'goal_pos': goal_pos
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
mlp Ran
ran; our contract draft was wrong, not the code
akjayant/mbppol/src/core.py
pointer only (licence: MIT); the stored copy get_code serves, from idsia/policyevaluator, is pointer only (licence: NONE) · get_code("7569e2a512ec417c")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from idsia/policyevaluator, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

assign_params_from_flat Not yet run
unverified (harvested, no recorded run)
akjayant/mbppol/src/utils/mpi_tf.py
pointer only (licence: MIT); the stored copy get_code serves, from milanganai/milanganai.github.io, is pointer only (licence: NONE) · get_code("042875cf711accd0")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from milanganai/milanganai.github.io, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

init_weights Not yet run
unverified (harvested, no recorded run)
akjayant/mbppol/src/model.py
code served (permissive licence) · get_code("0661bf42848a4988")
How it ran

no invocation recorded.

restore_tf_graph Not yet run
unverified (harvested, no recorded run)
akjayant/mbppol/src/utils/logx.py
pointer only (licence: MIT); the stored copy get_code serves, from milanganai/milanganai.github.io, is pointer only (licence: NONE) · get_code("8a70bbd1276e274c")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from milanganai/milanganai.github.io, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

sync_params Not yet run
unverified (harvested, no recorded run)
akjayant/mbppol/src/utils/mpi_tf.py
pointer only (licence: MIT); the stored copy get_code serves, from milanganai/milanganai.github.io, is pointer only (licence: NONE) · get_code("82d7bab9b4ade8d1")
How it ran

The body of this sample is withheld: this paper's link allows the body inline (MIT), but get_code serves the stored copy, first harvested from milanganai/milanganai.github.io, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

Also present in other papers' code

Of 13 samples lifted from this paper's linked repositories, 13 carry a function identity and 0 do not. Of the 13 distinct functions with an identity, 9 are also present in other papers whose code we hold (8 with at least one paper dated earlier, 1 the same month, 9 later) and 4 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
colorize 17 other papers
5 dated earlier, 1 the same month, 11 later
1802.09477 (2018-02)
Addressing Function Approximation Error in Actor-Critic Methods
1909.12255 (2019-09)
Harnessing Structures for Value-Based Planning and Reinforcement Learning
2006.07442 (2020-06)
Self-Imitation Learning via Generalized Lower Bound Q-learning
and 14 more not named here
combined_shape 14 other papers
9 dated earlier, 5 later
1906.06818 (2019-06)
Stacked Capsule Autoencoders
1912.13465 (2019-12)
Reward-Conditioned Policies
2006.07442 (2020-06)
Self-Imitation Learning via Generalized Lower Bound Q-learning
and 11 more not named here
mlp 12 other papers
8 dated earlier, 4 later · 2 only as an assembly brick
1905.01240 (2019-05)
Information asymmetry in KL-regularized RL
2006.07442 (2020-06)
Self-Imitation Learning via Generalized Lower Bound Q-learning
2006.09359 (2020-06)
AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
and 9 more not named here
count_vars 8 other papers
5 dated earlier, 3 later
2006.07442 (2020-06)
Self-Imitation Learning via Generalized Lower Bound Q-learning
2010.09635 (2020-10)
Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control
2106.09110 (2021-06)
Safe Reinforcement Learning Using Advantage-Based Intervention
and 5 more not named here
flat_concat 6 other papers
1 dated earlier, 5 later
2006.07442 (2020-06)
Self-Imitation Learning via Generalized Lower Bound Q-learning
2302.00808 (2023-02)
ACPO: A Policy Optimization Algorithm for Average MDPs with Constraints
2302.02299 (2023-02)
Sample Dropout: A Simple yet Effective Variance Reduction Technique in Deep Policy Optimization
and 3 more not named here
assign_params_from_flat 4 other papers
1 dated earlier, 3 later
2006.07442 (2020-06)
Self-Imitation Learning via Generalized Lower Bound Q-learning
2302.00808 (2023-02)
ACPO: A Policy Optimization Algorithm for Average MDPs with Constraints
2401.14758 (2024-01)
Off-Policy Primal-Dual Safe Reinforcement Learning
and 1 more not named here
restore_tf_graph 4 other papers
2 dated earlier, 2 later
1802.09477 (2018-02)
Addressing Function Approximation Error in Actor-Critic Methods
2006.07442 (2020-06)
Self-Imitation Learning via Generalized Lower Bound Q-learning
2605.09638 (2026-05)
Plan2Cleanse: Test-Time Backdoor Defense via Monte-Carlo Planning in Deep Reinforcement Learning
and 1 more not named here
init_weights 2 other papers
1 dated earlier, 1 later
2201.11927 (2022-01)
Constrained Variational Policy Optimization for Safe Reinforcement Learning
2401.14758 (2024-01)
Off-Policy Primal-Dual Safe Reinforcement Learning
sync_params 1 other paper
1 later
2605.11975 (2026-05)
Stochastic Minimum-Cost Reach-Avoid Reinforcement Learning

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

During initial iterations of training in most Reinforcement Learning (RL) algorithms, agents perform a significant number of random exploratory steps. In the real world, this can limit the practicality of these algorithms as it can lead to potentially dangerous behavior. Hence safe exploration is a critical issue in applying RL algorithms in the real world. This problem has been recently well studied under the Constrained Markov Decision Process (CMDP) Framework, where in addition to single-stage rewards, an agent receives single-stage costs or penalties as well depending on the state transitions. The prescribed cost functions are responsible for mapping undesirable behavior at any given time-step to a scalar value. The goal then is to find a feasible policy that maximizes reward returns while constraining the cost returns to be below a prescribed threshold during training as well as deployment. We propose an On-policy Model-based Safe Deep RL algorithm in which we learn the transition dynamics of the environment in an online manner as well as find a feasible optimal policy using the Lagrangian Relaxation-based Proximal Policy Optimization. We use an ensemble of neural networks with different initializations to tackle epistemic and aleatoric uncertainty issues faced during environment model learning. We compare our approach with relevant model-free and model-based approaches in Constrained RL using the challenging Safe Reinforcement Learning benchmark -the Open AI Safety Gym. We demonstrate that our algorithm is more sample efficient and results in lower cumulative hazard violations as compared to constrained model-free approaches. Further, our approach shows better reward performance than other constrained model-based approaches in the literature.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2210.07573")
get_code_for_paper("2210.07573")
have("2210.07573")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2210.07573.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2210.07573.svg)](https://syntology.ai/paper/2210.07573)

Connect an agent — have() is free.