SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2406.19626 · NeurIPS · 2024

Safety through feedback in Constrained RL

Pradeep Varakantham, Shashank Reddy Chirra, Praveen Paruchuri

arXiv · PDF · Open in the Atlas

Code that ran

4 of 12 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (1 of them constructed an object rather than computing a result; 0 honoured the contract we drafted, 0 violated it and 1 has no contract verdict; 3 failures of our instrument; 8 no recorded run), measured 2026-11-22; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2406.19626.json · badge: /api/ran/2406.19626.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
shshnkreddy/RLSF marked official in Papers with Code
Papers with Code archive
4 ran · 8 no recorded run
FunctionStatusWhere it lives
RolloutBuffer_PPO_lag Ran
ran on a synthesized input · constructed an object; no result was computed
shshnkreddy/RLSF/Sources/algo/prefim.py
pointer only (licence: NONE) · get_code("f8d32bf33dff4c53")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, shshnkreddy/RLSF when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

batched_inf Ran
ran; our contract draft was wrong, not the code
shshnkreddy/RLSF/Sources/algo/prefim.py
pointer only (licence: NONE) · get_code("34fc1c160d02198e")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, shshnkreddy/RLSF when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

compute_train_stats Ran
ran; our contract draft was wrong, not the code
shshnkreddy/RLSF/Sources/algo/prefim.py
pointer only (licence: NONE) · get_code("afacbe0d7d9a41e5")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, shshnkreddy/RLSF when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

gen_aug_states Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
shshnkreddy/RLSF/Sources/algo/prefim.py
pointer only (licence: NONE) · get_code("eea59308f3780233")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, shshnkreddy/RLSF when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

Algorithm Not yet run
unverified (harvested, no recorded run)
shshnkreddy/RLSF/Sources/algo/prefim.py
pointer only (licence: NONE) · get_code("61bc5c3a0320b9ea")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, shshnkreddy/RLSF when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

CNNStateIndependentPolicy Not yet run
unverified (harvested, no recorded run)
shshnkreddy/RLSF/Sources/algo/prefim.py
pointer only (licence: NONE) · get_code("560aa2dfb0a27188")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, shshnkreddy/RLSF when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

Classifier_network Not yet run
unverified (harvested, no recorded run)
shshnkreddy/RLSF/Sources/algo/prefim.py
pointer only (licence: NONE) · get_code("b6b22a66653c0c54")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, shshnkreddy/RLSF when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

PPO_continuous Not yet run
unverified (harvested, no recorded run)
shshnkreddy/RLSF/Sources/algo/prefim.py
pointer only (licence: NONE) · get_code("cfbd6a0a577ba90c")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, shshnkreddy/RLSF when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

PPO_lag Not yet run
unverified (harvested, no recorded run)
shshnkreddy/RLSF/Sources/algo/prefim.py
pointer only (licence: NONE) · get_code("a4cb8de4018bf2bb")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, shshnkreddy/RLSF when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

PREFIM Not yet run
unverified (harvested, no recorded run)
shshnkreddy/RLSF/Sources/algo/prefim.py
pointer only (licence: NONE) · get_code("89cefdacf466af47")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, shshnkreddy/RLSF when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

StateIndependentPolicy Not yet run
unverified (harvested, no recorded run)
shshnkreddy/RLSF/Sources/algo/prefim.py
pointer only (licence: NONE) · get_code("c2567abd86d8eb58")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, shshnkreddy/RLSF when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

hinge_loss Not yet run
unverified (harvested, no recorded run)
shshnkreddy/RLSF/Sources/algo/prefim.py
pointer only (licence: NONE) · get_code("59c2525415607d22")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, shshnkreddy/RLSF when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

Also present in other papers' code

Of 12 samples lifted from this paper's linked repositories, 12 carry a function identity and 0 do not. Of the 12 distinct functions with an identity, 1 are also present in other papers whose code we hold (1 with at least one paper dated earlier, 0 the same month, 0 later) and 11 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
StateIndependentPolicy
brick of this paper's assembly: helper
1 other paper
1 dated earlier
1707.06347 (2017-07)
Proximal Policy Optimization Algorithms

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

In safety-critical RL settings, the inclusion of an additional cost function is often favoured over the arduous task of modifying the reward function to ensure the agent's safe behaviour. However, designing or evaluating such a cost function can be prohibitively expensive. For instance, in the domain of self-driving, designing a cost function that encompasses all unsafe behaviours (e.g., aggressive lane changes, risky overtakes) is inherently complex, it must also consider all the actors present in the scene making it expensive to evaluate. In such scenarios, the cost function can be learned from feedback collected offline in between training rounds. This feedback can be system generated or elicited from a human observing the training process. Previous approaches have not been able to scale to complex environments and are constrained to receiving feedback at the state level which can be expensive to collect. To this end, we introduce an approach that scales to more complex domains and extends beyond state-level feedback, thus, reducing the burden on the evaluator. Inferring the cost function in such settings poses challenges, particularly in assigning credit to individual states based on trajectorylevel feedback. To address this, we propose a surrogate objective that transforms the problem into a state-level supervised classification task with noisy labels, which can be solved efficiently. Additionally, it is often infeasible to collect feedback for every trajectory generated by the agent, hence, two fundamental questions arise: (1) Which trajectories should be presented to the human? and (2) How many trajectories are necessary for effective learning? To address these questions, we introduce a novelty-based sampling mechanism that selectively involves the evaluator only when the the agent encounters a novel trajectory, and discontinues querying once the trajectories are no longer novel. We showcase the efficiency of our method through experimentation on several benchmark Safety Gymnasium environments and realistic self-driving scenarios. Our method demonstrates nearoptimal performance, comparable to when the cost function is known, by relying solely on trajectory-level feedback across multiple domains. This highlights both the effectiveness and scalability of our approach. The code to replicate these results can be found at https://github.com/shshnkreddy/RLSF * Corresponding Author 38th Conference on Neural Information Processing Systems (NeurIPS 2024).

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2406.19626")
get_code_for_paper("2406.19626")
have("2406.19626")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2406.19626.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2406.19626.svg)](https://syntology.ai/paper/2406.19626)

Connect an agent — have() is free.