We lifted 8 functions out of this paper's own repositories and ran 7 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| somsagar07/FailureShiftRL | canonical | 7 of 8 |
| Function | Status | Where it lives |
|---|---|---|
| calculate_entropy | Ran | somsagar07/FailureShiftRL/Utils/plots.py code served (permissive licence) · get_code("4c3d4d72af55afa2") |
| create_prompt | Ran | somsagar07/FailureShiftRL/Baselines/Generation/config.py code served (permissive licence) · get_code("2af53582222a76e3") |
| find_rank_of_number | Ran | somsagar07/FailureShiftRL/Utils/plots.py code served (permissive licence) · get_code("256cc093063aa12a") |
| get_prompt | Ran | somsagar07/FailureShiftRL/Baselines/Generation/config.py code served (permissive licence) · get_code("9f09587c18b2d305") |
| rotate_image | Ran | somsagar07/FailureShiftRL/Baselines/Classification/config.py code served (permissive licence) · get_code("7e9b52bec18e3b51") |
| saturation_image | Ran | somsagar07/FailureShiftRL/Baselines/Classification/config.py code served (permissive licence) · get_code("c90c12e68affa0a4") |
| w_distance | Ran | somsagar07/FailureShiftRL/Utils/plots.py code served (permissive licence) · get_code("cf9843491a69506e") |
| darken_image | Not yet run | somsagar07/FailureShiftRL/Baselines/Classification/config.py code served (permissive licence) · get_code("31d2a6519cd49b42") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
In large deep neural networks that seem to perform surprisingly well on many tasks, we also observe a few failures related to accuracy, social biases, and alignment with human values, among others. Therefore, before deploying these models, it is crucial to characterize this failure landscape for engineers to debug and legislative bodies to audit models. Nevertheless, it is infeasible to exhaustively test for all possible combinations of factors that could lead to a model's failure. In this paper, we introduce a post-hoc method that utilizes \emph{deep reinforcement learning} to explore and construct the landscape of failure modes in pre-trained discriminative and generative models. With the aid of limited human feedback, we then demonstrate how to restructure the failure landscape to be more desirable by moving away from the discovered failure modes. We empirically show the effectiveness of the proposed method across common Computer Vision, Natural Language Processing, and Vision-Language tasks.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2406.07145")
get_code_for_paper("2406.07145")
have("2406.07145")
Connect an agent — have() is free.