We lifted 3 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| force-sight/forcesight | canonical | 2 of 3 |
| Function | Status | Where it lives |
|---|---|---|
| camera_matrix | Ran | force-sight/forcesight/utils/realsense_utils.py code served (permissive licence) · get_code("00967d9ccc53e16b") |
| fisheye_distortion | Ran | force-sight/forcesight/utils/realsense_utils.py code served (permissive licence) · get_code("26088ea86061319e") |
| pred_metrics | Not yet run | force-sight/forcesight/prediction/trainer.py code served (permissive licence) · get_code("25c97938af46d915") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We present ForceSight, a system for text-guided mobile manipulation that predicts visual-force goals using a deep neural network. Given a single RGBD image combined with a text prompt, ForceSight determines a target end-effector pose in the camera frame (kinematic goal) and the associated forces (force goal). Together, these two components form a visual-force goal. Prior work has demonstrated that deep models outputting human-interpretable kinematic goals can enable dexterous manipulation by real robots. Forces are critical to manipulation, yet have typically been relegated to lower-level execution in these systems. When deployed on a mobile manipulator equipped with an eye-in-hand RGBD camera, ForceSight performed tasks such as precision grasps, drawer opening, and object handovers with an 81% success rate in unseen environments with object instances that differed significantly from the training data. In a separate experiment, relying exclusively on visual servoing and ignoring force goals dropped the success rate from 90% to 45%, demonstrating that force goals can significantly enhance performance. The appendix, videos, code, and trained models are available at https://force-sight.github.io/.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2309.12312")
get_code_for_paper("2309.12312")
have("2309.12312")
Connect an agent — have() is free.