Act to See, See to Act: Diffusion-Driven Perception-Action Interplay for Adaptive Policies

Jing Wang, Jing Tang, Li Cheng, Weiting Peng, Zeyu Gong, Xihua Wang, Bo Tao

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 5 functions out of this paper's own repositories and ran 4 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• ARISE-Initiative/robomimic

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Existing imitation learning methods decouple perception and action, which overlooks the causal reciprocity between sensory representation and action execution that humans naturally leverage for adaptive behaviors. To bridge this gap, we introduce Action-Guided Diffusion Policy (DP-AG), a unified representation learning that explicitly models a dynamic interplay between perception and action through probabilistic latent dynamics. DP-AG encodes latent observations into a Gaussian posterior via variational inference and evolves them using an action-guided SDE, where the Vector-Jacobian Product (VJP) of the diffusion policy's noise predictions serves as a structured stochastic force driving latent updates. To promote bidirectional learning between perception and action, we introduce a cycle-consistent contrastive loss that organizes the gradient flow of the noise predictor into a coherent perception-action loop, enforcing mutually consistent transitions in both latent updates and action refinements. Theoretically, we derive a variational lower bound for the action-guided SDE, and prove that the contrastive objective enhances continuity in both latent and action trajectories. Empirically, DP-AG significantly outperforms state-of-the-art methods across simulation benchmarks and real-world UR5 manipulation tasks. As a result, our DP-AG offers a promising step toward bridging biological adaptability and artificial policy learning. Code is available on our project website: https://jingwang18.github.io/dp-ag.github.io/. Robust decision-making relies on a continuous interplay between perception and action [O'regan and Noë, 2001]. Humans naturally embody this principle by dynamically refining their environmental understanding through feedback from their own actions [Brooks, 1991]. Motivated by this, we propose Action-Guided Diffusion Policy (DP-AG), a representation learning framework for IL that explicitly models the perception-action interplay through probabilistic latent dynamics. We build upon DP 39th Conference on Neural Information Processing Systems (NeurIPS 2025).

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2509.25822")
get_code_for_paper("2509.25822")
have("2509.25822")

Connect an agent — have() is free.