Olivier Sigaud, Nicolas Thome, Zakariae El, Philippe Gratias-Quiquandon
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Diffusion policies (DPs) have emerged as expressive policy representations for robot learning, often used with imitation learning methods such as behavioral cloning (BC). However, while their success has largely been confined to BC, direct reinforcement learning (RL) fine-tuning remains challenging because actions are generated through a multi-step denoising process. In this work, we propose MODIP, a framework for the offline-to-online fine-tuning of DPs. Rather than directly applying RL to the DP, MODIP leverages a world model (WM) to guide policy adaptation and keeps the simplicity and stability of BC. We utilize model predictive control (MPC) to generate high-quality trajectories within the WM, and use them as supervised targets for fine-tuning the DP. To make MPC planning efficient, MODIP uses a terminal state value instead of a policy-dependent stateaction value, reducing inference time. Additionally, MODIP trains critics with policy-independent TD targets, reducing training time. Experiments on D4RL (MuJoCo, Kitchen) and RoboMimic tasks show that MODIP improves diffusion policies beyond BC, and is competitive with or outperforms diffusion policy RL fine-tuning methods and strong model-based baselines such as TD-MPC2. Code is available at: github.com/elasriz/DPMPC/ When a reward signal is available, reinforcement learning (RL) provides a natural way to improve a pretrained DP beyond imitation learning. Recent RL methods such as Diffusion-QL (Wang et al., 2022), EDP (Kang et al., 2023), DPPO (Ren et al., 2025), and DSRL (Wagenmaker et al., 2025) show that DPs can indeed be improved using RL. However, fine-tuning DPs with RL remains more challenging than fine-tuning standard Gaussian policies. Since actions are generated through a multi-step denoising process, policy optimization is more complex and may suffer from instability. In this work, we take a different approach. Rather than directly optimizing the DP with RL, we use model predictive control (MPC) and BC as an indirect policy improvement operator. Our method, MODIP, learns a latent world model, a reward model, and a terminal value function. At interaction time, a DP-guided MPC planner uses the DP as an expressive action sequence prior within a samplingbased trajectory optimizer such as Model Predictive Path Integral control (MPPI) (Williams et al., 2017). The planner combines diffusion-generated candidates with randomly sampled trajectories and refines them through model-based rollouts in the learned latent space. It scores them using predicted rewards together with the terminal value V . The resulting trajectories are then used to fine-tune the DP with the standard supervised denoising loss. As a result, the policy retains the simplicity and stability of supervised learning, while improving through a progressively better training data distribution induced by MPC.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2606.10825")
get_code_for_paper("2606.10825")
have("2606.10825")
Connect an agent — have() is free.