SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2406.04339 · NeurIPS · 2024

RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulation

Shanghang Zhang, Renrui Zhang, Robotics, Zhenyu Wang, Yandong Guo, Kaichen Zhou, Xiaoqi Li, Jiaming Liu, Mengzhen Liu, Pengju An, Senqiao Yang, Openvla Manipllm, and 1 more

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.

Abstract

A fundamental objective in robot manipulation is to enable models to comprehend visual scenes and execute actions. Although existing Vision-Language-Action (VLA) models for robots can handle a range of basic tasks, they still face challenges in two areas: (1) insufficient reasoning ability to tackle complex tasks, and (2) high computational costs for VLA model fine-tuning and inference. The recently proposed state space model (SSM) known as Mamba demonstrates promising capabilities in non-trivial sequence modeling with linear inference complexity. Inspired by this, we introduce RoboMamba, an end-to-end robotic VLA model that leverages Mamba to deliver both robotic reasoning and action capabilities, while maintaining efficient fine-tuning and inference. Specifically, we first integrate the vision encoder with Mamba, aligning visual tokens with language embedding through co-training, empowering our model with visual common sense and roboticrelated reasoning. To further equip RoboMamba with SE(3) pose prediction abilities, we explore an efficient fine-tuning strategy with a simple policy head. We find that once RoboMamba possesses sufficient reasoning capability, it can acquire manipulation skills with minimal fine-tuning parameters (0.1% of the model) and time. In experiments, RoboMamba demonstrates outstanding reasoning capabilities on general and robotic evaluation benchmarks. Meanwhile, our model showcases impressive pose prediction results in both simulation and real-world experiments, achieving inference speeds 3 times faster than existing VLA models. Our project web page: https://sites.google.com/view/robomamba-web

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2406.04339")
get_code_for_paper("2406.04339")
have("2406.04339")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2406.04339.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2406.04339.svg)](https://syntology.ai/paper/2406.04339)

Connect an agent — have() is free.