We lifted 11 functions out of this paper's own repositories and ran 9 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| Max-Fu/icrt | canonical | 9 of 11 |
| Function | Status | Where it lives |
|---|---|---|
| apply_rotary_emb | Ran | Max-Fu/icrt/icrt/models/policy/llama.py code served (permissive licence) · get_code("373a3280ff76d40d") |
| get_2d_sincos_pos_embed | Ran | Max-Fu/icrt/icrt/models/backbones/pos_embed.py code served (permissive licence) · get_code("cd809ededc7ba250") |
| get_2d_sincos_pos_embed_for_coords | Ran | Max-Fu/icrt/icrt/models/backbones/pos_embed.py code served (permissive licence) · get_code("45dd229701ef23a9") |
| get_default_vision_aug | Ran | Max-Fu/icrt/icrt/models/backbones/preprocess.py code served (permissive licence) · get_code("a9138b353bdf772d") |
| precompute_freqs_cis | Ran | Max-Fu/icrt/icrt/models/policy/llama.py code served (permissive licence) · get_code("14a84c2cbfebc413") |
| reshape_for_broadcast | Ran | Max-Fu/icrt/icrt/models/policy/llama.py code served (permissive licence) · get_code("61fd8b1b4e07b8c9") |
| to_pil | Ran | Max-Fu/icrt/icrt/models/backbones/preprocess.py code served (permissive licence) · get_code("7823943fccca7f70") |
| undo_vision_transform | Ran | Max-Fu/icrt/icrt/models/policy/icrt_wrapper.py code served (permissive licence) · get_code("101148492ee6fc64") |
| unnormalize_fn | Ran | Max-Fu/icrt/icrt/models/backbones/preprocess.py code served (permissive licence) · get_code("9a73cbb66f31231c") |
| get_2d_sincos_pos_embed_from_grid | Not yet run | Max-Fu/icrt/icrt/models/backbones/pos_embed.py code served (permissive licence) · get_code("b7482295ffca3e2c") |
| handle_flash_attn | Not yet run | Max-Fu/icrt/icrt/models/backbones/utils.py code served (permissive licence) · get_code("1a3e01d4552a3b11") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
We explore how to enhance next-token prediction models to perform in-context imitation learning on a real robot, where the robot executes new tasks by interpreting contextual information provided during the input phase, without updating its underlying policy parameters. We propose In-Context Robot Transformer (ICRT), a causal transformer that performs autoregressive prediction on sensorimotor trajectories without relying on any linguistic data or reward function. This formulation enables flexible and training-free execution of new tasks at test time, achieved by prompting the model with sensorimotor trajectories of the new task composing of image observations, actions and states tuples, collected through human teleoperation. Experiments with a Franka Emika robot demonstrate that the ICRT can adapt to new tasks specified by prompts, even in environment configurations that differ from both the prompt and the training data. In a multitask environment setup, ICRT significantly outperforms current state-of-the-art next-token prediction models in robotics on generalizing to unseen tasks. Code, checkpoints and data are available on https://icrt.dev/
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2408.15980")
get_code_for_paper("2408.15980")
have("2408.15980")
Connect an agent — have() is free.