Qing Li, Siyuan Huang, Yixin Chen, Xiaojian Ma, Ziyu Zhu, Zhidong Deng
We lifted 6 functions out of this paper's own repositories and ran 4 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| 3d-vista/3D-VisTA | — | 4 of 6 |
| Function | Status | Where it lives |
|---|---|---|
| Registry | Ran | 3d-vista/3D-VisTA/model/vision/unified_encoder.py code served (permissive licence) · get_code("af392075ecc51271") |
| TransformerEncoderLayer | Ran | 3d-vista/3D-VisTA/model/vision/unified_encoder.py code served (permissive licence) · get_code("8f3dbe47a6eb7b40") |
| generate_causal_mask | Ran | 3d-vista/3D-VisTA/model/vision/unified_encoder.py code served (permissive licence) · get_code("ef7878a2c111e5e2") |
| generate_mm_casual_mask | Ran | 3d-vista/3D-VisTA/model/vision/unified_encoder.py code served (permissive licence) · get_code("e4b5ceca275ea601") |
| UnifiedSpatialCrossEncoderV2 | Not yet run | 3d-vista/3D-VisTA/model/vision/unified_encoder.py code served (permissive licence) · get_code("5a6f1742cb13f1fc") |
| init_weights | Not yet run | 3d-vista/3D-VisTA/model/vision/unified_encoder.py code served (permissive licence) · get_code("6c816339122f69e9") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
3D vision-language grounding (3D-VL) is an emerging field that aims to connect the 3D physical world with natural language, which is crucial for achieving embodied intelligence. Current 3D-VL models rely heavily on sophisticated modules, auxiliary losses, and optimization tricks, which calls for a simple and unified model. In this paper, we propose 3D-VisTA, a pre-trained Transformer for 3D Vision and Text Alignment that can be easily adapted to various downstream tasks. 3D-VisTA simply utilizes self-attention layers for both single-modal modeling and multi-modal fusion without any sophisticated task-specific design. To further enhance its performance on 3D-VL tasks, we construct ScanScribe, the first large-scale 3D scene-text pairs dataset for 3D-VL pre-training. ScanScribe contains 2,995 RGB-D scans for 1,185 unique indoor scenes originating from ScanNet and 3R-Scan datasets, along with paired 278K scene descriptions generated from existing 3D-VL tasks, templates, and GPT-3. 3D-VisTA is pre-trained on ScanScribe via masked language/object modeling and scene-text matching. It achieves state-of-the-art results on various 3D-VL tasks, ranging from visual grounding and dense captioning to question answering and situated reasoning. Moreover, 3D-VisTA demonstrates superior data efficiency, obtaining strong performance even with limited annotations during downstream task fine-tuning.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2308.04352")
get_code_for_paper("2308.04352")
have("2308.04352")
Connect an agent — have() is free.