Siyuan Huang, Xiaojian Ma, Baoxiong Jia, Jiangyong Huang, Xiongkun Linghu, Xuesong Niu
We lifted 3 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| MSR3D/MSR3D | canonical | 2 of 2 |
| copy not recorded | — | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| disabled_train | Ran | this paper's copy was not recorded; identical code first harvested from zhouc20/LatentGraphDiffusion pointer only · get_code("4cb732f513d69dfd") |
| extract_number | Ran | MSR3D/MSR3D/evaluator/evaluate_msqa.py code served (permissive licence) · get_code("64da74993d3cde5c") |
| extract_question | Ran | MSR3D/MSR3D/evaluator/evaluate_msqa.py code served (permissive licence) · get_code("88bd34c3dd6b90a7") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Situation awareness is essential for understanding and reasoning about 3D scenes in embodied AI agents. However, existing datasets and benchmarks for situated understanding are limited in data modality, diversity, scale, and task scope. To address these limitations, we propose Multi-modal Situated Question Answering (MSQA), a large-scale multi-modal situated reasoning dataset, scalably collected leveraging 3D scene graphs and vision-language models (VLMs) across a diverse range of real-world 3D scenes. MSQA includes 251K situated question-answering pairs across 9 distinct question categories, covering complex scenarios within 3D scenes. We introduce a novel interleaved multi-modal input setting in our benchmark to provide text, image, and point cloud for situation and question description, resolving ambiguity in previous single-modality convention (e.g., text). Additionally, we devise the Multi-modal Situated Next-step Navigation (MSNN) benchmark to evaluate models' situated reasoning for navigation. Comprehensive evaluations on MSQA and MSNN highlight the limitations of existing vision-language models and underscore the importance of handling multi-modal interleaved inputs and situation modeling. Experiments on data scaling and cross-domain transfer further demonstrate the efficacy of leveraging MSQA as a pre-training dataset for developing more powerful situated reasoning models.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2409.02389")
get_code_for_paper("2409.02389")
have("2409.02389")
Connect an agent — have() is free.