We lifted 11 functions out of this paper's own repositories and ran 7 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| thunlp/llava-uhd | canonical | 7 of 11 |
| Function | Status | Where it lives |
|---|---|---|
| extract_answer | Ran | thunlp/llava-uhd/VLMEvalKit/vlmeval/vlm/llava_qwen2_uhd_v3.py code served (permissive licence) · get_code("a76fe5da7ade58fb") |
| extract_choice | Ran | thunlp/llava-uhd/VLMEvalKit/vlmeval/vlm/llava_qwen2_uhd_v3.py code served (permissive licence) · get_code("46ca4942d73607af") |
| extract_patches | Ran | thunlp/llava-uhd/llava/mm_utils.py code served (permissive licence) · get_code("b0a851cae92754ff") |
| extract_yes_no | Ran | thunlp/llava-uhd/VLMEvalKit/vlmeval/vlm/llava_qwen2_uhd_v3.py code served (permissive licence) · get_code("fb3f8b5f16ed11cb") |
| get_2d_sincos_pos_embed | Ran | thunlp/llava-uhd/llava/model/multimodal_projector/uhd_v1_resampler.py code served (permissive licence) · get_code("77e8a3ac46f3afec") |
| get_abs_pos | Ran | thunlp/llava-uhd/llava/model/multimodal_projector/uhd_v1_resampler.py code served (permissive licence) · get_code("17e90d5853a48fa7") |
| unpad_image | Ran | thunlp/llava-uhd/llava/model/llava_arch.py code served (permissive licence) · get_code("7606525af238fb64") |
| auto_pad_images | Not yet run | thunlp/llava-uhd/llava/mm_utils.py code served (permissive licence) · get_code("3c8823948724e269") |
| get_2d_sincos_pos_embed_from_grid | Not yet run | thunlp/llava-uhd/llava/model/multimodal_projector/uhd_v1_resampler.py code served (permissive licence) · get_code("2884deadbb637c3e") |
| resize_and_center_crop | Not yet run | thunlp/llava-uhd/llava/mm_utils.py code served (permissive licence) · get_code("c80358dff96c25ce") |
| safe_load_tokenizer | Not yet run | thunlp/llava-uhd/llava/conversation.py code served (permissive licence) · get_code("b20705b0dfd77ffc") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Vision transformers (ViTs) are widely employed in multimodal large language models (MLLMs) for visual encoding. However, they exhibit inferior performance on tasks regarding fine-grained visual perception. We attribute this to the limitations of ViTs in capturing diverse multi-modal visual levels, such as low-level details. To address this issue, we present LLaVA-UHD v2, an MLLM with advanced perception abilities by introducing a well-designed vision-language projector, the Hierarchical window (Hiwin) transformer. Hiwin transformer enhances MLLM's ability to capture diverse multi-modal visual granularities, by incorporating our constructed high-resolution semantic pyramid. Specifically, Hiwin transformer comprises two key modules: (i) a visual detail injection module, which progressively injects low-level visual details into high-level language-aligned semantics features, thereby forming an inverse semantic pyramid (ISP), and (ii) a hierarchical window attention module, which leverages cross-scale windows to condense multi-level semantics from the ISP. Extensive experiments show that LLaVA-UHD v2 outperforms compared MLLMs on a wide range of benchmarks. Notably, our design achieves an average boost of 3.7% across 14 benchmarks compared with the baseline method, 9.3% on DocVQA for instance. All the data and code will be publicly available to facilitate future research.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2412.13871")
get_code_for_paper("2412.13871")
have("2412.13871")
Connect an agent — have() is free.