We lifted 14 functions out of this paper's own repositories and ran 7 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| opengvlab/videochat-flash | canonical | 5 of 7 |
| opengvlab/internvideo2 | canonical | 2 of 7 |
| Function | Status | Where it lives |
|---|---|---|
| repeat_kv | Ran | opengvlab/videochat-flash/llava-train_videochat/llava/model/language_model/modeling_qwen2_flash.py code served (permissive licence) · get_code("30d7eec482ebf6b1") |
| apply_rotary_pos_emb | Ran | opengvlab/videochat-flash/llava-train_videochat/llava/model/language_model/modeling_qwen2_flash.py code served (permissive licence) · get_code("d61c483a3c2b3156") |
| extract_patches | Ran | opengvlab/videochat-flash/llava-train_videochat/llava/mm_utils.py code served (permissive licence) · get_code("b0a851cae92754ff") |
| find_all_linear_names | Ran | opengvlab/internvideo2/xtuner-train_internvideo2_5/xtuner/model/utils.py code served (permissive licence) · get_code("53281c424cf5891f") |
| packing_collate | Ran | opengvlab/internvideo2/xtuner-train_internvideo2_5/unify_internvl2_train_r16.py code served (permissive licence) · get_code("e6bb171e5f95a784") |
| rotate_half | Ran | opengvlab/videochat-flash/llava-train_videochat/llava/model/language_model/modeling_qwen2_flash.py code served (permissive licence) · get_code("b99eea6376d1e212") |
| unpad_image | Ran | opengvlab/videochat-flash/llava-train_videochat/llava/model/llava_arch.py code served (permissive licence) · get_code("2c7b71210b8a363f") |
| auto_pad_images | Not yet run | opengvlab/videochat-flash/llava-train_videochat/llava/mm_utils.py code served (permissive licence) · get_code("3c8823948724e269") |
| build_model | Not yet run | opengvlab/internvideo2/xtuner-train_internvideo2_5/xtuner/apis/model.py code served (permissive licence) · get_code("dcdc7ef233f3e1e8") |
| convert_state_dict_to_hf | Not yet run | opengvlab/internvideo2/xtuner-train_internvideo2_5/xtuner/model/llava.py code served (permissive licence) · get_code("f8035a195671164a") |
| create_reference_model | Not yet run | opengvlab/internvideo2/xtuner-train_internvideo2_5/xtuner/model/dpo.py code served (permissive licence) · get_code("269c10dd16c744e8") |
| get_peft_model_state_dict | Not yet run | opengvlab/internvideo2/xtuner-train_internvideo2_5/xtuner/model/utils.py code served (permissive licence) · get_code("9356d2ed9b73134b") |
| reduce_mean | Not yet run | opengvlab/internvideo2/xtuner-train_internvideo2_5/xtuner/model/reward.py code served (permissive licence) · get_code("a0a407c2a1c1f7d3") |
| resize_and_center_crop | Not yet run | opengvlab/videochat-flash/llava-train_videochat/llava/mm_utils.py code served (permissive licence) · get_code("c80358dff96c25ce") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large multimodal models (LMMs) have recently emerged as a powerful tool for long video understanding (LVU), prompting the development of standardized LVU benchmarks to evaluate their performance. However, our investigation reveals a rather sober lesson for existing LVU benchmarks. First, most existing benchmarks rely heavily on multiple-choice questions (MCQs), whose evaluation results are inflated due to the possibility of guessing the correct answer; Second, a significant portion of questions in these benchmarks have strong priors to allow models to answer directly without even reading the input video. For example, Gemini-1.5-Pro can achieve over 50\% accuracy given a random frame from a long video on Video-MME. We also observe that increasing the number of frames does not necessarily lead to improvement on existing benchmarks, which is counterintuitive. As a result, the validity and robustness of current LVU benchmarks are undermined, impeding a faithful assessment of LMMs' long-video understanding capability. To tackle this problem, we propose VideoEval-Pro, a realistic LVU benchmark containing questions with open-ended short-answer, which truly require understanding the entire video. VideoEval-Pro assesses both segment-level and full-video understanding through perception and reasoning tasks. By evaluating 21 proprietary and open-source video LMMs, we conclude the following findings: (1) video LMMs show drastic performance ($>$25\%) drops on open-ended questions compared with MCQs; (2) surprisingly, higher MCQ scores do not lead to higher open-ended scores on VideoEval-Pro; (3) compared to other MCQ benchmarks, VideoEval-Pro benefits more from increasing the number of input frames. Our results show that VideoEval-Pro offers a more realistic and reliable measure of long video understanding, providing a clearer view of progress in this domain.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2505.14640")
get_code_for_paper("2505.14640")
have("2505.14640")
Connect an agent — have() is free.