We lifted 9 functions out of this paper's own repositories and ran 6 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| lavi-lab/tg-vid | canonical | 6 of 9 |
| Function | Status | Where it lives |
|---|---|---|
| RandomMaskingGenerator | Ran | lavi-lab/tg-vid/stllm/models/utils.py pointer only (licence: NONE) · get_code("8bd2d829faa2f7b2") |
| apply_rotary_pos_emb | Ran | lavi-lab/tg-vid/stllm/models/modeling_llama_mem.py pointer only (licence: NONE) · get_code("9b4dff79d5e6102c") |
| disabled_train | Ran | lavi-lab/tg-vid/stllm/models/blip2.py pointer only (licence: NONE) · get_code("4cb732f513d69dfd") |
| get_sinusoid_encoding_table | Ran | lavi-lab/tg-vid/stllm/models/utils.py pointer only (licence: NONE) · get_code("da651e3979a18f84") |
| rotate_half | Ran | lavi-lab/tg-vid/stllm/models/modeling_llama_mem.py pointer only (licence: NONE) · get_code("b99eea6376d1e212") |
| tile | Ran | lavi-lab/tg-vid/stllm/models/base_model.py pointer only (licence: NONE) · get_code("cb33571427334815") |
| all_gather_with_grad | Not yet run | lavi-lab/tg-vid/stllm/models/base_model.py pointer only (licence: NONE) · get_code("0ec9fc2025c16f65") |
| create_eva_vit_g | Not yet run | lavi-lab/tg-vid/stllm/models/eva_vit.py pointer only (licence: NONE) · get_code("ce86d2ef6c0e458f") |
| forward | Not yet run | lavi-lab/tg-vid/stllm/models/peft_model.py pointer only (licence: NONE) · get_code("515cc4dd73dc0caf") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Video Large Language Models (Video LLMs) have achieved impressive performance on video-and-language tasks, such as video question answering. However, most existing Video LLMs neglect temporal information in video data, leading to struggles with temporal-aware video understanding. To address this gap, we propose a Time Gating Video LLM (TG-Vid) designed to enhance temporal modeling through a novel Time Gating module (TG). The TG module employs a time gating mechanism on its sub-modules, comprising gating spatial attention, gating temporal attention, and gating MLP. This architecture enables our model to achieve a robust understanding of temporal information within videos. Extensive evaluation of temporal-sensitive video benchmarks (i.e., MVBench, TempCompass, and NExT-QA) demonstrates that our TG-Vid model significantly outperforms the existing Video LLMs. Further, comprehensive ablation studies validate that the performance gains are attributed to the designs of our TG module. Our code is available at https://github.com/LaVi-Lab/TG-Vid.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2410.05714")
get_code_for_paper("2410.05714")
have("2410.05714")
Connect an agent — have() is free.