Jing Liu, Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu
We lifted 42 functions out of this paper's own repositories and ran 15 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| TXH-mercury/VALOR | canonical | 9 of 23 |
| TXH-mercury/VAST | canonical | 1 of 11 |
| txh-mercury/vast | — | 5 of 8 |
| Function | Status | Where it lives |
|---|---|---|
| swish | Ran | TXH-mercury/VALOR/model/bert.py code served (permissive licence) · get_code("0f786c407fb1ee4c") |
| Contra_head | Ran | txh-mercury/vast/model/vast.py code served (permissive licence) · get_code("d0dc6ea5061f7dc5") |
| GELU | Ran | txh-mercury/vast/model/vast.py code served (permissive licence) · get_code("48d525c2e5cbc84a") |
| GatherLayer | Ran | txh-mercury/vast/model/vast.py code served (permissive licence) · get_code("a839c0c942f1b694") |
| Match_head | Ran | txh-mercury/vast/model/vast.py code served (permissive licence) · get_code("2e0587cc12b477e2") |
| TokenMasker | Ran | txh-mercury/vast/model/vast.py code served (permissive licence) · get_code("3e58c21b8cf01cc4") |
| basic_clean | Ran | TXH-mercury/VALOR/model/clip_tokenizer.py code served (permissive licence) · get_code("98f385d847636a3e") |
| disabled_train | Ran | TXH-mercury/VAST/model/general_module.py code served (permissive licence) · get_code("4cb732f513d69dfd") |
| drop_path | Ran | TXH-mercury/VALOR/model/videoswin.py code served (permissive licence) · get_code("55120f2026b56aa2") |
| gelu | Ran | TXH-mercury/VALOR/model/transformer.py code served (permissive licence) · get_code("fdc64f4c72036ae4") |
| gelu | Ran | TXH-mercury/VALOR/model/bert.py code served (permissive licence) · get_code("40e9fee2e0b7e278") |
| get_pairs | Ran | TXH-mercury/VALOR/model/clip_tokenizer.py code served (permissive licence) · get_code("d919ae32e5e4e616") |
| trunc_normal_ | Ran | TXH-mercury/VALOR/model/videoswin.py code served (permissive licence) · get_code("7810fed39bf5c241") |
| url_to_filename | Ran | TXH-mercury/VALOR/model/file_utils.py code served (permissive licence) · get_code("af64ec220e8bcdbc") |
| whitespace_clean | Ran | TXH-mercury/VALOR/model/clip_tokenizer.py code served (permissive licence) · get_code("9542161e9640b858") |
| MMGeneralModule | Not yet run | txh-mercury/vast/model/vast.py code served (permissive licence) · get_code("b6fa06654838e4da") |
| VAST | Not yet run | txh-mercury/vast/model/vast.py code served (permissive licence) · get_code("c22a88dfed0bd02e") |
| all_gather_list | Not yet run | TXH-mercury/VAST/utils/distributed.py code served (permissive licence) · get_code("eee800bfe0a47a9f") |
| all_gather_with_grad | Not yet run | txh-mercury/vast/model/vast.py code served (permissive licence) · get_code("0ec9fc2025c16f65") |
| build_optimizer | Not yet run | TXH-mercury/VAST/utils/build_optimizer.py code served (permissive licence) · get_code("cc4ffe92873b4ee6") |
| cached_path | Not yet run | TXH-mercury/VALOR/model/file_utils.py code served (permissive licence) · get_code("349d780dd4a89a37") |
| clean | Not yet run | TXH-mercury/VALOR/inference.py code served (permissive licence) · get_code("67dae5beb14eebd7") |
| clones | Not yet run | TXH-mercury/VALOR/model/transformer.py code served (permissive licence) · get_code("98110ed59394ae02") |
| compute_max_audio_sample_num_for_position_embeddings | Not yet run | TXH-mercury/VAST/utils/args.py code served (permissive licence) · get_code("0122d74913461cf9") |
| compute_max_vision_sample_num_for_position_embeddings | Not yet run | TXH-mercury/VAST/utils/args.py code served (permissive licence) · get_code("30fd4a1a3eedb19c") |
| concat_all_gather | Not yet run | TXH-mercury/VAST/utils/distributed.py code served (permissive licence) · get_code("73cecca9f3575f09") |
| cook_refs | Not yet run | TXH-mercury/VALOR/scorer/bleu_scorer.py code served (permissive licence) · get_code("6c465efd98518ed2") |
| cook_test | Not yet run | TXH-mercury/VALOR/scorer/bleu_scorer.py code served (permissive licence) · get_code("fb961e80ca3db085") |
| execCmd | Not yet run | TXH-mercury/VAST/utils/offline_process_data.py code served (permissive licence) · get_code("77a45c1e515dbe2e") |
| filename_to_url | Not yet run | TXH-mercury/VALOR/model/file_utils.py code served (permissive licence) · get_code("db0ac56aaf6e35e6") |
| get_best_name | Not yet run | TXH-mercury/VAST/utils/pipeline.py code served (permissive licence) · get_code("36075dd106d2bbbe") |
| get_padded_tokens | Not yet run | TXH-mercury/VALOR/inference.py code served (permissive licence) · get_code("b65e5286fa81fca8") |
| load_from_pretrained_dir | Not yet run | TXH-mercury/VAST/utils/build_model.py code served (permissive licence) · get_code("7d297bdebd282cba") |
| load_from_resume | Not yet run | TXH-mercury/VAST/utils/build_model.py code served (permissive licence) · get_code("647f671c01bde501") |
| load_vocab | Not yet run | TXH-mercury/VALOR/model/bert_tokenizer.py code served (permissive licence) · get_code("fb95c4b13cdf89a2") |
| parse_with_config | Not yet run | TXH-mercury/VAST/utils/args.py code served (permissive licence) · get_code("15b65a6a6e8fd954") |
| precook | Not yet run | TXH-mercury/VALOR/scorer/bleu_scorer.py code served (permissive licence) · get_code("aad08f61905b40a5") |
| split | Not yet run | TXH-mercury/VALOR/inference.py code served (permissive licence) · get_code("d92ebd9f3d882489") |
| trans | Not yet run | TXH-mercury/VALOR/model/modeling.py code served (permissive licence) · get_code("8b5247b663a8f1b5") |
| valid | Not yet run | TXH-mercury/VALOR/model/modeling.py code served (permissive licence) · get_code("3df7530a6ef7c69d") |
| whitespace_tokenize | Not yet run | TXH-mercury/VALOR/model/bert_tokenizer.py code served (permissive licence) · get_code("da7295883cd7da14") |
| window_partition | Not yet run | TXH-mercury/VALOR/model/videoswin.py code served (permissive licence) · get_code("ed4b0e420c5e9bde") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Vision and text have been fully explored in contemporary video-text foundational models, while other modalities such as audio and subtitles in videos have not received sufficient attention. In this paper, we resort to establish connections between multi-modality video tracks, including Vision, Audio, and Subtitle, and Text by exploring an automatically generated large-scale omni-modality video caption dataset called VAST-27M. Specifically, we first collect 27 million opendomain video clips and separately train a vision and an audio captioner to generate vision and audio captions. Then, we employ an off-the-shelf Large Language Model (LLM) to integrate the generated captions, together with subtitles and instructional prompts into omni-modality captions. Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various tasks including vision-text, audio-text, and multi-modal video-text tasks (retrieval, captioning and QA). Extensive experiments have been conducted to demonstrate the effectiveness of our proposed VAST-27M corpus and VAST foundation model. VAST achieves 22 new state-of-the-art results on various cross-modality benchmarks. Code, model and dataset will be released at https://github.com/TXH-mercury/VAST.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2305.18500")
get_code_for_paper("2305.18500")
have("2305.18500")
Connect an agent — have() is free.