We lifted 6 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| zengyan-97/x2-vlm | canonical | 2 of 5 |
| copy not recorded | — | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| build_mlp | Ran | this paper's copy was not recorded; identical code first harvested from zengyan-97/X-VLM pointer only · get_code("51e7514fd736cca3") |
| get_acc | Ran | zengyan-97/x2-vlm/XGQA.py pointer only (licence: BSD-3-Clause) · get_code("10aeef229798e5ff") |
| itm_eval | Ran | zengyan-97/x2-vlm/Retrieval.py pointer only (licence: BSD-3-Clause) · get_code("c7f85279d772ea19") |
| reinit_scheduler_properties_mysched | Not yet run | zengyan-97/x2-vlm/Pretrain.py pointer only (licence: BSD-3-Clause) · get_code("e3d6fbf640a143ee") |
| run_image_iter | Not yet run | zengyan-97/x2-vlm/Pretrain.py pointer only (licence: BSD-3-Clause) · get_code("3365d5bf2acc203f") |
| run_region_iter | Not yet run | zengyan-97/x2-vlm/Pretrain.py pointer only (licence: BSD-3-Clause) · get_code("ee513e1972c86c60") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Vision language pre-training aims to learn alignments between vision and language from a large amount of data. Most existing methods only learn image-text alignments. Some others utilize pre-trained object detectors to leverage vision language alignments at the object level. In this paper, we propose to learn multi-grained vision language alignments by a unified pre-training framework that learns multi-grained aligning and multi-grained localization simultaneously. Based on it, we present X$^2$-VLM, an all-in-one model with a flexible modular architecture, in which we further unify image-text pre-training and video-text pre-training in one model. X$^2$-VLM is able to learn unlimited visual concepts associated with diverse text descriptions. Experiment results show that X$^2$-VLM performs the best on base and large scale for both image-text and video-text tasks, making a good trade-off between performance and model scale. Moreover, we show that the modular design of X$^2$-VLM results in high transferability for it to be utilized in any language or domain. For example, by simply replacing the text encoder with XLM-R, X$^2$-VLM outperforms state-of-the-art multilingual multi-modal pre-trained models without any multilingual pre-training. The code and pre-trained models are available at https://github.com/zengyan-97/X2-VLM.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2211.12402")
get_code_for_paper("2211.12402")
have("2211.12402")
Connect an agent — have() is free.