Yueting Zhuang, Qi Tian, Lingxi Xie, Juncheng Li, Xin He, † Longhui, Long Qian, Linchao Zhu, Siliang Tang
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large-scale vision-language pre-training has shown impressive advances in a wide range of downstream tasks. Existing methods mainly model the cross-modal alignment by the similarity of the global representations of images and texts, or advanced cross-modal attention upon image and text features. However, they fail to explicitly learn the fine-grained semantic alignment between visual regions and textual phrases, as only global image-text alignment information is available. In this paper, we introduce LOUPE , a fine-grained semantically aLigned visiOn-langUage PrE-training framework, which learns fine-grained semantic alignment from the novel perspective of game-theoretic interactions. To efficiently compute the game-theoretic interactions, we further propose an uncertainty-aware neural Shapley interaction learning module. Experiments show that LOUPE achieves stateof-the-art performance on a variety of vision-language tasks. Furthermore, without any object-level human annotations and fine-tuning, LOUPE achieves competitive performance on object detection and visual grounding. More importantly, LOUPE opens a new promising direction of learning fine-grained semantics from largescale raw image-text pairs. The repository of this work is at https://github. com/YYJMJC/LOUPE.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2208.02515")
get_code_for_paper("2208.02515")
have("2208.02515")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2208.02515.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2208.02515)
Connect an agent — have() is free.