Jianfeng Gao, Yonatan Bisk, Jianwei Yang
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.
Contrastive learning has been widely used to train transformer-based vision-language models for video-text alignment and multi-modal representation learning. This paper presents a new algorithm called Token-Aware Cascade contrastive learning (TACo) that improves contrastive learning using two novel techniques. The first is the token-aware contrastive loss which is computed by taking into account the syntactic classes of words. This is motivated by the observation that for a video-text pair, the content words in the text, such as nouns and verbs, are more likely to be aligned with the visual contents in the video than the function words. Second, a cascade sampling method is applied to generate a small set of hard negative examples for efficient loss estimation for multi-modal fusion layers. To validate the effectiveness of TACo, in our experiments we finetune pretrained models for a set of downstream tasks including text-video retrieval (YouCook2, MSR-VTT and ActivityNet), video action step localization (CrossTask), video action segmentation (COIN). The results show that our models attain consistent improvements across different experimental settings over previous methods, setting new state-of-the-art on three public text-video retrieval benchmarks of YouCook2, MSR-VTT and ActivityNet.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2108.09980")
get_code_for_paper("2108.09980")
have("2108.09980")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2108.09980.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2108.09980)
Connect an agent — have() is free.