Pichao Wang, Xiaohang Sun, Sarah Ibrahimi, Amanmeet Garg, Ashutosh Sanan, Mohamed Omar
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.
Text-to-video retrieval systems have recently made significant progress by utilizing pre-trained models trained on large-scale image-text pairs. However, most of the latest methods primarily focus on the video modality while disregarding the audio signal for this task. Nevertheless, a recent advancement by ECLIPSE has improved long-range text-tovideo retrieval by developing an audiovisual video representation. Nonetheless, the objective of the text-to-video retrieval task is to capture the complementary audio and video information that is pertinent to the text query rather than simply achieving better audio and video alignment. To address this issue, we introduce TEFAL, a TExt-conditioned Feature ALignment method that produces both audio and video representations conditioned on the text query. Instead of using only an audiovisual attention block, which could suppress the audio information relevant to the text query, our approach employs two independent cross-modal attention blocks that enable the text to attend to the audio and video representations separately. Our proposed method's efficacy is demonstrated on four benchmark datasets that include audio: MSR-VTT, LSMDC, VATEX, and Charades, and achieves better than state-of-the-art performance consistently across the four datasets. This is attributed to the additional text-query-conditioned audio representation and the complementary information it adds to the text-queryconditioned video representation. * This work was done while Sarah Ibrahimi and Mohamed Omar were at Amazon Prime Video.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2307.12964")
get_code_for_paper("2307.12964")
have("2307.12964")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2307.12964.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2307.12964)
Connect an agent — have() is free.