Bhiksha Raj, Khoa Luu, Thanh-Dat Truong, Ngan Le, Hoang Pham, Chi Nhan Duong, The De Vu
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Turn-taking has played an essential role in structuring the regulation of a conversation. The task of identifying the main speaker (who is properly taking his/her turn of speaking) and the interrupters (who are interrupting or reacting to the main speaker's utterances) remains a challenging task. Although some prior methods have partially addressed this task, there still remain some limitations. Firstly, a direct association of Audio and Visual features may limit the correlations to be extracted due to different modalities. Secondly, the relationship across temporal segments helping to maintain the consistency of localization, separation and conversation contexts is not effectively exploited. Finally, the interactions between speakers that usually contain the tracking and anticipatory decisions about transition to a new speaker is usually ignored. Therefore, this work introduces a new Audio-Visual Transformer approach to the problem of localization and highlighting the main speaker in both audio and visual channels of a multi-speaker conversation video in the wild. The proposed method exploits different types of correlations presented in both visual and audio signals. The temporal audio-visual relationships across spatial-temporal space are anticipated and optimized via the self-attention mechanism in a Transformer structure. Moreover, a newly collected dataset is introduced for the main speaker detection. To the best of our knowledge, it is one of the first studies that is able to automatically localize and highlight the main speaker in both visual and audio channels in multi-speaker conversation videos. * The dataset and implementation will be publicly available at https://github.com/uark-cviu/Right2Talk
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2108.03256")
get_code_for_paper("2108.03256")
have("2108.03256")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2108.03256.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2108.03256)
Connect an agent — have() is free.