Pascale Fung, Samuel Cahyawijaya, Zihan Liu, Wenliang Dai
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Existing works on multimodal affective computing tasks, such as emotion recognition, generally adopt a two-phase pipeline, first extracting feature representations for each single modality with hand-crafted algorithms and then performing end-to-end learning with the extracted features. However, the extracted features are fixed and cannot be further fine-tuned on different target tasks, and manually finding feature extraction algorithms does not generalize or scale well to different tasks, which can lead to sub-optimal performance. In this paper, we develop a fully end-to-end model that connects the two phases and optimizes them jointly. In addition, we restructure the current datasets to enable the fully end-to-end training. Furthermore, to reduce the computational overhead brought by the end-to-end model, we introduce a sparse cross-modal attention mechanism for the feature extraction. Experimental results show that our fully end-to-end model significantly surpasses the current state-of-theart models based on the two-phase pipeline. Moreover, by adding the sparse cross-modal attention, our model can maintain performance with around half the computation in the feature extraction part.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2103.09666")
get_code_for_paper("2103.09666")
have("2103.09666")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2103.09666.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2103.09666)
Connect an agent — have() is free.