Bowen Shi, Wei-Ning Hsu
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
While audio-visual speech models can yield superior performance and robustness compared to audio-only models, their development and adoption are hindered by the lack of labeled and unlabeled audio-visual data and the cost to deploy one model per modality. In this paper, we present u-HuBERT, a self-supervised pre-training framework that can leverage both multimodal and unimodal speech with a unified masked cluster prediction objective. By utilizing modality dropout during pre-training, we demonstrate that a single fine-tuned model can achieve performance on par or better than the state-of-the-art modality-specific models. Moreover, our model fine-tuned only on audio can perform well with audio-visual and visual speech input, achieving zero-shot modality generalization for multiple speech processing tasks. In particular, our single model yields 1.2%/1.4%/27.2% speech recognition word error rate on LRS3 with audio-visual/audio/visual input. 1 1 Codes and models are available at https://github.com/facebookresearch/av_hubert 36th Conference on Neural Information Processing Systems (NeurIPS 2022).
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2207.07036")
get_code_for_paper("2207.07036")
have("2207.07036")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2207.07036.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2207.07036)
Connect an agent — have() is free.