Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, Abdelrahman Mohamed
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Video recordings of speech contain correlated audio and visual information, providing a strong signal for speech representation learning from the speaker's lip movements and the produced sound. We introduce Audio-Visual Hidden Unit BERT (AV-HuBERT), a self-supervised representation learning framework for audio-visual speech, which masks multi-stream video input and predicts automatically discovered and iteratively refined multimodal hidden units. AV-HuBERT learns powerful audio-visual speech representation benefiting both lip-reading and automatic speech recognition. On the largest public lip-reading benchmark LRS3 (433 hours), AV-HuBERT achieves 32.5% WER with only 30 hours of labeled data, outperforming the former state-of-the-art approach (33.6%) trained with a thousand times more transcribed video data (31K hours) (Makino et al., 2019). The lip-reading WER is further reduced to 26.9% when using all 433 hours of labeled data from LRS3 and combined with self-training. Using our audio-visual representation on the same benchmark for audio-only speech recognition leads to a 40% relative WER reduction over the state-of-the-art performance (1.3% vs 2.3%). Our code and models are available at https://github.com/ facebookresearch/av_hubert
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2201.02184")
get_code_for_paper("2201.02184")
have("2201.02184")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2201.02184.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2201.02184)
Connect an agent — have() is free.