Mit, Karren Yang, Dejan Marković, Steven Krenn, Alexander Richard, Vasu Agrawal
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Since facial actions such as lip movements contain significant information about speech content, it is not surprising that audio-visual speech enhancement methods are more accurate than their audio-only counterparts. Yet, state-of-the-art approaches still struggle to generate clean, realistic speech without noise artifacts and unnatural distortions in challenging acoustic environments. In this paper, we propose a novel audio-visual speech enhancement framework for high-fidelity telecommunications in AR/VR. Our approach leverages audio-visual speech cues to generate the codes of a neural speech codec, enabling efficient synthesis of clean, realistic speech from noisy signals. Given the importance of speaker-specific cues in speech, we focus on developing personalized models that work well for individual speakers. We demonstrate the efficacy of our approach on a new audio-visual speech dataset collected in an unconstrained, large vocabulary setting, as well as existing audio-visual datasets, outperforming speech enhancement baselines on both quantitative metrics and human evaluation studies. Please see the supplemental video for qualitative results 1 .
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2203.17263")
get_code_for_paper("2203.17263")
have("2203.17263")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2203.17263.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2203.17263)
Connect an agent — have() is free.