We lifted 3 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| DomhnallBoyle/sv2s | reimplementation | 3 of 3 |
| Function | Status | Where it lives |
|---|---|---|
| generate_speaker_content_mapping | Ran | DomhnallBoyle/sv2s/preprocessor.py pointer only (licence: NONE) · get_code("a12c51887485e8d5") |
| get_speaker_and_content | Ran | DomhnallBoyle/sv2s/preprocessor.py pointer only (licence: NONE) · get_code("e9478d2def2ab086") |
| get_speaker_embedding_video_path | Ran | DomhnallBoyle/sv2s/preprocessor.py pointer only (licence: NONE) · get_code("985fc857e6d3a411") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Video-to-speech synthesis (also known as lip-to-speech) refers to the translation of silent lip movements into the corresponding audio. This task has received an increasing amount of attention due to its self-supervised nature (i.e., can be trained without manual labelling) combined with the ever-growing collection of audio-visual data available online. Despite these strong motivations, contemporary video-to-speech works focus mainly on small- to medium-sized corpora with substantial constraints in both vocabulary and setting. In this work, we introduce a scalable video-to-speech framework consisting of two components: a video-to-spectrogram predictor and a pre-trained neural vocoder, which converts the mel-frequency spectrograms into waveform audio. We achieve state-of-the art results for GRID and considerably outperform previous approaches on LRW. More importantly, by focusing on spectrogram prediction using a simple feedforward model, we can efficiently and effectively scale our method to very large and unconstrained datasets: To the best of our knowledge, we are the first to show intelligible results on the challenging LRS3 dataset.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2205.02058")
get_code_for_paper("2205.02058")
have("2205.02058")
Connect an agent — have() is free.