We lifted 3 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| luosiallen/Diff-Foley | canonical | 0 of 1 |
| copy not recorded | — | 2 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| convert_to_custom_text_state_dict | Ran | this paper's copy was not recorded; identical code first harvested from deep-real/dcp pointer only · get_code("f19962ebb134b3d7") |
| get_cast_dtype | Ran | this paper's copy was not recorded; identical code first harvested from zsxkib/PuLID pointer only · get_code("dcd422d66b0581d8") |
| build_model_from_openai_state_dict | Not yet run | luosiallen/Diff-Foley/training/open_cavp_main/src/open_clip/model.py code served (permissive licence) · get_code("e4c056f11a6a6acf") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The Video-to-Audio (V2A) model has recently gained attention for its practical application in generating audio directly from silent videos, particularly in video/film production. However, previous methods in V2A have limited generation quality in terms of temporal synchronization and audio-visual relevance. We present Diff-Foley, a synchronized Video-to-Audio synthesis method with a latent diffusion model (LDM) that generates high-quality audio with improved synchronization and audio-visual relevance. We adopt contrastive audio-visual pretraining (CAVP) to learn more temporally and semantically aligned features, then train an LDM with CAVP-aligned visual features on spectrogram latent space. The CAVP-aligned features enable LDM to capture the subtler audio-visual correlation via a cross-attention module. We further significantly improve sample quality with `double guidance'. Diff-Foley achieves state-of-the-art V2A performance on current large scale V2A dataset. Furthermore, we demonstrate Diff-Foley practical applicability and generalization capabilities via downstream finetuning. Project Page: see https://diff-foley.github.io/
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2306.17203")
get_code_for_paper("2306.17203")
have("2306.17203")
Connect an agent — have() is free.