We lifted 12 functions out of this paper's own repositories and ran 11 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| bytedance/make-an-audio-2 | pwc_unofficial | 11 of 12 |
| Function | Status | Where it lives |
|---|---|---|
| Normalize | Ran | bytedance/make-an-audio-2/ldm/models/autoencoder1d.py code served (permissive licence) · get_code("9fcdaa6e423e8aa7") |
| add_audio_path | Ran | bytedance/make-an-audio-2/wav_evaluation/cal_clap_score.py code served (permissive licence) · get_code("70427d38625cc3ad") |
| cal_score_by_csv | Ran | bytedance/make-an-audio-2/wav_evaluation/cal_clap_score.py code served (permissive licence) · get_code("30048529b8b97c1c") |
| default | Ran | bytedance/make-an-audio-2/ldm/modules/attention.py code served (permissive licence) · get_code("424012cb37b31172") |
| discriminator_loss_mse | Ran | bytedance/make-an-audio-2/ldm/modules/losses_audio/contperceptual.py code served (permissive licence) · get_code("0eb23950574e39a7") |
| exists | Ran | bytedance/make-an-audio-2/ldm/modules/attention.py code served (permissive licence) · get_code("aa5486a3650902d8") |
| hinge_d_loss | Ran | bytedance/make-an-audio-2/ldm/modules/losses_audio/vqperceptual.py code served (permissive licence) · get_code("22a83df891d750c6") |
| hinge_d_loss_with_exemplar_weights | Ran | bytedance/make-an-audio-2/ldm/modules/losses_audio/vqperceptual.py code served (permissive licence) · get_code("af24442fe485c871") |
| nonlinearity | Ran | bytedance/make-an-audio-2/ldm/models/autoencoder1d.py code served (permissive licence) · get_code("3137073275f8c21a") |
| uniq | Ran | bytedance/make-an-audio-2/ldm/modules/attention.py code served (permissive licence) · get_code("9a299fe5ae09e407") |
| vanilla_d_loss | Ran | bytedance/make-an-audio-2/ldm/modules/losses_audio/vqperceptual.py code served (permissive licence) · get_code("263c54290c553ddd") |
| build_csv_from_wavs | Not yet run | bytedance/make-an-audio-2/wav_evaluation/cal_clap_score.py code served (permissive licence) · get_code("425b2c5baac91537") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Large diffusion models have been successful in text-to-audio (T2A) synthesis tasks, but they often suffer from common issues such as semantic misalignment and poor temporal consistency due to limited natural language understanding and data scarcity. Additionally, 2D spatial structures widely used in T2A works lead to unsatisfactory audio quality when generating variable-length audio samples since they do not adequately prioritize temporal information. To address these challenges, we propose Make-an-Audio 2, a latent diffusion-based T2A method that builds on the success of Make-an-Audio. Our approach includes several techniques to improve semantic alignment and temporal consistency: Firstly, we use pre-trained large language models (LLMs) to parse the text into structured <event & order> pairs for better temporal information capture. We also introduce another structured-text encoder to aid in learning semantic alignment during the diffusion denoising process. To improve the performance of variable length generation and enhance the temporal information extraction, we design a feed-forward Transformer-based diffusion denoiser. Finally, we use LLMs to augment and transform a large amount of audio-label data into audio-text datasets to alleviate the problem of scarcity of temporal data. Extensive experiments show that our method outperforms baseline models in both objective and subjective metrics, and achieves significant gains in temporal information understanding, semantic consistency, and sound quality.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2305.18474")
get_code_for_paper("2305.18474")
have("2305.18474")
Connect an agent — have() is free.