Stefano Ermon, Caiming Xiong, Yun Fu, Ning Yu, Chen Xing, Can Qin, Zeyuan Chen, Ran Xu, Shu Zhang
We lifted 2 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| salesforce/gluegen | — | 2 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| Translator | Ran | salesforce/gluegen/stable-diffusion/ldm/modules/encoders/translators.py code served (permissive licence) · get_code("41ac69f43d02b2be") |
| translator_base | Ran | salesforce/gluegen/stable-diffusion/ldm/modules/encoders/translators.py code served (permissive licence) · get_code("9c19abef45eb8a7c") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Text-to-image (T2I) models based on diffusion processes have achieved remarkable success in controllable image generation using user-provided captions. However, the tight coupling between the current text encoder and image decoder in T2I models makes it challenging to replace or upgrade. Such changes often require massive fine-tuning or even training from scratch with the prohibitive expense. To address this problem, we propose GlueGen, which applies a newly proposed GlueNet model to align features from single-modal or multi-modal encoders with the latent space of an existing T2I model. The approach introduces a new training objective that leverages parallel corpora to align the representation spaces of different encoders. Empirical results show that GlueNet can be trained efficiently and enables various capabilities beyond previous state-of-the-art models: 1) multilingual language models such as XLM-Roberta can be aligned with existing T2I models, allowing for the generation of high-quality images from captions beyond English; 2) GlueNet can align multi-modal encoders such as AudioCLIP with the Stable Diffusion model, enabling sound-to-image generation; 3) it can also upgrade the current text encoder of the latent diffusion model for challenging case generation. By the alignment of various feature representations, the GlueNet allows for flexible and efficient integration of new functionality into existing T2I models and sheds light on X-to-image (X2I) generation. 1
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2303.10056")
get_code_for_paper("2303.10056")
have("2303.10056")
Connect an agent — have() is free.