Lior Wolf, Sagie Benaim, Serge Belongie, Idan Schwartz, Vésteinn Snaebjarnarson, Hila Chefer
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Recent advances in text-to-image diffusion models have enabled the generation of diverse and high-quality images. While impressive, the images often fall short of depicting subtle details and are susceptible to errors due to ambiguity in the input text. One way of alleviating these issues is to train diffusion models on class-labeled datasets. This approach has two disadvantages: (i) supervised datasets are generally small compared to large-scale scraped text-image datasets on which text-to-image models are trained, affecting the quality and diversity of the generated images, or (ii) the input is a hard-coded label, as opposed to free-form text, limiting the control over the generated images. In this work, we propose a non-invasive fine-tuning technique that capitalizes on the expressive potential of freeform text while achieving high accuracy through discriminative signals from a pretrained classifier. This is done by iteratively modifying the embedding of an added input token of a text-to-image diffusion model, by steering generated images toward a given target class according to a classifier. Our method is fast compared to prior fine-tuning methods and does not require a collection of in-class images or retraining of a noise-tolerant classifier. We evaluate our method extensively, showing that the generated images are: (i) more accurate and of higher quality than standard diffusion models, (ii) can be used to augment training data in a low-resource setting, and (iii) reveal information about the data used to train the guiding classifier. The code is available at https://github.com/idansc/ discriminative_class_tokens. * Equal contribution. wish to render an image of a clothes 'iron', but could instead be presented with an image of the elemental metal.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2303.17155")
get_code_for_paper("2303.17155")
have("2303.17155")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2303.17155.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2303.17155)
Connect an agent — have() is free.