SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2407.00247 · ACL · 2024

Prompt Refinement with Image Pivot for Text-to-Image Generation

Yingwei Pan, Ting Yao, Tao Mei, Qingyao Ai, Yiqun Liu, Jiaxin Mao, Shaoping Ma, Jingtao Zhan

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

For text-to-image generation, automatically refining user-provided natural language prompts into the keyword-enriched prompts favored by systems is essential for the user experience. Such a prompt refinement process is analogous to translating the prompt from "user languages" into "system languages". However, the scarcity of such parallel corpora makes it difficult to train a prompt refinement model. Inspired by zero-shot machine translation techniques, we introduce Prompt Refinement with Image Pivot (PRIP). PRIP innovatively uses the latent representation of a user-preferred image as an intermediary "pivot" between the user and system languages. It decomposes the refinement process into two data-rich tasks: inferring representations of user-preferred images from user languages and subsequently translating image representations into system languages. Thus, it can leverage abundant data for training. Extensive experiments show that PRIP substantially outperforms a wide range of baselines and effectively transfers to unseen systems in a zero-shot manner 1 .

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2407.00247")
get_code_for_paper("2407.00247")
have("2407.00247")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2407.00247.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2407.00247.svg)](https://syntology.ai/paper/2407.00247)

Connect an agent — have() is free.