Isabela Albuquerque, Olivia Wiles, Emanuele Bugliarello, Ira Ktena, Anant Nawalgaria, Yasumasa Onoe, Jordi Pont-Tuset, Su Wang, Chuhan Zhang, Ivana Kajić, Pinelopi Papalampidi, Chris Knutsen, and 2 more
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
While text-to-image (T2I) generative models have become ubiquitous, they do not necessarily generate images that align with a given prompt. While many metrics and benchmarks have been proposed to evaluate T2I models for alignment, the impact of the evaluation components (prompt sets, human annotations, evaluation task) has not been systematically measured. We find that looking at only one slice of data, i.e. one set of skills or human annotations, is not enough to obtain stable conclusions that generalise to new conditions or slices when evaluating T2I models or alignment metrics. We address this by introducing an evaluation suite of >100K annotations across four human annotation templates that comprehensively evaluates models' capabilities across a a range of methods for gathering human annotations and comparing models. In particular, we propose (1) a carefully curated set of prompts -Gecko2K; (2) a statistically grounded method of comparing T2I models; and (3) a framework to systematically evaluate metrics under three evaluation tasksmodel ordering, pair-wise instance scoring, point-wise instance scoring. Using this evaluation suite, we compare a wide range of metrics and find that a given metric may do better in one setting but worse in another. As a result, we introduce a new, interpretable auto-eval metric that is consistently better correlated with human ratings than existing ones on our evaluation suite-across different human templates and evaluation settings-and on TIFA160.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2404.16820")
get_code_for_paper("2404.16820")
have("2404.16820")
Connect an agent — have() is free.