UMIC: An Unreferenced Metric for Image Captioning via Contrastive Learning

Trung Bui, Kyomin Jung, Franck Dernoncourt, Seunghyun Yoon, Hwanhee Lee

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 14 functions out of this paper's own repositories and ran 7 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• hwanheelee1993/UMIC — marked official in Papers with Code; named in the paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Despite the success of various text generation metrics such as BERTScore, it is still difficult to evaluate the image captions without enough reference captions due to the diversity of the descriptions. In this paper, we introduce a new metric UMIC, an Unreferenced Metric for Image Captioning which does not require reference captions to evaluate image captions. Based on Vision-and-Language BERT, we train UMIC to discriminate negative captions via contrastive learning. Also, we observe critical problems of the previous benchmark dataset (i.e., human annotations) on image captioning metric, and introduce a new collection of human annotations on the generated captions. We validate UMIC on four datasets, including our new dataset, and show that UMIC has a higher correlation than all previous metrics that require multiple references. We release the benchmark dataset and pre-trained models to compute the UMIC 1 . 1 https://github.com/hwanheelee1993/UMIC Ref 1: A dog standing in the snow with a stick in its mouth. Ref 2: A little dog holding sticks in its mouth. Candidate: A dog standing on the snow with a dog CIDEr with Ref 1: 3.166 CIDEr with Ref 2: 0.281 Human Judgments : 1.875 out of 5 References -two giraffe standing next to each other in a field. -two giraffes are climbing a hill with mountains in the background. Candidate -three giraffes standing in a field of grass BLEU1: 0.324 ROUGE-L: 0.320 METEOR: 0.173 CIDER: 0.866 SPICE: 0.289 UMIC: 0.352 UMIC / 𝑪 : 0.770 Human: 0.200

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2106.14019")
get_code_for_paper("2106.14019")
have("2106.14019")

Connect an agent — have() is free.