SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2301.02280 · CVPR · 2023

Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training

Abhimanyu Dubey, Simon Vandenhende, Filip Radenovic, Dhruv Mahajan, Yash Patel, Abhishek Kadian, Todor Mihaylov, Vignesh Ramanathan, Yi Wen, In Ctu, Prague, Swag-Vit

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Vision-language models trained with contrastive learning on large-scale noisy data are becoming increasingly popular for zero-shot recognition problems. In this paper we improve the following three aspects of the contrastive pre-training pipeline: dataset noise, model initialization and the training objective. First, we propose a straightforward filtering strategy titled Complexity, Action, and Textspotting (CAT) that significantly reduces dataset size, while achieving improved performance across zero-shot visionlanguage tasks. Next, we propose an approach titled Concept Distillation to leverage strong unimodal representations for contrastive training that does not increase training complexity while outperforming prior work. Finally, we modify the traditional contrastive alignment objective, and propose an importance-sampling approach to up-sample the importance of hard-negatives without adding additional complexity. On an extensive zero-shot benchmark of 29 tasks, our Distilled and Hard-negative Training (DiHT) approach improves on 20 tasks compared to the baseline. Furthermore, for few-shot linear probing, we propose a novel approach that bridges the gap between zero-shot and fewshot performance, substantially improving over prior work. Models are available at github.com/facebookresearch/diht.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2301.02280")
get_code_for_paper("2301.02280")
have("2301.02280")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2301.02280.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2301.02280.svg)](https://syntology.ai/paper/2301.02280)

Connect an agent — have() is free.