We lifted 2 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| facebookresearch/sieve | canonical | 2 of 2 |
| Function | Status | Where it lives |
|---|---|---|
| read_parquet | Ran | facebookresearch/sieve/sentence_similarity_inference.py code served (permissive licence) · get_code("91846b837e240a2a") |
| remove_phrases | Ran | facebookresearch/sieve/sentence_similarity_inference.py code served (permissive licence) · get_code("6f989626dd6209f2") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Vision-Language Models (VLMs) are pretrained on large, diverse, and noisy web-crawled datasets. This underscores the critical need for dataset pruning, as the quality of these datasets is strongly correlated with the performance of VLMs on downstream tasks. Using CLIPScore from a pretrained model to only train models using highly-aligned samples is one of the most successful methods for pruning. We argue that this approach suffers from multiple limitations including: false positives and negatives due to CLIP's pretraining on noisy labels. We propose a pruning signal, Sieve, that employs synthetic captions generated by image-captioning models pretrained on small, diverse, and well-aligned image-text pairs to evaluate the alignment of noisy image-text pairs. To bridge the gap between the limited diversity of generated captions and the high diversity of alternative text (alt-text), we estimate the semantic textual similarity in the embedding space of a language model pretrained on unlabeled text corpus. Using DataComp, a multimodal dataset filtering benchmark, when evaluating on 38 downstream tasks, our pruning approach, surpasses CLIPScore by 2.6\% and 1.7\% on medium and large scale respectively. In addition, on retrieval tasks, Sieve leads to a significant improvement of 2.7% and 4.5% on medium and large scale respectively.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2310.02110")
get_code_for_paper("2310.02110")
have("2310.02110")
Connect an agent — have() is free.