We lifted 10 functions out of this paper's own repositories and ran 7 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| clip-vil/CLIP-ViL | canonical | 0 of 1 |
| jianjieluo/openai-clip-feature | pwc_unofficial | 7 of 9 |
| Function | Status | Where it lives |
|---|---|---|
| basic_clean | Ran | jianjieluo/openai-clip-feature/clip/simple_tokenizer.py pointer only (licence: MIT) · get_code("98f385d847636a3e") |
| custom_path2id | Ran | jianjieluo/openai-clip-feature/clip_visual_feats.py pointer only (licence: MIT) · get_code("012bdd69e64bf2d1") |
| get_pairs | Ran | jianjieluo/openai-clip-feature/clip/simple_tokenizer.py pointer only (licence: MIT) · get_code("d919ae32e5e4e616") |
| load_json | Ran | jianjieluo/openai-clip-feature/basic_utils.py pointer only (licence: MIT) · get_code("2d946250dd2f5a4f") |
| load_jsonl | Ran | jianjieluo/openai-clip-feature/basic_utils.py pointer only (licence: MIT) · get_code("f1d1cccccf038785") |
| load_pickle | Ran | jianjieluo/openai-clip-feature/basic_utils.py pointer only (licence: MIT) · get_code("570ad34bd1af44a8") |
| whitespace_clean | Ran | jianjieluo/openai-clip-feature/clip/simple_tokenizer.py pointer only (licence: MIT) · get_code("9542161e9640b858") |
| build_model | Not yet run | clip-vil/CLIP-ViL/CLIP-ViL-Direct/vqa/clip/model.py code served (permissive licence) · get_code("0a59bad0d461993c") |
| build_model | Not yet run | jianjieluo/openai-clip-feature/clip/model.py pointer only (licence: MIT) · get_code("25a074ccf6e3f98a") |
| load | Not yet run | jianjieluo/openai-clip-feature/clip/clip.py pointer only (licence: MIT) · get_code("2da7ec0975be872a") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Most existing Vision-and-Language (V&L) models rely on pre-trained visual encoders, using a relatively small set of manually-annotated data (as compared to web-crawled data), to perceive the visual world. However, it has been observed that large-scale pretraining usually can result in better generalization performance, e.g., CLIP (Contrastive Language-Image Pre-training), trained on a massive amount of image-caption pairs, has shown a strong zero-shot capability on various vision tasks. To further study the advantage brought by CLIP, we propose to use CLIP as the visual encoder in various V&L models in two typical scenarios: 1) plugging CLIP into task-specific fine-tuning; 2) combining CLIP with V&L pre-training and transferring to downstream tasks. We show that CLIP significantly outperforms widely-used visual encoders trained with in-domain annotated data, such as BottomUp-TopDown. We achieve competitive or better results on diverse V&L tasks, while establishing new state-of-the-art results on Visual Question Answering, Visual Entailment, and V&L Navigation tasks. We release our code at https://github.com/clip-vil/CLIP-ViL.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2107.06383")
get_code_for_paper("2107.06383")
have("2107.06383")
Connect an agent — have() is free.