Kai-Wei Chang, Aditya Grover, Yu Yang, Hritik Bansal, Nishad Singhi
We lifted 3 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| nishadsinghi/cleanclip | — | 1 of 3 |
| Function | Status | Where it lives |
|---|---|---|
| _augment_image | Ran | nishadsinghi/cleanclip/src/data.py code served (permissive licence) · get_code("fabf8a8a0fd3b8e4") |
| ImageCaptionDataset | Not yet run | nishadsinghi/cleanclip/src/data.py code served (permissive licence) · get_code("e7e095e74efcd040") |
| _augment_text | Not yet run | nishadsinghi/cleanclip/src/data.py code served (permissive licence) · get_code("dfc7c7c26c450b84") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Multimodal contrastive pretraining has been used to train multimodal representation models, such as CLIP, on large amounts of paired image-text data. However, previous studies have revealed that such models are vulnerable to backdoor attacks. Specifically, when trained on backdoored examples, CLIP learns spurious correlations between the embedded backdoor trigger and the target label, aligning their representations in the joint embedding space. Injecting even a small number of poisoned examples, such as 75 examples in 3 million pretraining data, can significantly manipulate the model's behavior, making it difficult to detect or unlearn such correlations. To address this issue, we propose CleanCLIP, a finetuning framework that weakens the learned spurious associations introduced by backdoor attacks by independently re-aligning the representations for individual modalities. We demonstrate that unsupervised finetuning using a combination of multimodal contrastive and unimodal self-supervised objectives for individual modalities can significantly reduce the impact of the backdoor attack. Additionally, we show that supervised finetuning on task-specific labeled image data removes the backdoor trigger from the CLIP vision encoder. We show empirically that CleanCLIP maintains model performance on benign examples while erasing a range of backdoor attacks on multimodal contrastive learning. Code and pretrained checkpoints are available at https://github. com/nishadsinghi/CleanCLIP.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2303.03323")
get_code_for_paper("2303.03323")
have("2303.03323")
Connect an agent — have() is free.