We lifted 1 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| OpenSparseLLMs/CLIP-MoE | canonical | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| setup_distributed | Ran | OpenSparseLLMs/CLIP-MoE/train/train_moe.py pointer only (licence: NONE) · get_code("9e6bd6ee6006a98f") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Contrastive Language-Image Pre-training (CLIP) has become a cornerstone in multimodal intelligence. However, recent studies discovered that CLIP can only encode one aspect of the feature space, leading to substantial information loss and indistinctive features. To mitigate this issue, this paper introduces a novel strategy that fine-tunes a series of complementary CLIP models and transforms them into a CLIP-MoE. Specifically, we propose a model-agnostic Diversified Multiplet Upcycling (DMU) framework for CLIP. Instead of training multiple CLIP models from scratch, DMU leverages a pre-trained CLIP and fine-tunes it into a diverse set with highly cost-effective multistage contrastive learning, thus capturing distinct feature subspaces efficiently. To fully exploit these fine-tuned models while minimizing computational overhead, we transform them into a CLIP-MoE, which dynamically activates a subset of CLIP experts, achieving an effective balance between model capacity and computational cost. Comprehensive experiments demonstrate the superior performance of CLIP-MoE across various zero-shot retrieval, zero-shot image classification tasks, and downstream Multimodal Large Language Model (MLLM) benchmarks when used as a vision encoder.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2409.19291")
get_code_for_paper("2409.19291")
have("2409.19291")
Connect an agent — have() is free.