We lifted 1 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| zhmeishi/trade-off_contrastive_learning | canonical | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| get_save_folder | Ran | zhmeishi/trade-off_contrastive_learning/finetune_eval.py code served (permissive licence) · get_code("eb55591bd9a10ff5") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Pre-training representations (a.k.a. foundation models) has recently become a prevalent learning paradigm, where one first pre-trains a representation using large-scale unlabeled data, and then learns simple predictors on top of the representation using small labeled data from the downstream tasks. There are two key desiderata for the representation: label efficiency (the ability to learn an accurate classifier on top of the representation with a small amount of labeled data) and universality (usefulness across a wide range of downstream tasks). In this paper, we focus on one of the most popular instantiations of this paradigm: contrastive learning with linear probing, i.e., learning a linear predictor on the representation pre-trained by contrastive learning. We show that there exists a trade-off between the two desiderata so that one may not be able to achieve both simultaneously. Specifically, we provide analysis using a theoretical data model and show that, while more diverse pre-training data result in more diverse features for different tasks (improving universality), it puts less emphasis on task-specific features, giving rise to larger sample complexity for down-stream supervised tasks, and thus worse prediction performance. Guided by this analysis, we propose a contrastive regularization method to improve the trade-off. We validate our analysis and method empirically with systematic experiments using real-world datasets and foundation models.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2303.00106")
get_code_for_paper("2303.00106")
have("2303.00106")
Connect an agent — have() is free.