SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2408.01031 · ECCV · 2024

POA: Pre-training Once for Models of All Sizes

Jian Wang, Lei Yu, Yingying Zhang, Ming Yang, Guo Ye, Jingdong Chen, Xin Guo, Lixiang Ru, Jiangwei Lao, Huimei He

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Large-scale self-supervised pre-training has paved the way for one foundation model to handle many different vision tasks. Most pre-training methodologies train a single model of a certain size at one time. Nevertheless, various computation or storage constraints in realworld scenarios require substantial efforts to develop a series of models with different sizes to deploy. Thus, in this study, we propose a novel tribranch self-supervised training framework, termed as POA (Pre-training Once for All), to tackle this aforementioned issue. Our approach introduces an innovative elastic student branch into a modern self-distillation paradigm. At each pre-training step, we randomly sample a sub-network from the original student to form the elastic student and train all branches in a self-distilling fashion. Once pre-trained, POA allows the extraction of pre-trained models of diverse sizes for downstream tasks. Remarkably, the elastic student facilitates the simultaneous pre-training of multiple models with different sizes, which also acts as an additional ensemble of models of various sizes to enhance representation learning. Extensive experiments, including k-nearest neighbors, linear probing evaluation and assessments on multiple downstream tasks demonstrate the effectiveness and advantages of our POA. It achieves state-of-the-art performance using ViT, Swin Transformer and ResNet backbones, producing around a hundred models with different sizes through a single pre-training session. The code is available at: https://github.com/Qichuzyy/POA.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2408.01031")
get_code_for_paper("2408.01031")
have("2408.01031")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2408.01031.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2408.01031.svg)](https://syntology.ai/paper/2408.01031)

Connect an agent — have() is free.