Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

Mitchell Wortsman, Ali Farhadi, Simon Kornblith, Ari Morcos, Yair Carmon, Gabriel Ilharco, Rebecca Roelofs, Ludwig Schmidt, Samir Gadre, Raphael Gontijo-Lopes, Hongseok Namkoong

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 17 functions out of this paper's own repositories and ran 8 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• mlfoundations/model-soups — marked official in Papers with Code; named in the paper
• Burf/ModelSoups
• facebookresearch/ModelRatatouille
• flowritecom/flow-merge
• shallowlearn/sportsreid
• hwk0702/keras2torch

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

The conventional recipe for maximizing model accuracy is to (1) train multiple models with various hyperparameters and (2) pick the individual model which performs best on a held-out validation set, discarding the remainder. In this paper, we revisit the second step of this procedure in the context of fine-tuning large pre-trained models, where fine-tuned models often appear to lie in a single low error basin. We show that averaging the weights of multiple models finetuned with different hyperparameter configurations often improves accuracy and robustness. Unlike a conventional ensemble, we may average many models without incurring any additional inference or memory costs-we call the results "model soups." When fine-tuning large pre-trained models such as CLIP, ALIGN, and a ViT-G pretrained on JFT, our soup recipe provides significant improvements over the best model in a hyperparameter sweep on ImageNet. The resulting ViT-G model, which attains 90.94% top-1 accuracy on ImageNet, achieved a new state of the art. Furthermore, we show that the model soup approach extends to multiple image classification and natural language processing tasks, improves out-of-distribution performance, and improves zero-shot performance on new downstream tasks. Finally, we analytically relate the performance similarity of weight-averaging and logitensembling to flatness of the loss and confidence of the predictions, and validate this relation empirically. Code is available at https://github. com/mlfoundations/model-soups.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2203.05482")
get_code_for_paper("2203.05482")
have("2203.05482")

Connect an agent — have() is free.