SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2206.00832 · NeurIPS · 2022

Fast Benchmarking of Accuracy vs. Training Time with Cyclic Learning Rates

Jonathan Frankle, Cory Stephenson, Davis Blalock, Jacob Portes, Mosaicml Mosaicml

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Benchmarking the tradeoff between neural network accuracy and training time is computationally expensive. Here we show how a multiplicative cyclic learning rate schedule can be used to construct a tradeoff curve in a single training run. We generate cyclic tradeoff curves for combinations of training methods such as Blurpool, Channels Last, Label Smoothing and MixUp, and highlight how these cyclic tradeoff curves can be used to efficiently evaluate the effects of algorithmic choices on network training. In order to make meaningful improvements in neural network training efficiency, ML practitioners must be able to compare between different choices of network architectures, hyperparameters, and training algorithms. One straightforward way to do this is to characterize the tradeoff between accuracy and training time with a "tradeoff curve." Tradeoff curves can be generated by varying the length of training for each model configuration; longer training runs take more time but tend to reach higher quality (Figure 1C). For a fixed model and task configuration, this method of generating tradeoff curves is an estimate of the theoretical Pareto frontier, i.e. the set of all of the best possible tradeoffs between training time and accuracy, where any further attempt to improve one of these metrics worsens the other. 2 * Corresponding author. Work done during a research internship at MosaicML. Code available at https: //github.com/jacobfulano/cyclic-learning-rate-schedules 2 We choose to use the language of "tradeoff curve" instead of Pareto frontier with the understanding that the tradeoff curve is likely a lower bound of the true Pareto frontier. 3 We refer here to the Channels Last memory format in PyTorch https://pytorch.org/blog/ tensor-memory-format-matters/ Has it Trained Yet? Workshop at the Conference on Neural Information Processing Systems (NeurIPS 2022).

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2206.00832")
get_code_for_paper("2206.00832")
have("2206.00832")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2206.00832.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2206.00832.svg)](https://syntology.ai/paper/2206.00832)

Connect an agent — have() is free.