SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 1812.01216 · 2018

Parameter Re-Initialization through Cyclical Batch Size Schedules

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.

Abstract

Optimal parameter initialization remains a crucial problem for neural network training. A poor weight initialization may take longer to train and/or converge to sub-optimal solutions. Here, we propose a method of weight re-initialization by repeated annealing and injection of noise in the training process. We implement this through a cyclical batch size schedule motivated by a Bayesian perspective of neural network training. We evaluate our methods through extensive experiments on tasks in language modeling, natural language inference, and image classification. We demonstrate the ability of our method to improve language modeling performance by up to 7.91 perplexity and reduce training iterations by up to $61\%$, in addition to its flexibility in enabling snapshot ensembling and use with adversarial training.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("1812.01216")
get_code_for_paper("1812.01216")
have("1812.01216")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/1812.01216.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/1812.01216.svg)](https://syntology.ai/paper/1812.01216)

Connect an agent — have() is free.