SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 1902.06720 · 2019

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent

Lechao Xiao, Jeffrey Pennington, Yasaman Bahri, Jascha Sohl-Dickstein, Roman Novak, Jaehoon Lee, Samuel Schoenholz

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

A longstanding goal in deep learning research has been to precisely characterize training and generalization. However, the often complex loss landscapes of neural networks have made a theory of learning dynamics elusive. In this work, we show that for wide neural networks the learning dynamics simplify considerably and that, in the infinite width limit, they are governed by a linear model obtained from the first-order Taylor expansion of the network around its initial parameters. Furthermore, mirroring the correspondence between wide Bayesian neural networks and Gaussian processes, gradient-based training of wide neural networks with a squared loss produces test set predictions drawn from a Gaussian process with a particular compositional kernel. While these theoretical results are only exact in the infinite width limit, we nevertheless find excellent empirical agreement between the predictions of the original network and those of the linearized version even for finite practically-sized networks. This agreement is robust across different architectures, optimization methods, and loss functions. * Both authors contributed equally to this work. Work done as a member of the Google AI Residency program (https://g.co/airesidency).

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("1902.06720")
get_code_for_paper("1902.06720")
have("1902.06720")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/1902.06720.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/1902.06720.svg)](https://syntology.ai/paper/1902.06720)

Connect an agent — have() is free.