SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 1705.09319 · 2017

Diagonal Rescaling For Neural Networks

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

We define a second-order neural network stochastic gradient training algorithm whose block-diagonal structure effectively amounts to normalizing the unit activations. Investigating why this algorithm lacks in robustness then reveals two interesting insights. The first insight suggests a new way to scale the stepsizes, clarifying popular algorithms such as RMSProp as well as old neural network tricks such as fanin stepsize scaling. The second insight stresses the practical importance of dealing with fast changes of the curvature of the cost.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("1705.09319")
get_code_for_paper("1705.09319")
have("1705.09319")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/1705.09319.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/1705.09319.svg)](https://syntology.ai/paper/1705.09319)

Connect an agent — have() is free.