Expected Gradients of Maxout Networks and Consequences to Parameter Initialization

Guido Mont, Hanna Tseran

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. If it links a repository, it is listed below.

Repositories linked to this paper

hanna-tseran/maxout_expected_gradients — marked official in Papers with Code; named in the paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

We study the gradients of a maxout network with respect to inputs and parameters and obtain bounds for the moments depending on the architecture and the parameter distribution. We observe that the distribution of the input-output Jacobian depends on the input, which complicates a stable parameter initialization. Based on the moments of the gradients, we formulate parameter initialization strategies that avoid vanishing and exploding gradients in wide networks. Experiments with deep fully-connected and convolutional networks show that this strategy improves SGD and Adam training of deep maxout networks. In addition, we obtain refined bounds on the expected number of linear regions, results on the expected curve length distortion, and results on the NTK. 3

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2301.06956")
get_code_for_paper("2301.06956")
have("2301.06956")

Connect an agent — have() is free.