We lifted 6 functions out of this paper's own repositories and ran 3 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| pierremarion23/scaling-resnets | canonical | 3 of 6 |
| Function | Status | Where it lives |
|---|---|---|
| create_linear_layer | Ran | pierremarion23/scaling-resnets/models.py code served (permissive licence) · get_code("48e7d979e32c6a3a") |
| get_eval_loss | Ran | pierremarion23/scaling-resnets/utils.py code served (permissive licence) · get_code("353b14393930b737") |
| get_results | Ran | pierremarion23/scaling-resnets/training.py code served (permissive licence) · get_code("d7d3afe051d48031") |
| get_prediction | Not yet run | pierremarion23/scaling-resnets/utils.py code served (permissive licence) · get_code("b08031160ce9f707") |
| get_true_targets_predictions | Not yet run | pierremarion23/scaling-resnets/utils.py code served (permissive licence) · get_code("e860988c364ea8fb") |
| load_dataset | Not yet run | pierremarion23/scaling-resnets/data.py code served (permissive licence) · get_code("2a81986a3f3895ed") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Deep ResNets are recognized for achieving state-of-the-art results in complex machine learning tasks. However, the remarkable performance of these architectures relies on a training procedure that needs to be carefully crafted to avoid vanishing or exploding gradients, particularly as the depth $L$ increases. No consensus has been reached on how to mitigate this issue, although a widely discussed strategy consists in scaling the output of each layer by a factor $α_L$. We show in a probabilistic setting that with standard i.i.d.~initializations, the only non-trivial dynamics is for $α_L = \frac{1}{\sqrt{L}}$; other choices lead either to explosion or to identity mapping. This scaling factor corresponds in the continuous-time limit to a neural stochastic differential equation, contrarily to a widespread interpretation that deep ResNets are discretizations of neural ordinary differential equations. By contrast, in the latter regime, stability is obtained with specific correlated initializations and $α_L = \frac{1}{L}$. Our analysis suggests a strong interplay between scaling and regularity of the weights as a function of the layer index. Finally, in a series of experiments, we exhibit a continuous range of regimes driven by these two parameters, which jointly impact performance before and after training.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2206.06929")
get_code_for_paper("2206.06929")
have("2206.06929")
Connect an agent — have() is free.