We lifted 9 functions out of this paper's own repositories and ran 4 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| dsgissin/Incremental-Learning | canonical | 4 of 9 |
| Function | Status | Where it lives |
|---|---|---|
| OMP | Ran | dsgissin/Incremental-Learning/OMP_experiment.py code served (permissive licence) · get_code("a4d5376b7b891798") |
| check_sets | Ran | dsgissin/Incremental-Learning/OMP_experiment.py code served (permissive licence) · get_code("aa7f2a54f01253aa") |
| get_canonical_model | Ran | dsgissin/Incremental-Learning/classification_experiment.py code served (permissive licence) · get_code("8990c6012616022d") |
| get_sorted_eig | Ran | dsgissin/Incremental-Learning/quadratic_network_experiment.py code served (permissive licence) · get_code("7ff75ea941baae2a") |
| exp_loss | Not yet run | dsgissin/Incremental-Learning/classification_experiment.py code served (permissive licence) · get_code("27a76f532debd141") |
| exponential_acc | Not yet run | dsgissin/Incremental-Learning/convolution_experiment.py code served (permissive licence) · get_code("46d7d64c9751bd24") |
| get_canonical_model | Not yet run | dsgissin/Incremental-Learning/convolution_experiment.py code served (permissive licence) · get_code("0c2b992c73e12273") |
| get_canonical_model | Not yet run | dsgissin/Incremental-Learning/matrix_sensing_experiment.py code served (permissive licence) · get_code("1ff3248281e7f39e") |
| run_toy_model | Not yet run | dsgissin/Incremental-Learning/OMP_experiment.py code served (permissive licence) · get_code("928ad3d19e0e28c9") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
A leading hypothesis for the surprising generalization of neural networks is that the dynamics of gradient descent bias the model towards simple solutions, by searching through the solution space in an incremental order of complexity. We formally define the notion of incremental learning dynamics and derive the conditions on depth and initialization for which this phenomenon arises in deep linear models. Our main theoretical contribution is a dynamical depth separation result, proving that while shallow models can exhibit incremental learning dynamics, they require the initialization to be exponentially small for these dynamics to present themselves. However, once the model becomes deeper, the dependence becomes polynomial and incremental learning can arise in more natural settings. We complement our theoretical findings by experimenting with deep matrix sensing, quadratic neural networks and with binary classification using diagonal and convolutional linear networks, showing all of these models exhibit incremental learning.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("1909.12051")
get_code_for_paper("1909.12051")
have("1909.12051")
Connect an agent — have() is free.