Yushi Qiu, Reiji Suda
We lifted 1 functions out of this paper's own repositories and ran 0 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| ihsuy/Train-by-Reconnect | — | 0 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| LaPerm | Not yet run | ihsuy/Train-by-Reconnect/train_by_reconnect/LaPerm.py code served (permissive licence) · get_code("2236035030277f4d") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
What makes untrained deep neural networks (DNNs) different from the trained performant ones? By zooming into the weights in well-trained DNNs, we found that it is the location of weights that holds most of the information encoded by the training. Motivated by this observation, we hypothesized that weights in DNNs trained using stochastic gradient-based methods can be separated into two dimensions: the location of weights, and their exact values. To assess our hypothesis, we propose a novel method called lookahead permutation (LaPerm) to train DNNs by reconnecting the weights. We empirically demonstrate LaPerm's versatility while producing extensive evidence to support our hypothesis: when the initial weights are random and dense, our method demonstrates speed and performance similar to or better than that of regular optimizers, e.g., Adam. When the initial weights are random and sparse (many zeros), our method changes the way neurons connect, achieving accuracy comparable to that of a well-trained dense network. When the initial weights share a single value, our method finds a weight agnostic neural network with far-better-than-chance accuracy. 34th Conference on Neural Information Processing Systems (NeurIPS 2020),
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2003.02570")
get_code_for_paper("2003.02570")
have("2003.02570")
Connect an agent — have() is free.