We lifted 8 functions out of this paper's own repositories and ran 4 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| RICE-EIC/Early-Bird-Tickets | canonical | 3 of 7 |
| copy not recorded | — | 1 of 1 |
| Function | Status | Where it lives |
|---|---|---|
| Gaussian | Ran | RICE-EIC/Early-Bird-Tickets/filter.py code served (permissive licence) · get_code("f33359b7e60c47c1") |
| conv3x3 | Ran | this paper's copy was not recorded; identical code first harvested from lixiaotong97/DSU pointer only · get_code("fac5364e2f53c6db") |
| filter | Ran | RICE-EIC/Early-Bird-Tickets/filter.py code served (permissive licence) · get_code("8828ea67235a9bc6") |
| rescale | Ran | RICE-EIC/Early-Bird-Tickets/filter.py code served (permissive licence) · get_code("f17e94d8064d5b77") |
| accuracy | Not yet run | RICE-EIC/Early-Bird-Tickets/EVAL_ResNet18_ImageNet.py code served (permissive licence) · get_code("f0c9a29156911331") |
| print_model_param_flops | Not yet run | RICE-EIC/Early-Bird-Tickets/compute_flops.py code served (permissive licence) · get_code("dbcc7f02b0356238") |
| print_model_param_nums | Not yet run | RICE-EIC/Early-Bird-Tickets/compute_flops.py code served (permissive licence) · get_code("929c091d6fd3e526") |
| test | Not yet run | RICE-EIC/Early-Bird-Tickets/main_resnet50.py code served (permissive licence) · get_code("d8a5e585e02729d7") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
(Frankle & Carbin, 2019) shows that there exist winning tickets (small but critical subnetworks) for dense, randomly initialized networks, that can be trained alone to achieve comparable accuracies to the latter in a similar number of iterations. However, the identification of these winning tickets still requires the costly train-prune-retrain process, limiting their practical benefits. In this paper, we discover for the first time that the winning tickets can be identified at the very early training stage, which we term as early-bird (EB) tickets, via low-cost training schemes (e.g., early stopping and low-precision training) at large learning rates. Our finding of EB tickets is consistent with recently reported observations that the key connectivity patterns of neural networks emerge early. Furthermore, we propose a mask distance metric that can be used to identify EB tickets with low computational overhead, without needing to know the true winning tickets that emerge after the full training. Finally, we leverage the existence of EB tickets and the proposed mask distance to develop efficient training methods, which are achieved by first identifying EB tickets via low-cost schemes, and then continuing to train merely the EB tickets towards the target accuracy. Experiments based on various deep networks and datasets validate: 1) the existence of EB tickets, and the effectiveness of mask distance in efficiently identifying them; and 2) that the proposed efficient training via EB tickets can achieve up to 4.7x energy savings while maintaining comparable or even better accuracy, demonstrating a promising and easily adopted method for tackling cost-prohibitive deep network training. Code available at https://github.com/RICE-EIC/Early-Bird-Tickets.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("1909.11957")
get_code_for_paper("1909.11957")
have("1909.11957")
Connect an agent — have() is free.