Jugal Kalita, Aaron Serianni
We lifted 5 functions out of this paper's own repositories and ran 0 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| aaronserianni/training-free-nas | canonical | 0 of 5 |
| Function | Status | Where it lives |
|---|---|---|
| create_int_feature | Not yet run | aaronserianni/training-free-nas/tensorflow-model/build_pretraining_dataset.py code served (permissive licence) · get_code("b03d45af664a39e3") |
| dct | Not yet run | aaronserianni/training-free-nas/modeling_electra.py code served (permissive licence) · get_code("0e1b2cabd3823f3f") |
| dct_2d | Not yet run | aaronserianni/training-free-nas/modeling_electra.py code served (permissive licence) · get_code("6cacf2c0ab871ae9") |
| get_softmax_output | Not yet run | aaronserianni/training-free-nas/tensorflow-model/run_pretraining.py code served (permissive licence) · get_code("170c7c6b569e1aa2") |
| secs_to_str | Not yet run | aaronserianni/training-free-nas/tensorflow-model/util/training_utils.py code served (permissive licence) · get_code("9c127cf70c5ee0db") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Neural architecture search (NAS) has allowed for the automatic creation of new and effective neural network architectures, offering an alternative to the laborious process of manually designing complex architectures. However, traditional NAS algorithms are slow and require immense amounts of computing power. Recent research has investigated training-free NAS metrics for image classification architectures, drastically speeding up search algorithms. In this paper, we investigate trainingfree NAS metrics for recurrent neural network (RNN) and BERT-based transformer architectures, targeted towards language modeling tasks. First, we develop a new trainingfree metric, named hidden covariance, that predicts the trained performance of an RNN architecture and significantly outperforms existing training-free metrics. We experimentally evaluate the effectiveness of the hidden covariance metric on the NAS-Bench-NLP benchmark. Second, we find that the current search space paradigm for transformer architectures is not optimized for training-free neural architecture search. Instead, a simple qualitative analysis can effectively shrink the search space to the best performing architectures. This conclusion is based on our investigation of existing training-free metrics and new metrics developed from recent transformer pruning literature, evaluated on our own benchmark of trained BERT architectures. Ultimately, our analysis shows that the architecture search space and the training-free metric must be developed together in order to achieve effective results. Our source code is available at https://github. com/aaronserianni/training-free-nas.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2306.00288")
get_code_for_paper("2306.00288")
have("2306.00288")
Connect an agent — have() is free.