NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search

Xuanyi Dong, Yi Yang

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 20 functions out of this paper's own repositories and ran 12 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• D-X-Y/AutoDL-Projects — marked official in Papers with Code; named in the paper
• D-X-Y/NAS-Bench-201 — marked official in Papers with Code; named in the paper
• D-X-Y/NAS-Projects — marked official in Papers with Code; named in the paper
• Edge-AI-Acceleration-Lab/RBFleX-NAS
• tomomasayamasaki/RBFleX-NAS
• yoichii/nashpobench2api

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Neural architecture search (NAS) has achieved breakthrough success in a great number of applications in the past few years. It could be time to take a step back and analyze the good and bad aspects in the field of NAS. A variety of algorithms search architectures under different search space. These searched architectures are trained using different setups, e.g., hyper-parameters, data augmentation, regularization. This raises a comparability problem when comparing the performance of various NAS algorithms. NAS-Bench-101 has shown success to alleviate this problem. In this work, we propose an extension to NAS-Bench-101: NAS-Bench-201 with a different search space, results on multiple datasets, and more diagnostic information. NAS-Bench-201 has a fixed search space and provides a unified benchmark for almost any up-to-date NAS algorithms. The design of our search space is inspired from the one used in the most popular cell-based searching algorithms, where a cell is represented as a directed acyclic graph. Each edge here is associated with an operation selected from a predefined operation set. For it to be applicable for all NAS algorithms, the search space defined in NAS-Bench-201 includes all possible architectures generated by 4 nodes and 5 associated operation options, which results in 15,625 neural cell candidates in total. The training log using the same setup and the performance for each architecture candidate are provided for three datasets. This allows researchers to avoid unnecessary repetitive training for selected architecture and focus solely on the search algorithm itself. The training time saved for every architecture also largely improves the efficiency of most NAS algorithms and brings a more computational cost friendly NAS community for a broader range of researchers. We provide additional diagnostic information such as fine-grained loss and accuracy, which can give inspirations to new designs of NAS algorithms. In further support of the proposed NAS-Bench-201, we have analyzed it from many aspects and benchmarked 10 recent NAS algorithms, which verify its applicability.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2001.00326")
get_code_for_paper("2001.00326")
have("2001.00326")

Connect an agent — have() is free.