SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2209.07238 · NeurIPS · 2022

Generalization Properties of NAS under Activation and Skip Connection Search

Fanghui Liu, Volkan Cevher, Grigorios Chrysos, Zhenyu Zhu

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.

Abstract

Neural Architecture Search (NAS) has fostered the automatic discovery of stateof-the-art neural architectures. Despite the progress achieved with NAS, so far there is little attention to theoretical guarantees on NAS. In this work, we study the generalization properties of NAS under a unifying framework enabling (deep) layer skip connection search and activation function search. To this end, we derive the lower (and upper) bounds of the minimum eigenvalue of the Neural Tangent Kernel (NTK) under the (in)finite-width regime using a certain search space including mixed activation functions, fully connected, and residual neural networks. We use the minimum eigenvalue to establish generalization error bounds of NAS in the stochastic gradient descent training. Importantly, we theoretically and experimentally show how the derived results can guide NAS to select the top-performing architectures, even in the case without training, leading to a trainfree algorithm based on our theory. Accordingly, our numerical validation shed light on the design of computationally efficient methods for NAS. Our analysis is non-trivial due to the coupling of various architectures and activation functions under the unifying framework and has its own interest in providing the lower bound of the minimum eigenvalue of NTK in deep learning theory.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2209.07238")
get_code_for_paper("2209.07238")
have("2209.07238")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2209.07238.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2209.07238.svg)](https://syntology.ai/paper/2209.07238)

Connect an agent — have() is free.