Tolga Ergen, Mert Pilanci
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. We have no repository linked to it either. That is a statement about our graph, not about the paper.
Understanding the fundamental principles behind the success of deep neural networks is one of the most important open questions in the current literature. To this end, we study the training problem of deep neural networks and introduce an analytic approach to unveil hidden convexity in the optimization landscape. We consider a deep parallel ReLU network architecture, which also includes standard deep networks and ResNets as its special cases. We then show that pathwise regularized training problems can be represented as an exact convex optimization problem. We further prove that the equivalent convex problem is regularized via a group sparsity inducing norm. Thus, a path regularized parallel ReLU network can be viewed as a parsimonious convex model in high dimensions. More importantly, since the original training problem may not be trainable in polynomial-time, we propose an approximate algorithm with a fully polynomial-time complexity in all data dimensions. Then, we prove strong global optimality guarantees for this algorithm. We also provide experiments corroborating our theory. (a) 2-layer NN with WD (b) 3-layer NN with WD (c) 3-layer NN with PR (Ours) Figure 1: Decision boundaries of 2-layer and 3-layer ReLU networks that are globally optimized with weight decay (WD) and path regularization (PR). Here, our convex training approach in (c) successfully learns the underlying spiral pattern for each class while the previously studied convex models in (a) and (b) fail (see Appendix A.1 for details).
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2110.09548")
get_code_for_paper("2110.09548")
have("2110.09548")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2110.09548.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2110.09548)
Connect an agent — have() is free.