SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2410.02629 · NeurIPS · 2024

Estimating Generalization Performance Along the Trajectory of Proximal SGD in Robust Regression

Kai Tan, Pierre Bellec

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

This paper studies the generalization performance of iterates obtained by Gradient Descent (GD), Stochastic Gradient Descent (SGD) and their proximal variants in high-dimensional robust regression problems. The number of features is comparable to the sample size and errors may be heavy-tailed. We introduce estimators that precisely track the generalization error of the iterates along the trajectory of the iterative algorithm. These estimators are provably consistent under suitable conditions. The results are illustrated through several examples, including Huber regression, pseudo-Huber regression, and their penalized variants with non-smooth regularizer. We provide explicit generalization error estimates for iterates generated from GD and SGD, or from proximal SGD in the presence of a non-smooth regularizer. The proposed risk estimates serve as effective proxies for the actual generalization error, allowing us to determine the optimal stopping iteration that minimizes the generalization error. Extensive simulations confirm the effectiveness of the proposed generalization error estimates. 0 min(1, x) dx, the Pseudo-Huber loss ρ(r; δ) = δ 2 ( 1 + (r/δ) 2 -1) or other Lipschitz loss functions to combat the possible heavy-tails of the additive noise. Typical examples of penalty functions include the L1/Lasso [27] penalty g(b) = λ∥b∥ 1 , group-Lasso penalty [29] for grouped variables, or their non-convex variants including for instance SCAD [12] or MCP [30]. In order to solve the optimization problem (2), practitioners resort to iterative algorithms, for instance gradient descent, accelerated gradient descent, stochastic gradient descent, and the corresponding proximal methods [20] in the presence of a non-smooth regularizer. Let the algorithm starts with 38th Conference on Neural Information Processing Systems (NeurIPS 2024).

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2410.02629")
get_code_for_paper("2410.02629")
have("2410.02629")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2410.02629.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2410.02629.svg)](https://syntology.ai/paper/2410.02629)

Connect an agent — have() is free.