SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2406.04443 · ICML · 2025

Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed

Eduard Gorbunov, Alexander Gasnikov, Samuel Horváth, Aleksandr Beznosikov, Martin Takáč, Savelii Chezhegov, Yaroslav Klyukin, Andrei Semenov

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Methods with adaptive stepsizes, such as Ada-Grad and Adam, are essential for training modern Deep Learning models, especially Large Language Models. Typically, the noise in the stochastic gradients is heavy-tailed for the later ones. Gradient clipping provably helps to achieve good high-probability convergence for such noises. However, despite the similarity between Ada-Grad/Adam and Clip-SGD, the current understanding of the high-probability convergence of AdaGrad/Adam-type methods is limited in this case. In this work, we prove that AdaGrad/Adam (and their delayed version) can have provably bad high-probability convergence if the noise is heavy-tailed. We also show that gradient clipping fixes this issue, i.e., we derive new highprobability convergence bounds with polylogarithmic dependence on the confidence level for AdaGrad-Norm and Adam-Norm with clipping and with/without delay for smooth convex/nonconvex stochastic optimization with heavy-tailed noise. We extend our results to the case of Clip-AdaGrad/Clip-Adam with delayed stepsizes. Our empirical evaluations highlight the superiority of clipped versions of AdaGrad/Adam in handling the heavy-tailed noise.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2406.04443")
get_code_for_paper("2406.04443")
have("2406.04443")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2406.04443.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2406.04443.svg)](https://syntology.ai/paper/2406.04443)

Connect an agent — have() is free.