Eduard Gorbunov, Alexander Gasnikov, Samuel Horváth, Aleksandr Beznosikov, Martin Takáč, Savelii Chezhegov, Yaroslav Klyukin, Andrei Semenov
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Methods with adaptive stepsizes, such as Ada-Grad and Adam, are essential for training modern Deep Learning models, especially Large Language Models. Typically, the noise in the stochastic gradients is heavy-tailed for the later ones. Gradient clipping provably helps to achieve good high-probability convergence for such noises. However, despite the similarity between Ada-Grad/Adam and Clip-SGD, the current understanding of the high-probability convergence of AdaGrad/Adam-type methods is limited in this case. In this work, we prove that AdaGrad/Adam (and their delayed version) can have provably bad high-probability convergence if the noise is heavy-tailed. We also show that gradient clipping fixes this issue, i.e., we derive new highprobability convergence bounds with polylogarithmic dependence on the confidence level for AdaGrad-Norm and Adam-Norm with clipping and with/without delay for smooth convex/nonconvex stochastic optimization with heavy-tailed noise. We extend our results to the case of Clip-AdaGrad/Clip-Adam with delayed stepsizes. Our empirical evaluations highlight the superiority of clipped versions of AdaGrad/Adam in handling the heavy-tailed noise.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2406.04443")
get_code_for_paper("2406.04443")
have("2406.04443")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2406.04443.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2406.04443)
Connect an agent — have() is free.