SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2303.03108 · CVPR · 2023

Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization

Peng Cui, Renzhe Xu, Xingxuan Zhang, Han Yu, Hao Zou

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Recently, flat minima are proven to be effective for improving generalization and sharpness-aware minimization (SAM) achieves state-of-the-art performance. Yet the current definition of flatness discussed in SAM and its followups are limited to the zeroth-order flatness (i.e., the worstcase loss within a perturbation radius). We show that the zeroth-order flatness can be insufficient to discriminate minima with low generalization error from those with high generalization error both when there is a single minimum or multiple minima within the given perturbation radius. Thus we present first-order flatness, a stronger measure of flatness focusing on the maximal gradient norm within a perturbation radius which bounds both the maximal eigenvalue of Hessian at local minima and the regularization function of SAM. We also present a novel training procedure named Gradient norm Aware Minimization (GAM) to seek minima with uniformly small curvature across all directions. Experimental results show that GAM improves the generalization of models trained with current optimizers such as SGD and AdamW on various datasets and networks. Furthermore, we show that GAM can help SAM find flatter minima and achieve better generalization. The code is available at https://github.com/xxgege/GAM .

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2303.03108")
get_code_for_paper("2303.03108")
have("2303.03108")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2303.03108.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2303.03108.svg)](https://syntology.ai/paper/2303.03108)

Connect an agent — have() is free.