SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2008.08186 · 2020

Prevalence of Neural Collapse during the terminal phase of deep learning training

X Han, Vardan Papyan, David Donoho

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Modern practice for training classification deepnets involves a Terminal Phase of Training (TPT), which begins at the epoch where training error first vanishes; During TPT, the training error stays effectively zero while training loss is pushed towards zero. Direct measurements of TPT, for three prototypical deepnet architectures and across seven canonical classification datasets, expose a pervasive inductive bias we call Neural Collapse, involving four deeply interconnected phenomena: (NC1) Cross-example within-class variability of last-layer training activations collapses to zero, as the individual activations themselves collapse to their class-means; (NC2) The classmeans collapse to the vertices of a Simplex Equiangular Tight Frame (ETF); (NC3) Up to rescaling, the last-layer classifiers collapse to the class-means, or in other words to the Simplex ETF, i.e. to a self-dual configuration; (NC4) For a given activation, the classifier's decision collapses to simply choosing whichever class has the closest train class-mean, i.e. the Nearest Class-Center (NCC) decision rule. The symmetric and very simple geometry induced by the TPT confers important benefits, including better generalization performance, better robustness, and better interpretability. Machine learning | Deep learning | Adversarial robustness | Simplex Equiangular Tight Frame | Nearest Class Center | Inductive bias V.P. and X.H. performed the experiments, formulated theorems, and derived proofs. D.D. guided the experimental design, formulated theorems, and derived proofs.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2008.08186")
get_code_for_paper("2008.08186")
have("2008.08186")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2008.08186.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2008.08186.svg)](https://syntology.ai/paper/2008.08186)

Connect an agent — have() is free.