SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 1710.10345 · 2017

The Implicit Bias of Gradient Descent on Separable Data

Nathan Srebro, Daniel Soudry, Shpigel Mor, Nacson

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

We examine gradient descent on unregularized logistic regression problems, with homogeneous linear predictors on linearly separable datasets. We show the predictor converges to the direction of the max-margin (hard margin SVM) solution. The result also generalizes to other monotone decreasing loss functions with an infimum at infinity, to multi-class problems, and to training a weight layer in a deep network in a certain restricted setting. Furthermore, we show this convergence is very slow, and only logarithmic in the convergence of the loss itself. This can help explain the benefit of continuing to optimize the logistic or cross-entropy loss even after the training error is zero and the training loss is extremely small, and, as we show, even if the validation loss increases. Our methodology can also aid in understanding implicit regularization in more complex models and with other optimization methods.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("1710.10345")
get_code_for_paper("1710.10345")
have("1710.10345")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/1710.10345.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/1710.10345.svg)](https://syntology.ai/paper/1710.10345)

Connect an agent — have() is free.