SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 1706.05394 · 2017

A Closer Look at Memorization in Deep Networks

Yoshua Bengio, Stanisław Jastrzębski, Devansh Arpit, Asja Fischer, Aaron Courville, Simon Lacoste-Julien, Emmanuel Bengio, David Krueger, Nicolas Ballas, Tegan Maharaj, Maxinder Kanwal

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first. In our experiments, we expose qualitative differences in gradient-based optimization of deep neural networks (DNNs) on noise vs. real data. We also demonstrate that for appropriately tuned explicit regularization (e.g., dropout) we can degrade DNN training performance on noise datasets without compromising generalization on real data. Our analysis suggests that the notions of effective capacity which are dataset independent are unlikely to explain the generalization performance of deep networks when trained with gradient based methods because training data itself plays an important role in determining the degree of memorization.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("1706.05394")
get_code_for_paper("1706.05394")
have("1706.05394")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/1706.05394.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/1706.05394.svg)](https://syntology.ai/paper/1706.05394)

Connect an agent — have() is free.