Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, Michael Mozer
We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
Humans are accustomed to environments that contain both regularities and exceptions. For example, at most gas stations, one pays prior to pumping, but the occasional rural station does not accept payment in advance. Likewise, deep neural networks can generalize across instances that share common patterns or structures, yet have the capacity to memorize rare or irregular forms. We analyze how individual instances are treated by a model via a consistency score. The score characterizes the expected accuracy for a held-out instance given training sets of varying size sampled from the data distribution. We obtain empirical estimates of this score for individual instances in multiple data sets, and we show that the score identifies out-of-distribution and mislabeled examples at one end of the continuum and strongly regular examples at the other end. We identify computationally inexpensive proxies to the consistency score using statistics collected during training. We show examples of potential applications to the analysis of deep-learning systems.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2002.03206")
get_code_for_paper("2002.03206")
have("2002.03206")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2002.03206.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2002.03206)
Connect an agent — have() is free.