SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 1910.09573 · 2019

Detecting Underspecification with Local Ensembles

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

We present local ensembles, a method for detecting underspecification -- when many possible predictors are consistent with the training data and model class -- at test time in a pre-trained model. Our method uses local second-order information to approximate the variance of predictions across an ensemble of models from the same class. We compute this approximation by estimating the norm of the component of a test point's gradient that aligns with the low-curvature directions of the Hessian, and provide a tractable method for estimating this quantity. Experimentally, we show that our method is capable of detecting when a pre-trained model is underspecified on test data, with applications to out-of-distribution detection, detecting spurious correlates, and active learning.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("1910.09573")
get_code_for_paper("1910.09573")
have("1910.09573")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/1910.09573.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/1910.09573.svg)](https://syntology.ai/paper/1910.09573)

Connect an agent — have() is free.