SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2205.13554 · NeurIPS · 2022

Training and Inference on Any-Order Autoregressive Models the Right Way

Stefano Ermon, Andy Shih, Dorsa Sadigh

arXiv · PDF · Open in the Atlas

Code that ran

We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Conditional inference on arbitrary subsets of variables is a core problem in probabilistic inference with important applications such as masked language modeling and image inpainting. In recent years, the family of Any-Order Autoregressive Models (AO-ARMs) -closely related to popular models such as BERT and XL-Net -has shown breakthrough performance in arbitrary conditional tasks across a sweeping range of domains. But, in spite of their success, in this paper we identify significant improvements to be made to previous formulations of AO-ARMs. First, we show that AO-ARMs suffer from redundancy in their probabilistic model, i.e., they define the same distribution in multiple different ways. We alleviate this redundancy by training on a smaller set of univariate conditionals that still maintains support for efficient arbitrary conditional inference. Second, we upweight the training loss for univariate conditionals that are evaluated more frequently during inference. Our method leads to improved performance with no compromises on tractability, giving state-of-the-art likelihoods in arbitrary conditional modeling on text (Text8), image (CIFAR10, ImageNet32), and continuous tabular data domains. 36th Conference on Neural Information Processing Systems (NeurIPS 2022).

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2205.13554")
get_code_for_paper("2205.13554")
have("2205.13554")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2205.13554.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2205.13554.svg)](https://syntology.ai/paper/2205.13554)

Connect an agent — have() is free.