Sheng Shen, Trevor Darrell, Yang Gao, Michael Luo, Brandon Trabucco, Xuanlin Li, Dong Park
We lifted 7 functions out of this paper's own repositories and ran 1 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.
| Repository | Role | Ran |
|---|---|---|
| xuanlinli17/autoregressive_inference | canonical | 1 of 7 |
| Function | Status | Where it lives |
|---|---|---|
| levenshtein | Ran | xuanlinli17/autoregressive_inference/voi/algorithms/levenshtein.py code served (permissive licence) · get_code("f1dd530f5c706f2f") |
| adaptive_search | Not yet run | xuanlinli17/autoregressive_inference/voi/algorithms/adaptive_search.py code served (permissive licence) · get_code("082c009a23cb13d4") |
| beam_search | Not yet run | xuanlinli17/autoregressive_inference/voi/algorithms/beam_search.py code served (permissive licence) · get_code("ec90483fc2b722f6") |
| greedy_search | Not yet run | xuanlinli17/autoregressive_inference/voi/algorithms/greedy_search.py code served (permissive licence) · get_code("6f6c9fbef829da96") |
| nucleus_sampling | Not yet run | xuanlinli17/autoregressive_inference/voi/algorithms/nucleus_sampling.py code served (permissive licence) · get_code("6d0af24619eaf852") |
| prepare_batch_for_lm_captioning | Not yet run | xuanlinli17/autoregressive_inference/voi/core/batch_prepare_utils.py code served (permissive licence) · get_code("4c81b400a2cb458f") |
| prepare_batch_for_lm_wmt | Not yet run | xuanlinli17/autoregressive_inference/voi/core/batch_prepare_utils.py code served (permissive licence) · get_code("899639890dfcac04") |
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
The predominant approach for language modeling is to process sequences from left to right, but this eliminates a source of information: the order by which the sequence was generated. One strategy to recover this information is to decode both the content and ordering of tokens. Existing approaches supervise content and ordering by designing problem-specific loss functions and pre-training with an ordering pre-selected. Other recent works use iterative search to discover problemspecific orderings for training, but suffer from high time complexity and cannot be efficiently parallelized. We address these limitations with an unsupervised parallelizable learner that discovers high-quality generation orders purely from training data-no domain knowledge required. The learner contains an encoder network and decoder language model that perform variational inference with autoregressive orders (represented as permutation matrices) as latent variables. The corresponding ELBO is not differentiable, so we develop a practical algorithm for end-to-end optimization using policy gradients. We implement the encoder as a Transformer with non-causal attention that outputs permutations in one forward pass. Permutations then serve as target generation orders for training an insertionbased Transformer language model. Empirical results in language modeling tasks demonstrate that our method is context-aware and discovers orderings that are competitive with or even better than fixed orders.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2110.15797")
get_code_for_paper("2110.15797")
have("2110.15797")
Connect an agent — have() is free.