SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2401.09967 · ACL · 2024

Sketch-Guided Constrained Decoding for Boosting Blackbox Large Language Models without Logit Access

Martin Josifoski, Chris Wendler, Robert West, Saibo Geng, Berkay Döner

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 7 functions out of this paper's own repositories and ran 2 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

RepositoryRoleRan
epfl-dlab/sketchgcd — 2 of 7
FunctionStatusWhere it lives
discard_last_incomplete_triplet Ran epfl-dlab/sketchgcd/src/constrained_generation/trie_constraint.py
pointer only (licence: NONE) · get_code("97f53de2469e4a78")
encode Ran epfl-dlab/sketchgcd/src/constrained_generation/trie_constraint.py
pointer only (licence: NONE) · get_code("af1e43bd7b90239e")
FullyExpandedLinearization Not yet run epfl-dlab/sketchgcd/src/constrained_generation/trie_constraint.py
pointer only (licence: NONE) · get_code("1737c26d0c0b8473")
LinearizationType Not yet run epfl-dlab/sketchgcd/src/constrained_generation/trie_constraint.py
pointer only (licence: NONE) · get_code("313fe6298b464995")
SubjectCollapsedLinearization Not yet run epfl-dlab/sketchgcd/src/constrained_generation/trie_constraint.py
pointer only (licence: NONE) · get_code("632dd34355f1c573")
_get_prefix_allowed_tokens_fn Not yet run epfl-dlab/sketchgcd/src/constrained_generation/trie_constraint.py
pointer only (licence: NONE) · get_code("c8f57072a4b89c0e")
get_linearization_class Not yet run epfl-dlab/sketchgcd/src/constrained_generation/trie_constraint.py
pointer only (licence: NONE) · get_code("9c568eb7a994df48")

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Constrained decoding, a technique for enforcing constraints on language model outputs, offers a way to control text generation without retraining or architectural modifications. Its application is, however, typically restricted to models that give users access to next-token distributions (usually via softmax logits), which poses a limitation with blackbox large language models (LLMs). This paper introduces sketchguided constrained decoding (SketchGCD), a novel approach to constrained decoding for blackbox LLMs, which operates without access to the logits of the blackbox LLM. SketchGCD utilizes a locally hosted auxiliary model to refine the output of an unconstrained blackbox LLM, effectively treating this initial output as a "sketch" for further elaboration. This approach is complementary to traditional logitbased techniques and enables the application of constrained decoding in settings where full model transparency is unavailable. We demonstrate the efficacy of SketchGCD through experiments in closed information extraction and constituency parsing, showing how it enhances the utility and flexibility of blackbox LLMs for complex NLP tasks. 1

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2401.09967")
get_code_for_paper("2401.09967")
have("2401.09967")

Connect an agent — have() is free.