SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2306.03081 · 2023

Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programs

arXiv · PDF · Open in the Atlas

Code that ran

6 of 7 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 0 honoured the contract we drafted, 0 violated it and 4 have no contract verdict; 2 failures of our instrument; 1 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2306.03081.json · badge: /api/ran/2306.03081.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
probcomp/hfppl canonical (classifier label)
classifier basis: census
4 ran · 0 no recorded run
probcomp/llamppl canonical (classifier label)
classifier basis: classifier_2025
2 ran · 1 no recorded run
FunctionStatusWhere it lives
find_c Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
probcomp/llamppl/llamppl/inference/smc_steer.py
code served (permissive licence) · get_code("010f2215d772a34e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned float64, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# The function find_c takes:
# - weights: a numpy array of numeric values
# - N: a numeric value (appears to be used in comparisons and arithmetic)

# Looking at the function body:
# - weights is sorted with np.sort()
# - weights is iterated over with len() and indexing
# - N is used in arithmetic operations (subtraction, division)
# - The function performs threshold-based calculations

# Create small test data
weights = np.array([0.5, 1.2, 0.3, 2.1, 0.8])
N = 3.0

ARGS = {
    "weights": weights,
    "N": N
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float64, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
log_softmax Ran
ran on a synthesized input
probcomp/hfppl/llamppl/util.py
code served (permissive licence) · get_code("7025f498d82408eb")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

def log_softmax(nums):
    """Compute log(softmax(nums)).

    Args:
        nums: a vector or numpy array of unnormalized log probabilities.

    Returns:
        np.array: an array of log (normalized) probabilities.
    """
    def logsumexp(nums):
        nums = np.asarray(nums)
        if np.all(nums == -np.inf):
            return -np.inf
        m = np.max(nums)
        return np.log(np.sum(np.exp(nums - m))) + m
    
    return nums - logsumexp(nums)

# Create test arguments
nums_input = np.array([1.0, 2.0, 3.0, 4.0])

ARGS = {
    "nums": nums_input
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
logsumexp Ran
ran on a synthesized input
probcomp/hfppl/llamppl/util.py
code served (permissive licence) · get_code("4aa1e61f6833bddb")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float64, ndim 0, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Construct valid arguments for logsumexp
# The function takes a single parameter 'nums' which should be array-like

# Test case 1: Basic array of numbers
nums_basic = np.array([1.0, 2.0, 3.0, 4.0])

# Test case 2: Array with negative infinity
nums_with_inf = np.array([1.0, -np.inf, 3.0])

# Test case 3: Array with all negative infinity (edge case)
nums_all_inf = np.array([-np.inf, -np.inf, -np.inf])

# Test case 4: List of numbers (also valid input)
nums_list = [1.0, 2.0, 3.0]

# Test case 5: Small values to test numerical stability
nums_small = np.array([-1000.0, -1001.0, -999.0])

ARGS = {
    "nums": nums_basic
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float64, ndim 0, dtype family float — no output shape is stated: observe_signature.py recorded float ndim 0, which disagrees with the fixture run
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
resample_optimal Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
probcomp/llamppl/llamppl/inference/smc_steer.py
code served (permissive licence) · get_code("cf0b16980037dd8d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Set random seed for reproducibility
np.random.seed(0)

# Create a small weights array
# The function expects a 1D array of weights
weights = np.array([0.1, 0.2, 0.3, 0.15, 0.25])

# N is the number of samples to resample
N = 3

ARGS = {
    "weights": weights,
    "N": N,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
softmax Ran
ran on a synthesized input
probcomp/hfppl/llamppl/util.py
code served (permissive licence) · get_code("20d709d931c06147")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

def softmax(nums):
    """Compute softmax(nums).
    
    Args:
        nums: a vector or numpy array of unnormalized log probabilities.
    
    Returns:
        np.array: an array of normalized probabilities.
    """
    def logsumexp(nums):
        nums = np.asarray(nums)
        if np.all(nums == -np.inf):
            return -np.inf
        m = np.max(nums)
        return np.log(np.sum(np.exp(nums - m))) + m

    def log_softmax(nums):
        return nums - logsumexp(nums)
    
    return np.exp(log_softmax(nums))

# Test with a simple numpy array of log probabilities
test_nums = np.array([1.0, 2.0, 3.0, 4.0])

ARGS = {
    "nums": test_nums
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
submodel Ran
ran on a synthesized input
probcomp/hfppl/llamppl/modeling.py
code served (permissive licence) · get_code("22a3190321a45a49")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned type. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import functools
import asyncio

class SubModel:

    def __init__(self):
        self.parent = None

    async def run_with_parent(self, parent):
        old_parent = self.parent
        self.parent = parent
        val = await self.forward()
        self.parent = old_parent
        return val

    async def forward(self):
        raise NotImplementedError('SubModel.forward() must be implemented by subclasses')

    async def sample(self, dist, proposal=None):
        return await self.parent.sample(dist, proposal)

    async def observe(self, dist, x):
        return await self.parent.observe(dist, x)

    async def intervene(self, dist, x):
        return await self.parent.intervene(dist, x)

    def condition(self, b):
        return self.parent.condition(b)

    def score(self, score):
        return self.parent.score(score)

    def twist(self, amt):
        return self.parent.twist(amt)

    async def call(self, submodel):
        return await submodel.run_with_parent(self.parent)

def submodel(f):
    """Decorator to create a SubModel implementation from an async function.

    For example:

    ```python
    @submodel
    async def sample_two_tokens(self, context):
        token1 = await self.sample(context.next_token())
        token2 = await self.sample(context.next_token())
        return token1, token2
    

    This SubModel can then be used from another model or submodel, using the syntax `await self.call(sample_two_tokens(context))`.
    """

    @functools.wraps(f, updated=())
    class SubModelImpl(SubModel):

        def __init__(self, *args, **kwargs):
            super().__init__()
            self.args = args
            self.kwargs = kwargs

        async def forward(self):
            return await f(self, *self.args, **self.kwargs)
    return SubModelImpl


# Setup code
async def example_async_function(self, x, y=10):
    """Example async function to decorate with @submodel"""
    return x + y


decorated_function = submodel(example_async_function)

# Test instantiation of the decorated function
instance = decorated_function(5, y=20)

ARGS = {
    "f": example_async_function
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type type — a type output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a type output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
smc_steer Not yet run
unverified (harvested, no recorded run)
probcomp/llamppl/llamppl/inference/smc_steer.py
code served (permissive licence) · get_code("f0ccf6490adcbcb1")
How it ran

no invocation recorded.

Also present in other papers' code

Of 7 samples lifted from this paper's linked repositories, 7 carry a function identity and 0 do not. Of the 7 distinct functions with an identity, 1 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 1 later) and 6 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
find_c 1 other paper
1 later
2402.06025 (2024-02)
Doing Experiments and Revising Rules with Natural Language and Probabilistic Reasoning

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Even after fine-tuning and reinforcement learning, large language models (LLMs) can be difficult, if not impossible, to control reliably with prompts alone. We propose a new inference-time approach to enforcing syntactic and semantic constraints on the outputs of LLMs, called sequential Monte Carlo (SMC) steering. The key idea is to specify language generation tasks as posterior inference problems in a class of discrete probabilistic sequence models, and replace standard decoding with sequential Monte Carlo inference. For a computational cost similar to that of beam search, SMC can steer LLMs to solve diverse tasks, including infilling, generation under syntactic constraints, and prompt intersection. To facilitate experimentation with SMC steering, we present a probabilistic programming library, LLaMPPL (https://github.com/probcomp/hfppl), for concisely specifying new generation tasks as language model probabilistic programs, and automating steering of LLaMA-family Transformers.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2306.03081")
get_code_for_paper("2306.03081")
have("2306.03081")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2306.03081.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2306.03081.svg)](https://syntology.ai/paper/2306.03081)

Connect an agent — have() is free.