Where the Ladder Stops

The ladder

Verification is a fixed four-rung ladder, and a sample's level is the highest contiguous rung passed from V0 upward. The moment one rung fails, everything above it is unreachable — not untested-but-probably-fine, unreachable. There is no confidence score anywhere in the schema; the level is the confidence.

V2 and V3 ask different questions, and the difference is the whole point. V2 asks whether two independent readings of a paper agree with each other — they could both be wrong in the same way. V3 checks the function against properties the true function must satisfy no matter how it is written. V2 is consensus; V3 is an oracle.

What was extracted from the paper

The spec is lifted from the paper, not from anyone's code. It fixes a signature, the array layouts, and the loss in closed form:

# declared layout
Q_theta_s, Q_theta_pre : (T, N, N)  where  Q[t-1, y, x]  is the rate Q_{y,x}(t)

# the loss
g = reward_val − (alpha/T) · Σ_{t=1..T} Σ_{x} x̄[t−1, x] ·
      Σ_{y≠x} ( −Q_theta_s[t−1, x, y] + Q_theta_pre[t−1, x, y]
                 + Q_theta_s[t−1, x, y] · log( Q_theta_s[t−1,x,y] / Q_theta_pre[t−1,x,y] ) )

Two implementations, written independently

Same spec, two different models, no sight of each other. Both are defensible readings — each followed one half of a spec that contradicts itself.

Q_s_xy   = Q_theta_s[t_idx, y, x]
Q_pre_xy = Q_theta_pre[t_idx, y, x]

if Q_s_xy > 0 and Q_pre_xy > 0:
    log_ratio = np.log(Q_s_xy / Q_pre_xy)
    term = -Q_s_xy + Q_pre_xy + Q_s_xy * log_ratio

reg_term += x_bar_trajectory[t - 1, x] * (
    -Q_theta_s[t - 1, x, y]
    + Q_theta_pre[t - 1, x, y]
    + Q_theta_s[t - 1, x, y] * np.log(
        Q_theta_s[t-1,x,y] / Q_theta_pre[t-1,x,y]))

import numpy as np


def drakes_loss_gradient(x_bar_trajectory, Q_theta_s, Q_theta_pre, reward_val, alpha, T):
    """
    Compute the DRAKES loss function g(theta_s) for a single batch sample.

    Args:
        x_bar_trajectory: numpy array of shape (T+1, N) where row t contains
                         the soft sample x_bar at time step t (probability distribution)
        Q_theta_s: numpy array of shape (T, N, N) where Q[t-1, y, x] is the rate
                  Q_{y,x}(t) for time step t (1-indexed)
        Q_theta_pre: numpy array of shape (T, N, N), pretrained rates
        reward_val: scalar float, the reward value r(x_bar_T)
        alpha: scalar float, regularization parameter
        T: integer, number of time steps

    Returns:
        float: the loss value g(theta_s)
    """
    x_bar_trajectory = np.asarray(x_bar_trajectory, dtype=float)
    Q_theta_s = np.asarray(Q_theta_s, dtype=float)
    Q_theta_pre = np.asarray(Q_theta_pre, dtype=float)
    reward_val = float(reward_val)
    alpha = float(alpha)
    T = int(T)

    N = x_bar_trajectory.shape[1]  # Vocabulary size

    reg_term = 0.0

    for t in range(1, T + 1):
        t_idx = t - 1  # 0-indexed for array access
        x_bar_prev = x_bar_trajectory[t - 1]  # shape (N,)

        for x in range(N):
            x_bar_x = x_bar_prev[x]

            for y in range(N):
                if y != x:
                    Q_s_xy = Q_theta_s[t_idx, y, x]
                    Q_pre_xy = Q_theta_pre[t_idx, y, x]

                    # -Q_s + Q_pre + Q_s * log(Q_s / Q_pre)
                    if Q_s_xy > 0 and Q_pre_xy > 0:
                        log_ratio = np.log(Q_s_xy / Q_pre_xy)
                        term = -Q_s_xy + Q_pre_xy + Q_s_xy * log_ratio
                    else:
                        term = -Q_s_xy + Q_pre_xy

                    reg_term += x_bar_x * term

    loss = reward_val - (alpha / T) * reg_term

    return float(loss)

import numpy as np

def drakes_loss_gradient(x_bar_trajectory, Q_theta_s, Q_theta_pre, reward_val, alpha, T):
    """
    Compute the DRAKES loss function g(theta_s) for a single batch sample.

    Parameters:
    x_bar_trajectory (numpy array): shape (T+1, N) where T is the number of time steps and N is the vocabulary size.
    Q_theta_s (numpy array): shape (T, N, N) where Q[t-1, y, x] is the rate Q_{y,x}(t) for time step t (1-indexed, so t in [1,T]).
    Q_theta_pre (numpy array): shape (T, N, N) where Q[t-1, y, x] is the rate Q_{y,x}(t) for time step t (1-indexed, so t in [1,T]).
    reward_val (float): scalar reward value.
    alpha (float): scalar regularization parameter.
    T (int): integer number of time steps.

    Returns:
    float: scalar loss value.
    """
    x_bar_trajectory = np.asarray(x_bar_trajectory, dtype=float)
    Q_theta_s = np.asarray(Q_theta_s, dtype=float)
    Q_theta_pre = np.asarray(Q_theta_pre, dtype=float)

    loss = reward_val

    reg_term = 0.0
    for t in range(1, T + 1):
        for x in range(Q_theta_s.shape[1]):
            for y in range(Q_theta_s.shape[2]):
                if x != y:
                    reg_term += x_bar_trajectory[t - 1, x] * (
                        -Q_theta_s[t - 1, x, y] + Q_theta_pre[t - 1, x, y]
                        + Q_theta_s[t - 1, x, y] * np.log(Q_theta_s[t - 1, x, y] / Q_theta_pre[t - 1, x, y])
                    )

    loss -= (alpha / T) * reg_term

    return float(loss)

Where they part

Both run. Both return a finite scalar. On the first of five inputs they agree to three significant figures — and then they don't:

Three matching digits is the trap. It is close enough to look like agreement in a summary table, and it is not agreement. V2 asks whether two independent readings of the paper produce the same number, and the answer here is no. Contiguity does the rest: V3 is unreachable, whatever it might have shown.

The cause, confirmed

The divergence is a transposition — Q[y,x] against Q[x,y] — and for a non-symmetric rate matrix those are different sums. Re-running IMPL B with its inputs pre-transposed settles it:

Which makes this a defect in the extracted spec, not in either implementation. That is a finding about our own pipeline, and it is the kind a self-graded system never surfaces — the check only has value because it is allowed to come back negative.

The routine under test

Not the whole paper — one executable routine from it. This is the beam-selection step, the argmax in Tree of Thoughts' state-selection line:

# from the paper: S_t ← argmax over subsets S of size b, maximizing Σ value(s)
def tot_select_top_states(values, b):
    """values: float array (n,). b: int, 1 ≤ b ≤ n.
    Returns (b,) indices of the b largest values, ordered by value
    descending. Ties prefer the lower index."""

It cleared V0, V1 and V2 the way any candidate must — two independent implementations, agreeing within 1e−6. The interesting rung is the next one.

V3: seven invariants, both implementations

Property tests don't compare the two implementations to each other. They assert things that must be true of any correct selection routine, and each one runs against both files independently. A property that holds on one implementation and not the other is recorded as a one-sided divergence and does not count.

Each is a real mathematical claim, not a smoke test. Translation invariance, for instance, says that shifting every score by a constant cannot change which states get selected — true of the argmax, and a property a subtly wrong implementation will fail:

values     = np.array([1.0, 4.0, 2.0, 7.0, 3.0])
original   = fn(values, 2)
translated = fn(values + 10.0, 2)

passed = np.array_equal(original, translated)

import numpy as np


def tot_select_top_states(values, b):
    """
    Select the top b states that maximize the sum of their values.

    This implements the argmax operation from the Tree of Thoughts BFS algorithm:
    S_t <- argmax_{S subset S'_t, |S|=b} sum_{s in S} V_t(s)

    Since we want to maximize the sum of values over a subset of size b,
    the optimal solution is to greedily select the b largest values.

    Args:
        values: 1D array-like of shape (n,) containing float scores for n states
        b: positive integer <= n representing the number of states to select

    Returns:
        1D numpy array of shape (b,) containing the indices of the b states
        with the largest values, sorted in descending order by their values.
        If there are ties, prefer lower indices.
    """
    values = np.asarray(values, dtype=float)

    # Get indices that would sort the values in descending order.
    # For stable sort (preferring lower indices on ties), use kind='stable'.
    sorted_indices = np.argsort(-values, kind='stable')

    top_b_indices = sorted_indices[:b]

    return top_b_indices

import numpy as np

def tot_select_top_states(values, b):
    """
    Select the top b states that maximize the sum of their values.

    Parameters:
    values (array-like): 1D array of shape (n,) containing float scores for n states.
    b (int): Positive integer <= n representing the number of states to select.

    Returns:
    array: 1D numpy array of shape (b,) containing the indices (0-indexed) of the b states with the largest values.
    """
    values = np.asarray(values, dtype=float)
    n = len(values)
    if b > n:
        raise ValueError("b cannot be greater than the number of states")

    # Get the indices of the top b states
    indices = np.argsort(-values)
    top_indices = indices[:b]

    return top_indices

# Helper function to test the implementation
def _test_tot_select_top_states():
    values = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
    b = 3
    result = tot_select_top_states(values, b)
    expected_result = np.array([4, 3, 2])
    assert np.array_equal(result, expected_result)

_test_tot_select_top_states()

Where these sit

The property-test sweep covers 3,065 methods attempted so far — a live, ongoing sweep, not a fixed batch. Counting the terminal outcome for each:

Of the 1,911, some 1,633 are clean in the strict sense used here: every property passed on both implementations, with no test flagged suspect and no one-sided divergence. Tree of Thoughts is one of those. "Transient error" here means exactly that — sampled directly, every one checked is a Bedrock endpoint connection failure mid-sweep, not a verification disagreement; those methods get retried, not written off.