SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2110.08182 · EMNLP · 2021

The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Color

Alessandro Roncone, Stéphane Aroca-Ouellette, Cory Paik, Katharina Kann

arXiv · PDF · Open in the Atlas

Code that ran

12 of 14 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 0 honoured the contract we drafted, 0 violated it and 12 have no contract verdict; 0 failures of our instrument; 2 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2110.08182.json · badge: /api/ran/2110.08182.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
nala-cub/coda canonical (classifier label)
classifier basis not recorded on the link
12 ran · 2 no recorded run
FunctionStatusWhere it lives
add_article Ran
ran on a synthesized input
nala-cub/coda/projects/coda/probing/dataset/_templates.py
code served (permissive licence) · get_code("d2121a771ce736f5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

def add_article(noun: str) -> str:
    article = 'an' if noun[0].lower() in list('aeiou') else 'a'
    return '%s %s' % (article, noun)

ARGS = {
    'noun': 'apple'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
build_update_fn Ran
ran on a synthesized input
nala-cub/coda/projects/coda/probing/representations/models.py
code served (permissive licence) · get_code("39f4ad8a949bcb18")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned PjitFunction. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import haiku as hk
import jax
import jax.numpy as jnp
import jax.random as jr
import numpy as np
import optax

# Define a simple forward function that the update function will use
def forward_fn(params, rng, inputs, dropout_rate=0.5):
    """Simple forward function that processes inputs through a small network."""
    # Simulate a simple neural network forward pass
    # inputs shape: (batch_size, input_dim)
    # outputs shape: (batch_size, n_classes)
    x = inputs
    if 'dense1' in params:
        x = jnp.dot(x, params['dense1']['w']) + params['dense1']['b']
        x = jax.nn.relu(x)
    if 'dense2' in params:
        x = jnp.dot(x, params['dense2']['w']) + params['dense2']['b']
    return x

# Create small params dict
def create_params():
    """Create a minimal params structure."""
    return {
        'dense1': {
            'w': jnp.ones((4, 3)),
            'b': jnp.zeros(3)
        },
        'dense2': {
            'w': jnp.ones((3, 2)),
            'b': jnp.zeros(2)
        }
    }

# Create optimizer with chain of transformations
opt = optax.chain(
    optax.trace(decay=0.9, nesterov=False),
    optax.scale_by_schedule(optax.constant_schedule(0.001)),
    optax.scale(-1.0)
)

# Initialize optimizer state
params = create_params()
opt_state = opt.init(params)

# Define arguments for build_update_fn
ARGS = {
    'forward_fn': forward_fn,
    'opt_update_fn': opt.update,
    'n_classes': 2
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
haiku, jax, numpy, optax — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type PjitFunction — a PjitFunction output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a PjitFunction output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
causal_step_loss Ran
ran on a synthesized input
nala-cub/coda/projects/coda/probing/_src/fwd_steps.py
code served (permissive licence) · get_code("79a31f93ab7a0cae")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from typing import Dict
from torch import Tensor

torch.manual_seed(0)

# Create a minimal model that returns logits
class MinimalLanguageModel(nn.Module):
    def __init__(self, vocab_size=8, hidden_size=4):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, hidden_size)
        self.lm_head = nn.Linear(hidden_size, vocab_size)
    
    def forward(self, input_ids, attention_mask=None):
        # input_ids: (batch_size, seq_len)
        x = self.embedding(input_ids)  # (batch_size, seq_len, hidden_size)
        logits = self.lm_head(x)  # (batch_size, seq_len, vocab_size)
        
        class Output:
            pass
        
        output = Output()
        output.logits = logits
        return output

# Create model instance
model = MinimalLanguageModel(vocab_size=8, hidden_size=4)

# Create model_input dictionary with required tensors
batch_size = 2
seq_len = 5
vocab_size = 8

model_input = {
    'input_ids': torch.randint(0, vocab_size, (batch_size, seq_len)),
    'attention_mask': torch.ones((batch_size, seq_len), dtype=torch.long)
}

# Set some attention_mask values to 0 to test masking
model_input['attention_mask'][0, 3:] = 0
model_input['attention_mask'][1, 4:] = 0

ARGS = {
    'model': model,
    'model_input': model_input
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
einops, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
compute_metrics Ran
ran on a synthesized input
nala-cub/coda/projects/coda/probing/_src/metrics.py
code served (permissive licence) · get_code("49aa65398fbd515b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np
from scipy.spatial.distance import jensenshannon
from scipy.special import rel_entr
from scipy import stats

def jensenshannon_div(p, q, base=None, *, axis=0, keepdims=False):
    """ Jensen Shannon Divergence.

  Based on scipy.distance.jensenshannon, without the sqrt.
  """
    p = np.asarray(p)
    q = np.asarray(q)
    p = p / np.sum(p, axis=axis, keepdims=True)
    q = q / np.sum(q, axis=axis, keepdims=True)
    m = (p + q) / 2.0
    left = rel_entr(p, m)
    right = rel_entr(q, m)
    left_sum = np.sum(left, axis=axis, keepdims=keepdims)
    right_sum = np.sum(right, axis=axis, keepdims=keepdims)
    js = left_sum + right_sum
    if base is not None:
        js /= np.log(base)
    return js / 2.0

def compute_metrics(preds, targets):
    """ Compute all metrics for a given example

  Args:
    preds: array containing predicted probabilities for each class
    targets:  array containing groundtruth probabilities for each class
  """
    spearman, spearman_p = stats.spearmanr(preds, targets)
    kendalls_tau, kendalls_tau_p = stats.kendalltau(preds, targets)
    metrics = {'spearman': spearman, 'spearman_p': spearman_p, 'kendalls_tau': kendalls_tau, 'kendalls_tau_p': kendalls_tau_p, 'corr_avg': (spearman + kendalls_tau) / 2, 'jensenshannon_dist': jensenshannon(preds, targets), 'jensenshannon_div': jensenshannon_div(preds, targets)}
    metrics['acc'] = int(np.argmax(preds) == np.argmax(targets))
    return metrics

# Setup code
preds = np.array([0.1, 0.2, 0.4, 0.3])
targets = np.array([0.15, 0.25, 0.35, 0.25])

ARGS = {
    'preds': preds,
    'targets': targets
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, scipy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
feature_extraction_step_single_token Ran
ran on a synthesized input
nala-cub/coda/projects/coda/probing/_src/fwd_steps.py
code served (permissive licence) · get_code("161a21838d44dc2d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from torch import Tensor
from typing import Dict

torch.manual_seed(0)

# Create a minimal model that returns hidden_states
class MockModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 4)
        self.device_val = 'cpu'
    
    @property
    def device(self):
        return self.device_val
    
    def forward(self, input_ids, attention_mask=None, **kwargs):
        # Return a mock output object with hidden_states
        batch_size = input_ids.shape[0]
        seq_length = input_ids.shape[1]
        hidden_dim = 4
        
        # Create hidden states for multiple layers
        hidden_states_layer1 = self.linear(torch.randn(batch_size, seq_length, hidden_dim))
        hidden_states_layer2 = self.linear(hidden_states_layer1)
        
        class Output:
            def __init__(self, hs):
                self.hidden_states = hs
        
        # Return output with hidden_states as a list/tuple of layer outputs
        return Output([hidden_states_layer1, hidden_states_layer2])

# Create model instance
model = MockModel()

# Create model input
model_input = {
    'input_ids': torch.tensor([[1, 2, 3, 4], [5, 6, 7, 8]], dtype=torch.long),
    'attention_mask': torch.tensor([[1, 1, 1, 1], [1, 1, 1, 0]], dtype=torch.long)
}

# Create meta dictionary with target token indices
# For batch size 2, we need indices for each sequence
meta = {
    'target_tok_idx': torch.tensor([2, 1], dtype=torch.long)  # indices into sequence dimension
}

ARGS = {
    'model': model,
    'model_input': model_input,
    'meta': meta
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
einops, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
initialize_models Ran
ran on a synthesized input
nala-cub/coda/projects/coda/probing/representations/models.py
code served (permissive licence) · get_code("5872051564104ccd")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import haiku as hk
import jax
import jax.numpy as jnp
import optax
import tree

# Set seed for reproducibility
jax.config.update('jax_platform_name', 'cpu')

# Create a simple network using Haiku
def simple_network(x):
    x = hk.Linear(8)(x)
    x = jax.nn.relu(x)
    x = hk.Linear(4)(x)
    return x

# Transform the network with Haiku
network = hk.transform(simple_network)

# Create an optimizer
opt = optax.adam(learning_rate=0.001)

# Parameters
batch_size = 2
input_shape = (4,)

# Create PRNG keys for initialization
key = jax.random.PRNGKey(0)
keys = jax.random.split(key, 3)
init_rngs = [keys[0], keys[1], keys[2]]

# Define ARGS dictionary
ARGS = {
    'network': network,
    'opt': opt,
    'batch_size': batch_size,
    'input_shape': input_shape,
    'init_rngs': init_rngs,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
haiku, jax, optax, toolz, tree — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
jensenshannon_div Ran
ran on a synthesized input
nala-cub/coda/projects/coda/probing/_src/metrics.py
code served (permissive licence) · get_code("0e5d9a2adb715044")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (3,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np
from scipy.special import rel_entr

torch_imported = False
try:
    import torch
    torch_imported = True
    torch.manual_seed(0)
except ImportError:
    pass

# Create small test arrays with distinct dimensions
p_test = np.array([[0.2, 0.3, 0.5],
                    [0.1, 0.4, 0.5]])

q_test = np.array([[0.3, 0.2, 0.5],
                    [0.2, 0.3, 0.5]])

ARGS = {
    'p': p_test,
    'q': q_test,
    'base': None,
    'axis': 0,
    'keepdims': False
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, scipy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (3,) — shape (3,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
keys_to_var_names Ran
ran on a synthesized input
nala-cub/coda/projects/coda/probing/dataset/_templates.py
code served (permissive licence) · get_code("3431e6b20c7cc562")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def _remove_articles(k: str) -> str:
    """ Remove article prefixes [1]."""
    k = re.sub('\\..*', '', k)
    k = re.sub('\\[.*\\]', '', k)
    k = re.sub('.*?:', '', k)
    return k

def _remove_counts(k: str) -> str:
    """ Remove count suffixes [1]."""
    return re.sub('\\d+$', '', k)

def keys_to_var_names(keys):
    """ Convert list of keys to list of variable names. """
    var_keys = [_remove_counts(_remove_articles(k)) for k in keys]
    assert len(keys) == len(var_keys)
    return var_keys


ARGS = {
    "keys": [
        "model.layer1[0].weight1",
        "encoder:hidden.state2",
        "decoder.output.bias",
        "attention3"
    ]
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
mlm_step Ran
ran on a synthesized input
nala-cub/coda/projects/coda/probing/_src/fwd_steps.py
code served (permissive licence) · get_code("2023f174e590228c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from torch import Tensor
from typing import Dict

torch.manual_seed(0)

TensorDict = Dict[str, Tensor]

# Create a minimal model that returns logits
class DummyModel(nn.Module):
    def __init__(self, vocab_size=10, seq_len=5):
        super().__init__()
        self.vocab_size = vocab_size
        self.seq_len = seq_len
        self.linear = nn.Linear(8, vocab_size)
        self.device_attr = 'cpu'
    
    @property
    def device(self):
        return torch.device('cpu')
    
    def forward(self, input_ids=None, **kwargs):
        # Return dummy logits of shape [batch_size, seq_len, vocab_size]
        batch_size = input_ids.shape[0] if input_ids is not None else 2
        logits = torch.randn(batch_size, self.seq_len, self.vocab_size)
        
        # Create a simple object with logits attribute
        class LogitOutput:
            pass
        
        output = LogitOutput()
        output.logits = logits
        return output

# Create model instance
model = DummyModel(vocab_size=10, seq_len=5)

# Create model_input dictionary
batch_size = 2
seq_len = 5
vocab_size = 10

model_input = {
    'input_ids': torch.randint(0, vocab_size, (batch_size, seq_len))
}

# Create meta dictionary with required fields
meta = {
    'option_input_ids': torch.randint(0, vocab_size, (batch_size, 3)),  # 3 options per example
    'target_tok_idx': torch.randint(0, seq_len, (batch_size,))  # one target index per example
}

ARGS = {
    'model': model,
    'model_input': model_input,
    'meta': meta
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
einops, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
recursive_format Ran
ran on a synthesized input
nala-cub/coda/projects/coda/probing/dataset/_templates.py
code served (permissive licence) · get_code("49a7caee9f2f5574")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re
import string
from typing import Dict, TypeVar
import tree

torch = None
try:
    import torch
except ImportError:
    pass

TemplateObj = TypeVar('TemplateObj')

def add_article(noun: str) -> str:
    article = 'an' if noun[0].lower() in list('aeiou') else 'a'
    return '%s %s' % (article, noun)

class SafeFormatter(string.Formatter):
    """ """

    def vformat(self, format_string, args, kwargs):
        args_len = len(args)
        tokens = []
        for lit, name, spec, conv in self.parse(format_string):
            lit = lit.replace('{', '{{').replace('}', '}}')
            if name is None:
                tokens.append(lit)
            else:
                conv = '!' + conv if conv else ''
                spec = ':' + spec if spec else ''
                fp = name.split('[')[0].split('.')[0]
                if not fp or fp.isdigit() or fp in kwargs:
                    tokens.extend([lit, '{', name, conv, spec, '}'])
                else:
                    tokens.extend([lit, '{{', name, conv, spec, '}}'])
        format_string = ''.join(tokens)
        return string.Formatter.vformat(self, format_string, args, kwargs)

def recursive_format(obj: TemplateObj, mapping: Dict, ignore_missing: bool=False) -> TemplateObj:
    """Formats all strings within an object, using mapping

  Args:
    obj: Object (leaves must be strings, regardless of type)
    mapping: format dictionary, maps keys to values
    ignore_missing:  If True, will not throw exception if a string contains a
      tag not present in mapping, and will keep the tag instead.
  Returns:
    Object of the same type as obj, with strings formatted (tags replaced
    by their value)
  """

    def formatfn(x):
        fmt = SafeFormatter()
        formatz = lambda x, m: x.format(**m) if not ignore_missing else fmt.format(x, **m)
        options = re.compile('{([^}]+):([^}]+)}')

        def mysub(match):
            options, thing = match.group(1, 2)
            ret = ''
            if 'a' in options:
                if ignore_missing and thing not in mapping:
                    return match.group()
                else:
                    word = formatz('{%s}' % thing, mapping)
                    ret += '%s ' % add_article(word).split()[0]
            ret += '{%s}' % thing
            return ret
        x = options.sub(mysub, x)
        return formatz(x, mapping)
    return tree.map_structure(formatfn, obj)


ARGS = {
    'obj': {
        'name': 'Hello {user}',
        'description': 'Welcome to {place}',
        'nested': {
            'title': 'You are {a:role}',
        },
        'items': ['Item: {item_name}', 'Count: {count}'],
    },
    'mapping': {
        'user': 'Alice',
        'place': 'Wonderland',
        'role': 'admin',
        'item_name': 'apple',
        'count': '42',
    },
    'ignore_missing': False,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch, tree — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
scale_metrics Ran
ran on a synthesized input
nala-cub/coda/projects/coda/probing/_src/result_util.py
code served (permissive licence) · get_code("589169ba57e69b61")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd
import numpy as np

# Create a sample DataFrame with some of the metric columns
df = pd.DataFrame({
    'spearman': [0.5, 0.75, 0.3],
    'pearson': [0.6, 0.8, 0.4],
    'kendalls_tau': [0.55, 0.7, 0.35],
    'acc': [0.9, 0.95, 0.85],
    'corr_avg': [0.58, 0.75, 0.38],
    'ngram_label_corr_avg': [0.62, 0.78, 0.42],
    'other_metric': [100, 200, 300]
})

ARGS = {
    'df': df
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
try_get_index Ran
ran on a synthesized input
nala-cub/coda/projects/coda/probing/_src/preprocess_steps.py
code served (permissive licence) · get_code("323ea09458443106")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np
from typing import Optional, TypeVar

torch.manual_seed(0)

Tok = TypeVar('Tok')

# Create a simple list of tokens
tokens = ['hello', 'world', 'foo', 'bar', 'baz']

# Token to search for
stoken = 'world'

# Start index
s = 0

# End index
e = 5

# Logging level
level = -2

ARGS = {
    'tokens': tokens,
    'stoken': stoken,
    's': s,
    'e': e,
    'level': level
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
absl, numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
apply_preprocessing Not yet run
unverified (harvested, no recorded run)
nala-cub/coda/projects/coda/probing/_src/preprocess_steps.py
code served (permissive licence) · get_code("1cb1a15754b73787")
How it ran

no invocation recorded.

hf_auto_configure Not yet run
unverified (harvested, no recorded run)
nala-cub/coda/projects/coda/probing/_src/config_util.py
code served (permissive licence) · get_code("98fbfaa5ff0b9d32")
How it ran

no invocation recorded.

Also present in other papers' code

Of 14 samples lifted from this paper's linked repositories, 14 carry a function identity and 0 do not. Of the 14 distinct functions with an identity, 0 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 0 later) and 14 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Recent work has raised concerns about the inherent limitations of text-only pretraining. In this paper, we first demonstrate that reporting bias, the tendency of people to not state the obvious, is one of the causes of this limitation, and then investigate to what extent multimodal training can mitigate this issue. To accomplish this, we 1) generate the Color Dataset (CoDa), a dataset of human-perceived color distributions for 521 common objects; 2) use CoDa to analyze and compare the color distribution found in text, the distribution captured by language models, and a human's perception of color; and 3) investigate the performance differences between text-only and multimodal models on CoDa. Our results show that the distribution of colors that a language model recovers correlates more strongly with the inaccurate distribution found in text than with the ground-truth, supporting the claim that reporting bias negatively impacts and inherently limits text-only training. We then demonstrate that multimodal models can leverage their visual training to mitigate these effects, providing a promising avenue for future research. * *Email has no accent, but includes the hyphen. Everyone knows that most bananas are [MASK].

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2110.08182")
get_code_for_paper("2110.08182")
have("2110.08182")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2110.08182.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2110.08182.svg)](https://syntology.ai/paper/2110.08182)

Connect an agent — have() is free.