| add_article |
Ran ran on a synthesized input |
nala-cub/coda/projects/coda/probing/dataset/_templates.py code served (permissive licence) · get_code("d2121a771ce736f5") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
def add_article(noun: str) -> str:
article = 'an' if noun[0].lower() in list('aeiou') else 'a'
return '%s %s' % (article, noun)
ARGS = {
'noun': 'apple'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| build_update_fn |
Ran ran on a synthesized input |
nala-cub/coda/projects/coda/probing/representations/models.py code served (permissive licence) · get_code("39f4ad8a949bcb18") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned PjitFunction. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import haiku as hk
import jax
import jax.numpy as jnp
import jax.random as jr
import numpy as np
import optax
# Define a simple forward function that the update function will use
def forward_fn(params, rng, inputs, dropout_rate=0.5):
"""Simple forward function that processes inputs through a small network."""
# Simulate a simple neural network forward pass
# inputs shape: (batch_size, input_dim)
# outputs shape: (batch_size, n_classes)
x = inputs
if 'dense1' in params:
x = jnp.dot(x, params['dense1']['w']) + params['dense1']['b']
x = jax.nn.relu(x)
if 'dense2' in params:
x = jnp.dot(x, params['dense2']['w']) + params['dense2']['b']
return x
# Create small params dict
def create_params():
"""Create a minimal params structure."""
return {
'dense1': {
'w': jnp.ones((4, 3)),
'b': jnp.zeros(3)
},
'dense2': {
'w': jnp.ones((3, 2)),
'b': jnp.zeros(2)
}
}
# Create optimizer with chain of transformations
opt = optax.chain(
optax.trace(decay=0.9, nesterov=False),
optax.scale_by_schedule(optax.constant_schedule(0.001)),
optax.scale(-1.0)
)
# Initialize optimizer state
params = create_params()
opt_state = opt.init(params)
# Define arguments for build_update_fn
ARGS = {
'forward_fn': forward_fn,
'opt_update_fn': opt.update,
'n_classes': 2
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- haiku, jax, numpy, optax — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type PjitFunction — a PjitFunction output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a PjitFunction output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| causal_step_loss |
Ran ran on a synthesized input |
nala-cub/coda/projects/coda/probing/_src/fwd_steps.py code served (permissive licence) · get_code("79a31f93ab7a0cae") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
from typing import Dict
from torch import Tensor
torch.manual_seed(0)
# Create a minimal model that returns logits
class MinimalLanguageModel(nn.Module):
def __init__(self, vocab_size=8, hidden_size=4):
super().__init__()
self.embedding = nn.Embedding(vocab_size, hidden_size)
self.lm_head = nn.Linear(hidden_size, vocab_size)
def forward(self, input_ids, attention_mask=None):
# input_ids: (batch_size, seq_len)
x = self.embedding(input_ids) # (batch_size, seq_len, hidden_size)
logits = self.lm_head(x) # (batch_size, seq_len, vocab_size)
class Output:
pass
output = Output()
output.logits = logits
return output
# Create model instance
model = MinimalLanguageModel(vocab_size=8, hidden_size=4)
# Create model_input dictionary with required tensors
batch_size = 2
seq_len = 5
vocab_size = 8
model_input = {
'input_ids': torch.randint(0, vocab_size, (batch_size, seq_len)),
'attention_mask': torch.ones((batch_size, seq_len), dtype=torch.long)
}
# Set some attention_mask values to 0 to test masking
model_input['attention_mask'][0, 3:] = 0
model_input['attention_mask'][1, 4:] = 0
ARGS = {
'model': model,
'model_input': model_input
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- einops, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| compute_metrics |
Ran ran on a synthesized input |
nala-cub/coda/projects/coda/probing/_src/metrics.py code served (permissive licence) · get_code("49aa65398fbd515b") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
from scipy.spatial.distance import jensenshannon
from scipy.special import rel_entr
from scipy import stats
def jensenshannon_div(p, q, base=None, *, axis=0, keepdims=False):
""" Jensen Shannon Divergence.
Based on scipy.distance.jensenshannon, without the sqrt.
"""
p = np.asarray(p)
q = np.asarray(q)
p = p / np.sum(p, axis=axis, keepdims=True)
q = q / np.sum(q, axis=axis, keepdims=True)
m = (p + q) / 2.0
left = rel_entr(p, m)
right = rel_entr(q, m)
left_sum = np.sum(left, axis=axis, keepdims=keepdims)
right_sum = np.sum(right, axis=axis, keepdims=keepdims)
js = left_sum + right_sum
if base is not None:
js /= np.log(base)
return js / 2.0
def compute_metrics(preds, targets):
""" Compute all metrics for a given example
Args:
preds: array containing predicted probabilities for each class
targets: array containing groundtruth probabilities for each class
"""
spearman, spearman_p = stats.spearmanr(preds, targets)
kendalls_tau, kendalls_tau_p = stats.kendalltau(preds, targets)
metrics = {'spearman': spearman, 'spearman_p': spearman_p, 'kendalls_tau': kendalls_tau, 'kendalls_tau_p': kendalls_tau_p, 'corr_avg': (spearman + kendalls_tau) / 2, 'jensenshannon_dist': jensenshannon(preds, targets), 'jensenshannon_div': jensenshannon_div(preds, targets)}
metrics['acc'] = int(np.argmax(preds) == np.argmax(targets))
return metrics
# Setup code
preds = np.array([0.1, 0.2, 0.4, 0.3])
targets = np.array([0.15, 0.25, 0.35, 0.25])
ARGS = {
'preds': preds,
'targets': targets
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, scipy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| feature_extraction_step_single_token |
Ran ran on a synthesized input |
nala-cub/coda/projects/coda/probing/_src/fwd_steps.py code served (permissive licence) · get_code("161a21838d44dc2d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
from torch import Tensor
from typing import Dict
torch.manual_seed(0)
# Create a minimal model that returns hidden_states
class MockModel(nn.Module):
def __init__(self):
super().__init__()
self.linear = nn.Linear(4, 4)
self.device_val = 'cpu'
@property
def device(self):
return self.device_val
def forward(self, input_ids, attention_mask=None, **kwargs):
# Return a mock output object with hidden_states
batch_size = input_ids.shape[0]
seq_length = input_ids.shape[1]
hidden_dim = 4
# Create hidden states for multiple layers
hidden_states_layer1 = self.linear(torch.randn(batch_size, seq_length, hidden_dim))
hidden_states_layer2 = self.linear(hidden_states_layer1)
class Output:
def __init__(self, hs):
self.hidden_states = hs
# Return output with hidden_states as a list/tuple of layer outputs
return Output([hidden_states_layer1, hidden_states_layer2])
# Create model instance
model = MockModel()
# Create model input
model_input = {
'input_ids': torch.tensor([[1, 2, 3, 4], [5, 6, 7, 8]], dtype=torch.long),
'attention_mask': torch.tensor([[1, 1, 1, 1], [1, 1, 1, 0]], dtype=torch.long)
}
# Create meta dictionary with target token indices
# For batch size 2, we need indices for each sequence
meta = {
'target_tok_idx': torch.tensor([2, 1], dtype=torch.long) # indices into sequence dimension
}
ARGS = {
'model': model,
'model_input': model_input,
'meta': meta
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- einops, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| initialize_models |
Ran ran on a synthesized input |
nala-cub/coda/projects/coda/probing/representations/models.py code served (permissive licence) · get_code("5872051564104ccd") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import haiku as hk
import jax
import jax.numpy as jnp
import optax
import tree
# Set seed for reproducibility
jax.config.update('jax_platform_name', 'cpu')
# Create a simple network using Haiku
def simple_network(x):
x = hk.Linear(8)(x)
x = jax.nn.relu(x)
x = hk.Linear(4)(x)
return x
# Transform the network with Haiku
network = hk.transform(simple_network)
# Create an optimizer
opt = optax.adam(learning_rate=0.001)
# Parameters
batch_size = 2
input_shape = (4,)
# Create PRNG keys for initialization
key = jax.random.PRNGKey(0)
keys = jax.random.split(key, 3)
init_rngs = [keys[0], keys[1], keys[2]]
# Define ARGS dictionary
ARGS = {
'network': network,
'opt': opt,
'batch_size': batch_size,
'input_shape': input_shape,
'init_rngs': init_rngs,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- haiku, jax, optax, toolz, tree — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| jensenshannon_div |
Ran ran on a synthesized input |
nala-cub/coda/projects/coda/probing/_src/metrics.py code served (permissive licence) · get_code("0e5d9a2adb715044") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (3,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
from scipy.special import rel_entr
torch_imported = False
try:
import torch
torch_imported = True
torch.manual_seed(0)
except ImportError:
pass
# Create small test arrays with distinct dimensions
p_test = np.array([[0.2, 0.3, 0.5],
[0.1, 0.4, 0.5]])
q_test = np.array([[0.3, 0.2, 0.5],
[0.2, 0.3, 0.5]])
ARGS = {
'p': p_test,
'q': q_test,
'base': None,
'axis': 0,
'keepdims': False
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, scipy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (3,) — shape (3,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| keys_to_var_names |
Ran ran on a synthesized input |
nala-cub/coda/projects/coda/probing/dataset/_templates.py code served (permissive licence) · get_code("3431e6b20c7cc562") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def _remove_articles(k: str) -> str:
""" Remove article prefixes [1]."""
k = re.sub('\\..*', '', k)
k = re.sub('\\[.*\\]', '', k)
k = re.sub('.*?:', '', k)
return k
def _remove_counts(k: str) -> str:
""" Remove count suffixes [1]."""
return re.sub('\\d+$', '', k)
def keys_to_var_names(keys):
""" Convert list of keys to list of variable names. """
var_keys = [_remove_counts(_remove_articles(k)) for k in keys]
assert len(keys) == len(var_keys)
return var_keys
ARGS = {
"keys": [
"model.layer1[0].weight1",
"encoder:hidden.state2",
"decoder.output.bias",
"attention3"
]
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| mlm_step |
Ran ran on a synthesized input |
nala-cub/coda/projects/coda/probing/_src/fwd_steps.py code served (permissive licence) · get_code("2023f174e590228c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
from torch import Tensor
from typing import Dict
torch.manual_seed(0)
TensorDict = Dict[str, Tensor]
# Create a minimal model that returns logits
class DummyModel(nn.Module):
def __init__(self, vocab_size=10, seq_len=5):
super().__init__()
self.vocab_size = vocab_size
self.seq_len = seq_len
self.linear = nn.Linear(8, vocab_size)
self.device_attr = 'cpu'
@property
def device(self):
return torch.device('cpu')
def forward(self, input_ids=None, **kwargs):
# Return dummy logits of shape [batch_size, seq_len, vocab_size]
batch_size = input_ids.shape[0] if input_ids is not None else 2
logits = torch.randn(batch_size, self.seq_len, self.vocab_size)
# Create a simple object with logits attribute
class LogitOutput:
pass
output = LogitOutput()
output.logits = logits
return output
# Create model instance
model = DummyModel(vocab_size=10, seq_len=5)
# Create model_input dictionary
batch_size = 2
seq_len = 5
vocab_size = 10
model_input = {
'input_ids': torch.randint(0, vocab_size, (batch_size, seq_len))
}
# Create meta dictionary with required fields
meta = {
'option_input_ids': torch.randint(0, vocab_size, (batch_size, 3)), # 3 options per example
'target_tok_idx': torch.randint(0, seq_len, (batch_size,)) # one target index per example
}
ARGS = {
'model': model,
'model_input': model_input,
'meta': meta
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- einops, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| recursive_format |
Ran ran on a synthesized input |
nala-cub/coda/projects/coda/probing/dataset/_templates.py code served (permissive licence) · get_code("49a7caee9f2f5574") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
import string
from typing import Dict, TypeVar
import tree
torch = None
try:
import torch
except ImportError:
pass
TemplateObj = TypeVar('TemplateObj')
def add_article(noun: str) -> str:
article = 'an' if noun[0].lower() in list('aeiou') else 'a'
return '%s %s' % (article, noun)
class SafeFormatter(string.Formatter):
""" """
def vformat(self, format_string, args, kwargs):
args_len = len(args)
tokens = []
for lit, name, spec, conv in self.parse(format_string):
lit = lit.replace('{', '{{').replace('}', '}}')
if name is None:
tokens.append(lit)
else:
conv = '!' + conv if conv else ''
spec = ':' + spec if spec else ''
fp = name.split('[')[0].split('.')[0]
if not fp or fp.isdigit() or fp in kwargs:
tokens.extend([lit, '{', name, conv, spec, '}'])
else:
tokens.extend([lit, '{{', name, conv, spec, '}}'])
format_string = ''.join(tokens)
return string.Formatter.vformat(self, format_string, args, kwargs)
def recursive_format(obj: TemplateObj, mapping: Dict, ignore_missing: bool=False) -> TemplateObj:
"""Formats all strings within an object, using mapping
Args:
obj: Object (leaves must be strings, regardless of type)
mapping: format dictionary, maps keys to values
ignore_missing: If True, will not throw exception if a string contains a
tag not present in mapping, and will keep the tag instead.
Returns:
Object of the same type as obj, with strings formatted (tags replaced
by their value)
"""
def formatfn(x):
fmt = SafeFormatter()
formatz = lambda x, m: x.format(**m) if not ignore_missing else fmt.format(x, **m)
options = re.compile('{([^}]+):([^}]+)}')
def mysub(match):
options, thing = match.group(1, 2)
ret = ''
if 'a' in options:
if ignore_missing and thing not in mapping:
return match.group()
else:
word = formatz('{%s}' % thing, mapping)
ret += '%s ' % add_article(word).split()[0]
ret += '{%s}' % thing
return ret
x = options.sub(mysub, x)
return formatz(x, mapping)
return tree.map_structure(formatfn, obj)
ARGS = {
'obj': {
'name': 'Hello {user}',
'description': 'Welcome to {place}',
'nested': {
'title': 'You are {a:role}',
},
'items': ['Item: {item_name}', 'Count: {count}'],
},
'mapping': {
'user': 'Alice',
'place': 'Wonderland',
'role': 'admin',
'item_name': 'apple',
'count': '42',
},
'ignore_missing': False,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch, tree — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| scale_metrics |
Ran ran on a synthesized input |
nala-cub/coda/projects/coda/probing/_src/result_util.py code served (permissive licence) · get_code("589169ba57e69b61") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import pandas as pd
import numpy as np
# Create a sample DataFrame with some of the metric columns
df = pd.DataFrame({
'spearman': [0.5, 0.75, 0.3],
'pearson': [0.6, 0.8, 0.4],
'kendalls_tau': [0.55, 0.7, 0.35],
'acc': [0.9, 0.95, 0.85],
'corr_avg': [0.58, 0.75, 0.38],
'ngram_label_corr_avg': [0.62, 0.78, 0.42],
'other_metric': [100, 200, 300]
})
ARGS = {
'df': df
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| try_get_index |
Ran ran on a synthesized input |
nala-cub/coda/projects/coda/probing/_src/preprocess_steps.py code served (permissive licence) · get_code("323ea09458443106") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from typing import Optional, TypeVar
torch.manual_seed(0)
Tok = TypeVar('Tok')
# Create a simple list of tokens
tokens = ['hello', 'world', 'foo', 'bar', 'baz']
# Token to search for
stoken = 'world'
# Start index
s = 0
# End index
e = 5
# Logging level
level = -2
ARGS = {
'tokens': tokens,
'stoken': stoken,
's': s,
'e': e,
'level': level
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- absl, numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| apply_preprocessing |
Not yet run unverified (harvested, no recorded run) |
nala-cub/coda/projects/coda/probing/_src/preprocess_steps.py code served (permissive licence) · get_code("1cb1a15754b73787") How it ran |
| hf_auto_configure |
Not yet run unverified (harvested, no recorded run) |
nala-cub/coda/projects/coda/probing/_src/config_util.py code served (permissive licence) · get_code("98fbfaa5ff0b9d32") How it ran |