| auroc |
Ran ran on a synthesized input |
lsc-1/pecola/lib/metrics.py code served (permissive licence) · get_code("a6359426276fb40b") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float64, ndim 0, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Set random seed for reproducibility
np.random.seed(42)
# Create small sample scores for IND (in-distribution) and OOD (out-of-distribution) data
ind_scores = np.array([0.8, 0.9, 0.85, 0.95]) # 4 in-distribution scores
ood_scores = np.array([0.2, 0.3, 0.15, 0.25]) # 4 out-of-distribution scores
# Define ARGS dict with all parameters for auroc function
ARGS = {
'ind_scores': ind_scores,
'ood_scores': ood_scores,
'order': 'largest2smallest'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float64, ndim 0, dtype family float — no output shape is stated: observe_signature.py recorded float ndim 0, which disagrees with the fixture run
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_base_score |
Ran ran, honoured the contract we drafted |
lsc-1/pecola/lib/inference/base.py code served (permissive licence) · get_code("61f9329047b0ab17") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
import numpy as np
torch.manual_seed(0)
class SimpleModel(nn.Module):
def __init__(self):
super().__init__()
self.linear = nn.Linear(4, 3)
def forward(self, input_ids, attention_mask=None):
# Return an object with logits attribute
logits = self.linear(input_ids)
class Output:
pass
output = Output()
output.logits = logits
return output
model = SimpleModel()
# Create a simple data loader as a list of tuples
# Each tuple is (input_ids, labels, attention_masks)
data_loader = [
(torch.randn(2, 4), torch.randint(0, 3, (2,)), torch.ones(2, 4)),
(torch.randn(2, 4), torch.randint(0, 3, (2,)), torch.ones(2, 4)),
]
ARGS = {
'model': model,
'data_loader': data_loader,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_cse_features |
Ran ran on a synthesized input |
lsc-1/pecola/lib/inference/draw.py code served (permissive licence) · get_code("0256d90810fbc438") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
import numpy as np
torch.manual_seed(0)
class MockModel(nn.Module):
def __init__(self):
super().__init__()
self.linear = nn.Linear(4, 8)
def forward(self, input_ids, attention_mask=None, return_dict=False, output_hidden_states=False):
# Simulate a model that returns pooler_output
batch_size = input_ids.shape[0]
pooler_output = torch.randn(batch_size, 8)
class Output:
pass
output = Output()
output.pooler_output = pooler_output
return output
def eval(self):
super().eval()
return self
# Create mock data loader with 2 batches
# Each item: (input_ids, batch_labels, attention_masks)
data_loader = [
(
torch.randint(0, 100, (2, 4)), # input_ids: batch_size=2, seq_len=4
torch.tensor([0, 1]), # batch_labels
torch.ones((2, 4)) # attention_masks
),
(
torch.randint(0, 100, (2, 4)), # input_ids: batch_size=2, seq_len=4
torch.tensor([1, 0]), # batch_labels
torch.ones((2, 4)) # attention_masks
)
]
model = MockModel()
ARGS = {
'model': model,
'data_loader': data_loader
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_d2u_score |
Ran ran on a synthesized input |
lsc-1/pecola/lib/inference/base.py code served (permissive licence) · get_code("13081870f2ef7a7d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
import numpy as np
torch.manual_seed(0)
# Create a minimal model that satisfies the function's usage
class DummyModel(nn.Module):
def __init__(self):
super().__init__()
self.linear = nn.Linear(4, 3) # 4 input features, 3 output classes
def forward(self, input_ids, attention_mask=None):
# Simple forward pass that returns logits
# input_ids: (batch, seq_len) -> flatten or pool to (batch, features)
batch_size = input_ids.shape[0]
# Return an object with .logits attribute
logits = self.linear(input_ids.float())
class Output:
pass
output = Output()
output.logits = logits
return output
# Create model
model = DummyModel()
# Create minimal data loader
# Each item should be (input_ids, labels, attention_masks)
data_loader = [
(
torch.randint(0, 10, (2, 4)), # input_ids: batch_size=2, seq_len=4
torch.randint(0, 3, (2,)), # labels: batch_size=2, 3 classes
torch.ones((2, 4)) # attention_masks: batch_size=2, seq_len=4
),
(
torch.randint(0, 10, (2, 4)),
torch.randint(0, 3, (2,)),
torch.ones((2, 4))
)
]
ARGS = {
"model": model,
"data_loader": data_loader
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_dropout_score |
Ran ran, honoured the contract we drafted |
lsc-1/pecola/lib/inference/dropout.py code served (permissive licence) · get_code("29588baf96ee353d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
import numpy as np
torch.manual_seed(0)
# Build a minimal model that satisfies the function's usage
class MinimalModel(nn.Module):
def __init__(self, num_classes=3):
super().__init__()
self.linear = nn.Linear(4, num_classes)
def forward(self, input_ids, attention_mask=None):
# The function calls model(input_ids, attention_mask=attention_masks)
# and expects outputs with a .logits attribute or raw logits
logits = self.linear(input_ids)
# Return an object with .logits attribute
class Output:
def __init__(self, logits):
self.logits = logits
return Output(logits)
# Create a minimal data loader
# The function expects tuples of (input_ids, labels, attention_masks)
def create_data_loader(batch_size=2, num_batches=2, input_dim=4, num_classes=3):
data_loader = []
for _ in range(num_batches):
input_ids = torch.randn(batch_size, input_dim)
labels = torch.randint(0, num_classes, (batch_size,))
attention_masks = torch.ones(batch_size, input_dim)
data_loader.append((input_ids, labels, attention_masks))
return data_loader
# Construct the model and data loader
model = MinimalModel(num_classes=3)
data_loader = create_data_loader(batch_size=2, num_batches=2, input_dim=4, num_classes=3)
ARGS = {
'model': model,
'data_loader': data_loader,
'passes': 5
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_features |
Ran ran on a synthesized input |
lsc-1/pecola/lib/inference/draw.py code served (permissive licence) · get_code("dd94c31be0cd6720") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
import numpy as np
torch.manual_seed(0)
# Build a minimal model that mimics RoBERTa's interface
class MinimalRoBERTa(nn.Module):
def __init__(self):
super().__init__()
# RoBERTa returns 13 hidden states (layer 0 through 12)
self.hidden_dim = 4
self.embeddings = nn.Embedding(100, self.hidden_dim)
self.layers = nn.ModuleList([nn.Linear(self.hidden_dim, self.hidden_dim) for _ in range(12)])
def forward(self, input_ids, attention_mask, return_dict=True, output_hidden_states=True):
# Embed input
hidden_state = self.embeddings(input_ids) # (batch, seq_len, hidden_dim)
# Generate hidden states through layers
hidden_states = [hidden_state]
for layer in self.layers:
hidden_state = layer(hidden_state)
hidden_states.append(hidden_state)
# Return as dict with hidden_states
class Output:
pass
output = Output()
output.hidden_states = hidden_states # List of 13 tensors
return output
class MinimalRoBERTaModel(nn.Module):
def __init__(self):
super().__init__()
self.roberta = MinimalRoBERTa()
# Create tiny batches for data loaders
# Each batch: (input_ids, batch_labels, attention_masks)
batch_size = 2
seq_len = 3
# First data loader (idx=0) with real labels
data_loader_1 = [
(torch.randint(0, 100, (batch_size, seq_len)),
torch.tensor([0, 1]),
torch.ones((batch_size, seq_len))),
(torch.randint(0, 100, (batch_size, seq_len)),
torch.tensor([1, 0]),
torch.ones((batch_size, seq_len)))
]
# Second data loader (idx=1) with dummy labels (will be replaced with -1)
data_loader_2 = [
(torch.randint(0, 100, (batch_size, seq_len)),
torch.tensor([0, 0]),
torch.ones((batch_size, seq_len))),
]
ARGS = {
'model': MinimalRoBERTaModel(),
'data_loaders': [data_loader_1, data_loader_2],
'pooler_pos': 0
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_grad_norm_score |
Ran ran on a synthesized input |
lsc-1/pecola/lib/inference/base.py code served (permissive licence) · get_code("e614043206c6e7fd") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (2,) as a later run recorded it. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
import numpy as np
torch.manual_seed(0)
# Create a minimal model that satisfies the function's requirements
class MinimalModel(nn.Module):
def __init__(self, num_classes=3):
super().__init__()
self.embedding = nn.Linear(4, 8)
self.classifier = nn.Module()
self.classifier.out_proj = nn.Linear(8, num_classes)
def forward(self, input_ids, attention_mask=None):
x = self.embedding(input_ids)
logits = self.classifier.out_proj(x)
return logits
# Create a minimal data loader (list of batches)
# Each batch is (input_ids, labels, attention_masks)
batch_size = 2
seq_len = 4
num_classes = 3
batch1 = (
torch.randn(batch_size, seq_len), # input_ids
torch.randint(0, num_classes, (batch_size,)), # labels
torch.ones(batch_size, seq_len), # attention_masks
)
batch2 = (
torch.randn(batch_size, seq_len), # input_ids
torch.randint(0, num_classes, (batch_size,)), # labels
torch.ones(batch_size, seq_len), # attention_masks
)
data_loader = [batch1, batch2]
# Create model instance
model = MinimalModel(num_classes=num_classes)
ARGS = {
"model": model,
"data_loader": data_loader,
"num_classes": num_classes,
"temperature": 1,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (2,) — shape (2,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_is_pos |
Ran ran on a synthesized input |
lsc-1/pecola/lib/metrics.py code served (permissive licence) · get_code("7b9990d7d4520898") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (5,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Set random seed for reproducibility
np.random.seed(42)
# Create small sample arrays for ind_scores and ood_scores
ind_scores = np.array([0.9, 0.8, 0.7], dtype=np.float32)
ood_scores = np.array([0.3, 0.2], dtype=np.float32)
# Valid order parameter
order = 'largest2smallest'
ARGS = {
'ind_scores': ind_scores,
'ood_scores': ood_scores,
'order': order
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (5,) — shape (5,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_key_positions |
Ran ran on a synthesized input |
lsc-1/pecola/lib/inference/generation.py code served (permissive licence) · get_code("9aab08fc71951fbb") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 2, dtype family int, shape (2, 2) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
import numpy as np
torch.manual_seed(0)
# Build a minimal BERT-like model that has the expected structure
class MinimalBertModel(nn.Module):
def __init__(self):
super().__init__()
self.bert = self.BertEncoder()
class BertEncoder(nn.Module):
def __init__(self):
super().__init__()
self.attention_layer = nn.Linear(4, 4)
def forward(self, input_ids, attention_mask, return_dict=True, output_attentions=True):
# Simulate BERT output with attention heads
batch_size = input_ids.size(0)
seq_len = input_ids.size(1)
num_heads = 2
# Create dummy attention tensor: (batch_size, num_heads, seq_len, seq_len)
attentions = torch.randn(batch_size, num_heads, seq_len, seq_len)
# Return a dict-like object with attentions
class Output:
pass
output = Output()
output.attentions = [attentions] # List of attention tensors
return output
class MinimalTokenizer:
def __init__(self):
self.pad_token_id = 0
# Create model and tokenizer
model = MinimalBertModel()
tokenizer = MinimalTokenizer()
# Create input tensors
batch_size = 2
seq_len = 5
input_ids = torch.tensor([
[1, 2, 3, 0, 0],
[4, 5, 0, 0, 0]
], dtype=torch.long)
attention_masks = torch.ones(batch_size, seq_len, dtype=torch.long)
topn = 2
ARGS = {
"model": model,
"tokenizer": tokenizer,
"input_ids": input_ids,
"attention_masks": attention_masks,
"topn": topn
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 2, dtype family int, shape (2, 2) — shape (2, 2) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_model1 |
Ran ran on a synthesized input |
lsc-1/pecola/lib/networks.py code served (permissive licence) · get_code("b78ad0b30fc6a1da") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned MockModel. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
from unittest.mock import MagicMock
torch.manual_seed(0)
# Create a minimal mock config class
class MockConfig:
def __init__(self):
self.hidden_size = 768
self.num_hidden_layers = 2
self.num_attention_heads = 8
self.intermediate_size = 3072
self.hidden_act = "gelu"
self.hidden_dropout_prob = 0.1
self.attention_probs_dropout_prob = 0.1
self.max_position_embeddings = 512
self.type_vocab_size = 2
self.initializer_range = 0.02
self.layer_norm_eps = 1e-12
self.pad_token_id = 0
self.vocab_size = 1000
# Create a minimal mock model class
class MockModel(nn.Module):
def __init__(self, config):
super().__init__()
self.config = config
self.embedding = nn.Embedding(config.vocab_size, config.hidden_size)
self.encoder = nn.Linear(config.hidden_size, config.hidden_size)
def forward(self, input_ids, attention_mask=None):
embeddings = self.embedding(input_ids)
return embeddings
# Create a minimal mock args class
class MockArgs:
def __init__(self):
self.model = "mock_model_name"
# Mock AutoConfig.from_pretrained
original_auto_config = __import__('transformers').AutoConfig.from_pretrained
def mock_config_from_pretrained(model_name, trust_remote_code=True):
return MockConfig()
# Mock AutoModel.from_pretrained
original_auto_model = __import__('transformers').AutoModel.from_pretrained
def mock_model_from_pretrained(model_name, config=None, trust_remote_code=True):
if config is None:
config = MockConfig()
return MockModel(config)
# Patch the functions
import transformers
transformers.AutoConfig.from_pretrained = mock_config_from_pretrained
transformers.AutoModel.from_pretrained = mock_model_from_pretrained
ARGS = {
"args": MockArgs()
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch, transformers — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type MockModel — a MockModel output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a MockModel output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_msp_score |
Ran ran, honoured the contract we drafted |
lsc-1/pecola/lib/inference/dropout.py code served (permissive licence) · get_code("a52d92262c816283") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
import numpy as np
torch.manual_seed(0)
# Define a minimal model that returns logits
class MinimalModel(nn.Module):
def __init__(self, num_classes=3):
super().__init__()
self.linear = nn.Linear(4, num_classes)
def forward(self, input_ids, attention_mask=None):
# Return an object with logits attribute
logits = self.linear(input_ids)
class Output:
pass
output = Output()
output.logits = logits
return output
# Create a tiny data loader (list of tuples)
# Each tuple: (input_ids, labels, attention_masks)
data_loader = [
(torch.randn(2, 4), torch.tensor([0, 1]), torch.ones(2, 4)),
(torch.randn(2, 4), torch.tensor([2, 0]), torch.ones(2, 4)),
]
model = MinimalModel(num_classes=3)
ARGS = {
"model": model,
"data_loader": data_loader,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_num_labels |
Ran ran, honoured the contract we drafted |
lsc-1/pecola/lib/exp.py code served (permissive licence) · get_code("41058bada6b6f29d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
# Analyzing the function:
# - get_num_labels takes one parameter: 'dataset' (a string)
# - It checks if 'clinc' is in the dataset string
# - It removes '_fewshot' from the dataset string
# - It looks up the resulting string in NUM_LABELS_MAP
# - The function only uses string operations on the dataset parameter
# Test with a dataset that exists in NUM_LABELS_MAP
ARGS = {
'dataset': 'hc3'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_ppl_confidence |
Ran ran on a synthesized input |
lsc-1/pecola/lib/inference/generation.py code served (permissive licence) · get_code("f70aa83668a8706c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
import numpy as np
from math import log
torch.manual_seed(0)
# Create a minimal tokenizer with required attributes
class MinimalTokenizer:
def __init__(self):
self.mask_token_id = 103
self.cls_token_id = 101
# Create a minimal model that implements the required interface
class MinimalModel(nn.Module):
def __init__(self, vocab_size=1000, hidden_dim=4):
super().__init__()
self.embedding = nn.Embedding(vocab_size, hidden_dim)
self.lm_head = nn.Linear(hidden_dim, vocab_size)
def forward(self, input_ids, attention_mask, mode='lm'):
# input_ids shape: (B, K)
embedded = self.embedding(input_ids) # (B, K, hidden_dim)
logits = self.lm_head(embedded) # (B, K, vocab_size)
return logits
# Create model and tokenizer
model = MinimalModel(vocab_size=1000, hidden_dim=4)
tokenizer = MinimalTokenizer()
# Create a minimal data loader
# Each item is (input_ids, labels, attention_masks)
batch_size = 2
seq_length = 3
vocab_size = 1000
input_ids = torch.randint(104, vocab_size, (batch_size, seq_length))
labels = torch.randint(104, vocab_size, (batch_size, seq_length))
attention_masks = torch.ones((batch_size, seq_length))
# Create data loader as a list of batches
data_loader = [
(input_ids, labels, attention_masks),
(input_ids, labels, attention_masks)
]
ARGS = {
'model': model,
'tokenizer': tokenizer,
'data_loader': data_loader
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| marginLoss |
Ran ran on a synthesized input |
lsc-1/pecola/lib/common.py code served (permissive licence) · get_code("6c82794bd0139e76") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Create pooled embeddings: shape (batch_size, embedding_dim)
# Using batch_size=2, embedding_dim=4
pooled = torch.randn(2, 4)
# Create labels: shape (batch_size,)
# Using 2 samples with 2 different classes
labels = torch.tensor([0, 1])
ARGS = {
"pooled": pooled,
"labels": labels,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| roc |
Ran ran on a synthesized input |
lsc-1/pecola/lib/metrics.py code served (permissive licence) · get_code("518d9f2cda25ff12") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 2, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Set random seed for reproducibility
np.random.seed(42)
# Create small sample score arrays for IND (in-distribution) and OOD (out-of-distribution)
ind_scores = np.array([0.8, 0.9, 0.7, 0.85], dtype=np.float32)
ood_scores = np.array([0.3, 0.4, 0.2, 0.35], dtype=np.float32)
# Valid order parameter
order = 'largest2smallest'
ARGS = {
'ind_scores': ind_scores,
'ood_scores': ood_scores,
'order': order
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 2, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| sclLoss |
Ran ran on a synthesized input |
lsc-1/pecola/lib/common.py code served (permissive licence) · get_code("e97eacfdcbaf68a1") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn.functional as F
torch.manual_seed(0)
# pooled: a normalized embedding tensor, shape (batch_size, embedding_dim)
# Should be 2D tensor where first dim is batch size, second is embedding dimension
pooled = torch.randn(2, 4)
# labels: a tensor of class labels for each sample in the batch
# Shape (batch_size,) with integer class indices
labels = torch.tensor([0, 1])
ARGS = {
"pooled": pooled,
"labels": labels,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| set_device_and_seed |
Ran ran on a synthesized input |
lsc-1/pecola/select_strategy.py code served (permissive licence) · get_code("96ab70351169ff04") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned device. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import random
import numpy as np
import os
torch.manual_seed(0)
ARGS = {
'random_seed': 42
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type device — a device output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a device output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| set_device_and_seed |
Ran ran on a synthesized input |
lsc-1/pecola/augumentation_data/yake_mask.py code served (permissive licence) · get_code("6a16ec67a0221341") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned device. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import random
import torch
import numpy as np
import os
torch.manual_seed(0)
ARGS = {
'random_seed': 42
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type device — a device output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a device output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| small_fix |
Ran ran on a synthesized input |
lsc-1/pecola/augumentation_data/mask_filling.py code served (permissive licence) · get_code("be4d25103faa09a4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
text = "Hello , world ! This is a test ? with various punctuation , marks ;"
ARGS = {
"text": text
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| take_mean_and_reshape |
Ran ran on a synthesized input |
lsc-1/pecola/lib/exp.py code served (permissive licence) · get_code("85474e5c0d2f6525") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# The function takes a single parameter 'x' which is passed to np.array()
# and then has .mean(axis=0) called on it, followed by .reshape(-1)
# So x should be array-like with at least 2 dimensions
# Create a small 2D array (batch_size=2, features=4)
x = np.array([
[1.0, 2.0, 3.0, 4.0],
[5.0, 6.0, 7.0, 8.0]
])
ARGS = {
"x": x
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| test_acc |
Ran ran on a synthesized input |
lsc-1/pecola/lib/common.py code served (permissive licence) · get_code("50ec38fc4fd612e1") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
import numpy as np
torch.manual_seed(0)
# Create a minimal model that satisfies the function's usage
class MinimalModel(nn.Module):
def __init__(self):
super().__init__()
self.linear = nn.Linear(4, 2) # 4 input features, 2 classes
def forward(self, input_ids, attention_mask=None):
# Return object with .logits attribute
logits = self.linear(input_ids)
class Output:
pass
output = Output()
output.logits = logits
return output
# Create a minimal model instance
model = MinimalModel()
# Create a minimal data loader - list of tuples (input_ids, labels, attention_masks)
# Each batch should have: input_ids (batch_size, feature_dim), labels (batch_size,), attention_masks (batch_size, feature_dim)
data_loader = [
(
torch.randn(2, 4), # input_ids: batch_size=2, feature_dim=4
torch.tensor([0, 1], dtype=torch.long), # labels: batch_size=2
torch.ones(2, 4) # attention_masks: batch_size=2, feature_dim=4
),
(
torch.randn(2, 4), # second batch
torch.tensor([1, 0], dtype=torch.long),
torch.ones(2, 4)
)
]
ARGS = {
'model': model,
'data_loader': data_loader,
'metric': 'acc',
'return_proba': False
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, sklearn, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_FFN_features |
Not yet run unverified (harvested, no recorded run) |
lsc-1/pecola/lib/inference/draw.py code served (permissive licence) · get_code("d1c6ff9d27f3294b") How it ran |
| get_keyword_confidence |
Not yet run unverified (harvested, no recorded run) |
lsc-1/pecola/lib/inference/generation.py code served (permissive licence) · get_code("24c1d85c8d245fbb") How it ran |
| get_model |
Not yet run unverified (harvested, no recorded run) |
lsc-1/pecola/lib/networks.py code served (permissive licence) · get_code("c5c3b0d2f5c9931c") How it ran |
| get_tokenizer |
Not yet run unverified (harvested, no recorded run) |
lsc-1/pecola/lib/networks.py code served (permissive licence) · get_code("73808e9a70c072ec") How it ran |