| batch_to_inputs |
Ran ran; our contract draft was wrong, not the code |
pvgladkov/knowledge-distillation/knowledge_distillation/bert_trainer.py code served (permissive licence) · get_code("3ea37bfa7bd6625b") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# The function expects batch to be indexable with [0], [1], [2], [3]
# Each element should be a tensor based on the keys being used
# (input_ids, attention_mask, token_type_ids, labels)
batch = [
torch.randint(0, 100, (2, 4)), # input_ids: batch_size=2, seq_len=4
torch.ones((2, 4), dtype=torch.long), # attention_mask: batch_size=2, seq_len=4
torch.zeros((2, 4), dtype=torch.long), # token_type_ids: batch_size=2, seq_len=4
torch.randint(0, 2, (2,)), # labels: batch_size=2
]
ARGS = {
'batch': batch
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| convert_dp_to_single |
Ran ran; our contract draft was wrong, not the code |
castorini/d-bert/dbert/finetune/classifier.py code served (permissive licence) · get_code("31cb7e5ee37e07ad") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
# The function convert_dp_to_single takes a state_dict (dictionary)
# and removes 'module.' prefix from keys that have it.
# This is typically used to convert DataParallel model state dicts to single GPU format.
# Create a sample state_dict with some keys prefixed with 'module.'
state_dict = {
'module.layer1.weight': torch.randn(4, 3),
'module.layer1.bias': torch.randn(4),
'module.layer2.weight': torch.randn(5, 4),
'layer3.weight': torch.randn(3, 5), # This one doesn't have 'module.' prefix
'layer3.bias': torch.randn(3),
}
ARGS = {
'state_dict': state_dict
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| convert_single_to_dp |
Ran ran; our contract draft was wrong, not the code |
castorini/d-bert/dbert/finetune/classifier.py code served (permissive licence) · get_code("76d68a1ed9ae4351") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
# The function convert_single_to_dp takes a state_dict (dictionary of model parameters)
# and adds 'module.' prefix to all keys that don't already have it.
# This is used to convert a single-GPU model state_dict to DataParallel format.
# Create a simple state_dict with some model parameters
state_dict = {
'layer1.weight': torch.randn(4, 3),
'layer1.bias': torch.randn(4),
'layer2.weight': torch.randn(5, 4),
'layer2.bias': torch.randn(5),
}
ARGS = {
'state_dict': state_dict
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| evaluate |
Ran ran, honoured the contract we drafted |
castorini/d-bert/dbert/distill/run/distill_birnn.py code served (permissive licence) · get_code("502edb147c17e8f6") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
# Build a minimal model that satisfies the function's usage
class MinimalModel(nn.Module):
def __init__(self):
super().__init__()
self.linear = nn.Linear(4, 3) # 4 input dims, 3 output classes
def forward(self, x):
return self.linear(x)
# Build a minimal batch class with required attributes
class Batch:
def __init__(self, sentence, label):
self.sentence = sentence
self.label = label
# Build a minimal dataset iterator
class DatasetIterator:
def __init__(self, batches):
self.batches = batches
self.index = 0
def init_epoch(self):
self.index = 0
def __iter__(self):
self.index = 0
return self
def __next__(self):
if self.index >= len(self.batches):
raise StopIteration
batch = self.batches[self.index]
self.index += 1
return batch
# Create model
model = MinimalModel()
# Create batches with distinct dimensions
# sentence: (batch_size=2, feature_dim=4)
# label: (batch_size=2,) with class indices in range [0, 2]
batch1 = Batch(
sentence=torch.randn(2, 4),
label=torch.tensor([0, 1], dtype=torch.long)
)
batch2 = Batch(
sentence=torch.randn(2, 4),
label=torch.tensor([2, 0], dtype=torch.long)
)
# Create dataset iterator
ds_iter = DatasetIterator([batch1, batch2])
# Create criterion
criterion = nn.CrossEntropyLoss()
ARGS = {
"model": model,
"ds_iter": ds_iter,
"criterion": criterion,
"export_eval_labels": False
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| evaluate_score |
Ran ran, honoured the contract we drafted |
castorini/d-bert/dbert/distill/run/distill_birnn.py code served (permissive licence) · get_code("e4a51df9636b003c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
torch.manual_seed(0)
# Create a minimal model that takes batch.sentence and returns scores
class MinimalModel(nn.Module):
def __init__(self):
super().__init__()
self.linear = nn.Linear(4, 1)
def forward(self, sentence):
# sentence is expected to be a tensor
return self.linear(sentence)
def eval(self):
super().eval()
return self
# Create a minimal batch class with required attributes
class MinimalBatch:
def __init__(self, sentence, score):
self.sentence = sentence
self.score = score
# Create a minimal dataset iterator with init_epoch method
class MinimalDatasetIterator:
def __init__(self, batches):
self.batches = batches
self.index = 0
def init_epoch(self):
self.index = 0
def __iter__(self):
self.index = 0
return self
def __next__(self):
if self.index >= len(self.batches):
raise StopIteration
batch = self.batches[self.index]
self.index += 1
return batch
# Create minimal batches
batch1 = MinimalBatch(
sentence=torch.randn(2, 4), # batch_size=2, feature_dim=4
score=torch.tensor([1.5, 2.5])
)
batch2 = MinimalBatch(
sentence=torch.randn(2, 4),
score=torch.tensor([3.5, 4.5])
)
# Create dataset iterator
ds_iter = MinimalDatasetIterator([batch1, batch2])
# Create model
model = MinimalModel()
# Create criterion (not used in function but required parameter)
criterion = nn.MSELoss()
ARGS = {
"model": model,
"ds_iter": ds_iter,
"criterion": criterion,
"export_eval_labels": False
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1
- Imports
- scipy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| init_sos |
Ran ran, honoured the contract we drafted |
castorini/d-bert/dbert/generate/finetune_transfoxl.py code served (permissive licence) · get_code("59e0610210960189") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
class MockTransformer(nn.Module):
def __init__(self):
super().__init__()
# wte is the word token embedding layer
# Initialize with a small embedding: vocab_size=10, embedding_dim=4
self.wte = nn.Embedding(10, 4)
class MockModel(nn.Module):
def __init__(self):
super().__init__()
self.transformer = MockTransformer()
model = MockModel()
ARGS = {
"model": model
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| mse_forward_monkey_patch |
Ran ran; our contract draft was wrong, not the code |
castorini/d-bert/dbert/finetune/classifier.py code served (permissive licence) · get_code("fb341ab6d38412a8") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned Tensor, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
# Build a minimal BERT-like model that has the expected interface
class MinimalBERT(nn.Module):
def __init__(self, hidden_size=4, vocab_size=100):
super().__init__()
self.embedding = nn.Embedding(vocab_size, hidden_size)
self.encoder = nn.Linear(hidden_size, hidden_size)
self.pooler = nn.Linear(hidden_size, hidden_size)
def forward(self, input_ids, token_type_ids=None, attention_mask=None, output_all_encoded_layers=False):
# Returns (sequence_output, pooled_output)
embedded = self.embedding(input_ids)
encoded = self.encoder(embedded)
# Pool by taking mean over sequence dimension
pooled = self.pooler(encoded.mean(dim=1))
return encoded, pooled
# Build the model that uses the monkey-patched forward method
class BertForSequenceClassification(nn.Module):
def __init__(self, num_labels=3, hidden_size=4):
super().__init__()
self.num_labels = num_labels
self.bert = MinimalBERT(hidden_size=hidden_size)
self.dropout = nn.Dropout(0.1)
self.classifier = nn.Linear(hidden_size, num_labels)
def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None, mse=False):
# This will be replaced by the monkey patch, but we define it for completeness
pass
# Create model instance
model = BertForSequenceClassification(num_labels=3, hidden_size=4)
# Create input tensors
batch_size = 2
seq_length = 5
input_ids = torch.randint(0, 100, (batch_size, seq_length))
token_type_ids = torch.zeros((batch_size, seq_length), dtype=torch.long)
attention_mask = torch.ones((batch_size, seq_length), dtype=torch.long)
# Create labels for MSE loss (continuous values)
labels_mse = torch.randn(batch_size, 1)
# Create labels for CrossEntropy loss (class indices)
labels_ce = torch.randint(0, 3, (batch_size,))
# Build ARGS dict with MSE=True case
ARGS = {
'self': model,
'input_ids': input_ids,
'token_type_ids': token_type_ids,
'attention_mask': attention_mask,
'labels': labels_mse,
'mse': True
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| sample_query |
Ran ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT) |
castorini/d-bert/dbert/generate/finetune_transfoxl.py code served (permissive licence) · get_code("36da55e25256092f") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned str. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
from torch.distributions.categorical import Categorical
torch.manual_seed(0)
class SimpleTokenizer:
def __init__(self):
self.vocab = {
'hello': 0,
'world': 1,
'test': 2,
'<eos>': 3,
'a': 4,
'b': 5,
'c': 6,
'd': 7,
}
self.idx_to_sym = {v: k for k, v in self.vocab.items()}
def tokenize(self, text):
return text.split()
def get_idx(self, token):
return self.vocab.get(token, 0)
def get_sym(self, idx):
return self.idx_to_sym.get(idx, '<unk>')
class SimpleModel(nn.Module):
def __init__(self, vocab_size=8, hidden_dim=4):
super().__init__()
self.embedding = nn.Embedding(vocab_size, hidden_dim)
self.linear = nn.Linear(hidden_dim, vocab_size)
def forward(self, input_ids, mems=None):
x = self.embedding(input_ids)
x = x.mean(dim=1, keepdim=True)
logits = self.linear(x)
present = None
return logits, present
model = SimpleModel(vocab_size=8, hidden_dim=4)
tokenizer = SimpleTokenizer()
text = "hello world"
ARGS = {
'model': model,
'tokenizer': tokenizer,
'text': text,
'n': 128,
'paired': False,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| transfo_encode |
Ran ran; our contract draft was wrong, not the code |
castorini/d-bert/dbert/generate/finetune_transfoxl.py code served (permissive licence) · get_code("4837663eda94ee99") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
class MockTokenizer:
def tokenize(self, text, add_eos=False):
# Simple tokenization: split by spaces and add eos token if requested
tokens = text.split()
if add_eos:
tokens.append('<eos>')
return tokens
def convert_tokens_to_ids(self, tokens):
# Simple mapping: each unique token gets an id
token_to_id = {
'<eos>': 1,
'hello': 2,
'world': 3,
'test': 4,
'query': 5,
}
return [token_to_id.get(t, 0) for t in tokens]
tokenizer = MockTokenizer()
queries = [
"hello world",
"test query here"
]
ARGS = {
'tokenizer': tokenizer,
'queries': queries,
'sos_idx': None,
'eos': True,
'max_len': 80,
'split_encode': False,
'idx2remap': None,
'return_raw': False,
'condition_model': False,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|