| compute_all |
Ran ran on a synthesized input |
uditarora/ood-text-emnlp/perplexity.py code served (permissive licence) · get_code("f3167e070ba7c977") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (2,) as a later run recorded it. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
import numpy as np
from unittest.mock import MagicMock
torch.manual_seed(0)
# Create a minimal model that satisfies the function's needs
class MinimalModel(nn.Module):
def __init__(self):
super().__init__()
self.config = MagicMock()
self.config.n_positions = 8
self.linear = nn.Linear(4, 1)
def forward(self, input_ids, labels=None):
# Return (loss, logits)
loss = torch.tensor(0.5)
return (loss, None)
# Create a minimal encodings object with input_ids attribute
class MinimalEncodings:
def __init__(self, seq_length=16):
self.input_ids = torch.randint(0, 100, (2, seq_length))
# Build model
model = MinimalModel()
# Build all_encodings - list of encoding objects
all_encodings = [
MinimalEncodings(seq_length=16),
MinimalEncodings(seq_length=20),
]
# Build fname
fname = "test_output"
# Build save_path (use a simple string, function will try to save there)
save_path = "/tmp"
# Build device
device = "cpu"
ARGS = {
"model": model,
"all_encodings": all_encodings,
"fname": fname,
"save_path": save_path,
"device": device,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (2,) — shape (2,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| compute_auroc |
Ran ran on a synthesized input |
uditarora/ood-text-emnlp/utils.py code served (permissive licence) · get_code("72563a8679267472") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Create sample probability predictions for in-distribution and out-of-distribution data
np.random.seed(0)
# In-distribution predictions: typically lower scores
id_pps = np.random.uniform(0.1, 0.4, size=10)
# Out-of-distribution predictions: typically higher scores
ood_pps = np.random.uniform(0.6, 0.9, size=10)
ARGS = {
"id_pps": id_pps,
"ood_pps": ood_pps,
"normalize": False,
"return_curve": False,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, sklearn — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| compute_far |
Ran ran on a synthesized input |
uditarora/ood-text-emnlp/utils.py code served (permissive licence) · get_code("4bd8eb730cf38934") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Create sample data for id_pps and ood_pps
# These should be 1D numpy arrays of probability scores
np.random.seed(0)
id_pps = np.random.rand(100)
ood_pps = np.random.rand(150)
ARGS = {
"id_pps": id_pps,
"ood_pps": ood_pps,
"rate": 5
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| compute_perplexity |
Ran ran on a synthesized input |
uditarora/ood-text-emnlp/perplexity.py code served (permissive licence) · get_code("b29cddb5f632a9d9") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
# Create a minimal model that satisfies the function's usage
class MinimalModel(nn.Module):
def __init__(self):
super().__init__()
# Dummy linear layer for forward pass
self.linear = nn.Linear(4, 4)
# Create a config with required n_positions attribute
self.config = type('Config', (), {'n_positions': 8})()
def forward(self, input_ids, labels=None):
# Return a tuple where first element is loss (scalar tensor)
batch_size = input_ids.size(0)
loss = torch.tensor(0.5, dtype=torch.float32)
return (loss,)
# Create minimal encodings object with input_ids attribute
class MinimalEncodings:
def __init__(self):
# Create input_ids: batch_size=2, seq_length=12, vocab_size=4
self.input_ids = torch.randint(0, 4, (2, 12), dtype=torch.long)
# Construct the arguments
model = MinimalModel()
encodings = MinimalEncodings()
ARGS = {
'model': model,
'encodings': encodings,
'stride': 2,
'device': 'cpu'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| compute_px |
Ran ran on a synthesized input |
uditarora/ood-text-emnlp/utils.py code served (permissive licence) · get_code("83b21e96c79e998a") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (2,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Analyze the function:
# - ppl: used in np.log(ppl), so should be a numeric scalar or array
# - lls: iterable of iterables, len() is called on each element
# The function computes lengths of each sub-list and uses them in calculation
# Create test data
ppl = 50.0 # perplexity value, a scalar float
# lls should be a list of lists (or similar iterables)
# Each inner list represents tokens/sequence
lls = [
[1, 2, 3, 4], # sequence of length 4
[5, 6, 7, 8, 9, 10] # sequence of length 6
]
ARGS = {
"ppl": ppl,
"lls": lls
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (2,) — shape (2,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| encode |
Ran ran on a synthesized input |
uditarora/ood-text-emnlp/msp_eval.py code served (permissive licence) · get_code("f9a4adc707cb9193") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
class MockTokenizer:
def encode_plus(self, text, add_special_tokens=False, return_token_type_ids=False,
max_length=None, pad_to_max_length=False, return_attention_mask=False,
return_tensors=None):
# Simulate tokenizer output
input_ids = torch.tensor([[101, 2054, 2003, 102, 0, 0]])
attention_mask = torch.tensor([[1, 1, 1, 1, 0, 0]])
return {
'input_ids': input_ids,
'attention_mask': attention_mask
}
tokenizer = MockTokenizer()
text = "What is this?"
ARGS = {
'tokenizer': tokenizer,
'text': text
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_dataloader |
Ran ran on a synthesized input |
uditarora/ood-text-emnlp/roberta_fine_tune.py code served (permissive licence) · get_code("b41e11872936971a") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataLoader. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
from torch.utils.data import DataLoader, RandomSampler, TensorDataset, SequentialSampler
torch.manual_seed(0)
# Create a minimal stand-in tokenizer that returns a tokenizer output object
class MockTokenizerOutput:
def __init__(self, input_ids, attention_mask):
self.input_ids = input_ids
self.attention_mask = attention_mask
class MockTokenizer:
def __call__(self, *args, padding=None, max_length=None, truncation=True):
# Return a mock output with input_ids and attention_mask
# Create 2 samples with max_length=4
batch_size = 2
seq_length = 4
input_ids = [[101, 2054, 2003, 102], [101, 1045, 2572, 102]]
attention_mask = [[1, 1, 1, 1], [1, 1, 1, 0]]
return MockTokenizerOutput(input_ids, attention_mask)
# Create mock tokenizer instance
tokenizer = MockTokenizer()
# Define tokenizer_args - these are passed to the tokenizer
tokenizer_args = ("Hello world", "How are you")
# Define labels - 2 labels for 2 samples
labels = [0, 1]
ARGS = {
"tokenizer_args": tokenizer_args,
"tokenizer": tokenizer,
"padding": "max_length",
"max_length": 4,
"batch_size": 2,
"truncation": True,
"labels": labels,
"shuffle": True,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DataLoader — a DataLoader output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataLoader output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| process_entailment |
Ran ran on a synthesized input |
uditarora/ood-text-emnlp/msp_eval.py code served (permissive licence) · get_code("374d3df6220579b7") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
# Create a minimal tokenizer stand-in that implements encode_plus
class MinimalTokenizer:
def encode_plus(self, text, add_special_tokens=False, return_token_type_ids=False,
max_length=150, pad_to_max_length=False, return_attention_mask=False,
return_tensors=None):
# Simple mock: return token IDs, attention mask as tensors
tokens = [101] + [102] * min(len(text.split()), max_length - 2) + [103]
token_ids = tokens[:max_length]
# Pad to max_length
token_ids = token_ids + [0] * (max_length - len(token_ids))
attention_mask = [1 if tid != 0 else 0 for tid in token_ids]
result = {
'input_ids': torch.tensor([token_ids], dtype=torch.long),
'attention_mask': torch.tensor([attention_mask], dtype=torch.long)
}
return result
# Create minimal dataset with required keys
dataset = [
{'sentence1': 'The cat sat', 'sentence2': 'on the mat'},
{'sentence1': 'Dogs are', 'sentence2': 'loyal pets'}
]
tokenizer = MinimalTokenizer()
ARGS = {
'dataset': dataset,
'tokenizer': tokenizer,
'key1': 'sentence1',
'key2': 'sentence2'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| process_hf_dataset |
Ran ran on a synthesized input |
uditarora/ood-text-emnlp/roberta_fine_tune.py code served (permissive licence) · get_code("d33fb2634b456ea3") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataLoader. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
from torch.utils.data import DataLoader, RandomSampler, TensorDataset, SequentialSampler
torch.manual_seed(0)
# Create a minimal tokenizer stand-in
class MockTokenizer:
def __call__(self, *args, padding=None, max_length=None, truncation=None):
# Return a mock tokenization result with input_ids and attention_mask
batch_size = len(args[0])
return MockTokenizationOutput(
input_ids=[[101, 2054, 2003, 102] for _ in range(batch_size)],
attention_mask=[[1, 1, 1, 1] for _ in range(batch_size)]
)
class MockTokenizationOutput:
def __init__(self, input_ids, attention_mask):
self.input_ids = input_ids
self.attention_mask = attention_mask
# Create a minimal dataset stand-in
class MockDataset:
def __init__(self):
self.splits = {
'validation': {
'text': ['sample text one', 'sample text two'],
'sentence': ['sentence one', 'sentence two'],
'premise': ['premise one', 'premise two'],
'hypothesis': ['hypothesis one', 'hypothesis two'],
'sentence1': ['sent1 one', 'sent1 two'],
'sentence2': ['sent2 one', 'sent2 two'],
'label': [0, 1]
}
}
def __getitem__(self, key):
return self.splits[key]
# Create mock dataset with different splits
dataset = MockDataset()
dataset.splits['test'] = dataset.splits['validation']
dataset.splits['validation_matched'] = dataset.splits['validation']
tokenizer = MockTokenizer()
ARGS = {
'dataset': dataset,
'split': 'eval',
'task_name': 'sst2',
'tokenizer': tokenizer,
'padding': 'max_length',
'max_length': 4,
'batch_size': 2,
'truncation': True,
'n': 2,
'shuffle': True
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DataLoader — a DataLoader output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataLoader output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| process_msp |
Ran ran on a synthesized input |
uditarora/ood-text-emnlp/msp_eval.py code served (permissive licence) · get_code("87d7cfb062dec677") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (2,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
torch.manual_seed(0)
# Create a minimal model that satisfies the function's usage
class MinimalModel(nn.Module):
def __init__(self):
super().__init__()
self.linear = nn.Linear(4, 5) # 5 output classes for softmax
def forward(self, input_ids, attention_mask):
# The function expects model to return something indexable with [0]
# and that result to be indexable with [0] again
batch_size = input_ids.shape[0]
output = self.linear(input_ids.float()) # (batch_size, 5)
return (output,) # Return tuple with output as first element
# Create minimal model instance
model = MinimalModel()
# Create sample encodings - list of dicts with input_ids and attention_mask
all_encodings = [
{
'input_ids': torch.randint(0, 100, (2, 4)), # batch=2, seq_len=4
'attention_mask': torch.ones((2, 4), dtype=torch.long)
},
{
'input_ids': torch.randint(0, 100, (2, 4)),
'attention_mask': torch.ones((2, 4), dtype=torch.long)
}
]
ARGS = {
'all_encodings': all_encodings,
'model': model
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (2,) — shape (2,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| setup |
Ran ran on a synthesized input |
uditarora/ood-text-emnlp/perplexity.py code served (permissive licence) · get_code("0975a4d4fcffe722") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
from transformers import AutoTokenizer, AutoModelWithLMHead
torch.manual_seed(0)
# Mock model class that mimics AutoModelWithLMHead behavior
class MockLMModel(nn.Module):
def __init__(self):
super().__init__()
self.linear = nn.Linear(4, 8)
def forward(self, input_ids):
return self.linear(input_ids.float())
def to(self, device):
super().to(device)
return self
# Mock tokenizer class
class MockTokenizer:
def __init__(self):
self.vocab_size = 50257
def encode(self, text):
return [1, 2, 3]
def decode(self, ids):
return "decoded text"
# Create mock instances
mock_model = MockLMModel()
mock_tokenizer = MockTokenizer()
# Patch AutoModelWithLMHead.from_pretrained to return our mock
original_from_pretrained = AutoModelWithLMHead.from_pretrained
def mock_from_pretrained(path):
return mock_model
AutoModelWithLMHead.from_pretrained = staticmethod(mock_from_pretrained)
# Patch AutoTokenizer.from_pretrained to return our mock
original_tokenizer_from_pretrained = AutoTokenizer.from_pretrained
def mock_tokenizer_from_pretrained(path):
return mock_tokenizer
AutoTokenizer.from_pretrained = staticmethod(mock_tokenizer_from_pretrained)
ARGS = {
'path': 'lvwerra/gpt2-imdb'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch, transformers — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| process_custom_dataset |
Not yet run unverified (harvested, no recorded run) |
uditarora/ood-text-emnlp/roberta_fine_tune.py code served (permissive licence) · get_code("30912a8b3a57ba75") How it ran |