| get_model_type |
Ran ran on a synthesized input |
nala-cub/prost/src/baselines/results.py code served (permissive licence) · get_code("3da1a72ed0d0e883") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
def get_model_type(x):
if x == 'openai-gpt' or x.startswith('allenai-unifiedqa'):
return x.split('-')[1]
elif x.startswith('albert'):
return ' '.join([x.split('-')[0], x.split('-')[-1]])
else:
return x.split('-')[0]
# Test cases to understand the function behavior
test_cases = [
'openai-gpt',
'allenai-unifiedqa-v2',
'albert-base-v2',
'bert-base-uncased',
]
# The function takes a single string parameter 'x' and processes it
# based on different model naming conventions
ARGS = {
'x': 'bert-base-uncased'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_and_check_config_yml |
Ran ran on a synthesized input |
nala-cub/prost/src/prost/prost.py code served (permissive licence) · get_code("404e639140047862") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import os
import yaml
# Create a temporary YAML file with valid config structure
temp_dir = tempfile.mkdtemp()
config_path = os.path.join(temp_dir, "config.yml")
config_data = {
'configs': {
'task1': {'param1': 'value1'},
'task2': {'param2': 'value2'}
}
}
with open(config_path, 'w') as f:
yaml.dump(config_data, f)
ARGS = {
'path': config_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- datasets, yaml — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_preds |
Ran ran on a synthesized input |
nala-cub/prost/src/baselines/results.py code served (permissive licence) · get_code("a410f0abdd3fb511") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
from pathlib import Path
import pandas as pd
import os
# Create a temporary directory structure with mock prediction files
tmpdir = tempfile.mkdtemp()
cachedir = Path(tmpdir)
preds_dir = cachedir / 'preds'
preds_dir.mkdir(parents=True, exist_ok=True)
# Create mock prediction CSV files
pred_data_1 = pd.DataFrame({
'example_idx': [0, 1],
'prediction': [0.5, 0.7]
})
pred_data_1.to_csv(preds_dir / 'run1_model_a_preds.csv', index=False)
pred_data_2 = pd.DataFrame({
'example_idx': [0, 1],
'prediction': [0.6, 0.8]
})
pred_data_2.to_csv(preds_dir / 'run2_model_b_preds.csv', index=False)
# Create a mock dataset directory structure
import sys
from unittest.mock import Mock, patch
# Mock the datasets.load_dataset to return a mock dataset
mock_ds = Mock()
mock_ds_data = pd.DataFrame({
'example_idx': [0, 1],
'text': ['sample text 1', 'sample text 2'],
'label': [0, 1]
})
mock_ds.__getitem__ = Mock(return_value=mock_ds_data)
mock_ds.set_format = Mock()
# Patch datasets.load_dataset
datasets_patcher = patch('datasets.load_dataset', return_value=mock_ds)
datasets_patcher.start()
ARGS = {
'cachedir': cachedir
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- datasets, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_raw_preds |
Ran ran on a synthesized input |
nala-cub/prost/src/baselines/results.py code served (permissive licence) · get_code("8b6c4bddb433b1c8") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import shutil
from pathlib import Path
import pandas as pd
# Create a temporary directory structure with mock prediction files
temp_dir = tempfile.mkdtemp()
cachedir = Path(temp_dir)
preds_dir = cachedir / 'preds'
preds_dir.mkdir(parents=True, exist_ok=True)
# Create sample CSV files for models that don't end with 'v1'
df1 = pd.DataFrame({
'prediction': [0.1, 0.2, 0.3],
'actual': [0, 1, 0]
})
df1.to_csv(preds_dir / 'model1_bert_preds.csv', index=False)
df2 = pd.DataFrame({
'prediction': [0.4, 0.5],
'actual': [1, 1]
})
df2.to_csv(preds_dir / 'model2_gpt_preds.csv', index=False)
# Create a CSV file for a model that ends with 'v1' (should be skipped)
df3 = pd.DataFrame({
'prediction': [0.6, 0.7],
'actual': [0, 0]
})
df3.to_csv(preds_dir / 'model3_roberta_v1_preds.csv', index=False)
ARGS = {
'cachedir': cachedir
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| make_enum_vars |
Ran ran on a synthesized input |
nala-cub/prost/src/prost/prost.py code served (permissive licence) · get_code("8adad0c7719ff831") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from enum import IntEnum
def to_ek(s: str) -> str:
return s.replace(' ', '').upper()
def make_enum_vars(k, vals):
return (k, IntEnum(k, {to_ek(v): i for i, v in enumerate(vals)}))
# Setup code
ARGS = {
'k': 'MyEnum',
'vals': ['foo bar', 'baz qux', 'hello world']
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| normalize_answer |
Ran ran on a synthesized input |
nala-cub/prost/src/baselines/unifiedqa.py code served (permissive licence) · get_code("77c174355b5aa8d2") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
import string
import toolz.curried as T
# The function takes a single parameter 's' which is a string
# The function normalizes text by:
# 1. Converting to lowercase
# 2. Removing punctuation
# 3. Fixing buggy characters
# 4. Removing articles (a, an, the)
# 5. Fixing whitespace
ARGS = {
"s": "The quick brown Fox! Has some buggy^characters<and>articles."
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- toolz — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| prep_example_prost_gcp |
Ran ran on a synthesized input |
nala-cub/prost/src/baselines/unifiedqa.py code served (permissive licence) · get_code("e1cb1fa52f9ae046") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# Construct a valid example dict for PROST
example = {
'ex_question': 'What is the capital of France?',
'A': 'Paris',
'B': 'London',
'C': 'Berlin',
'D': 'Madrid',
'label': 0, # Points to option A (Paris)
'context': 'France is a country in Western Europe.'
}
ARGS = {
'example': example
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| score_unifiedqa |
Ran ran on a synthesized input |
nala-cub/prost/src/baselines/unifiedqa.py code served (permissive licence) · get_code("effefd745752a218") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
import string
import toolz.curried as T
def normalize_answer(s: str) -> str:
"""Normalize UnifiedQA Generated Text [1]. """
import re
import string
def remove_articles(text):
return re.sub('\\b(a|an|the)\\b', ' ', text)
def white_space_fix(text):
return ' '.join(text.split())
def fix_buggy_characters(str):
return re.sub('[{}^\\\\`⁇<]', ' ', str)
def remove_punc(text):
exclude = set(string.punctuation)
return ''.join((ch for ch in text if ch not in exclude))
return T.pipe(s, str.lower, remove_punc, fix_buggy_characters, remove_articles, white_space_fix)
def score_unifiedqa(prediction: str, groundtruth: str) -> float:
if prediction == groundtruth:
return 3.0
prediction = normalize_answer(prediction)
groundtruth = normalize_answer(groundtruth)
if prediction == groundtruth:
return 2.0
if ' ' in prediction or ' ' in groundtruth:
prediction_split = prediction.split(' ')
groundtruth_split = groundtruth.split(' ')
overlap = list(set(prediction_split) & set(groundtruth_split))
return len(overlap) / max(len(prediction_split), len(groundtruth_split))
else:
return 0.0
ARGS = {
"prediction": "The quick brown fox jumps over the lazy dog",
"groundtruth": "A quick brown fox jumps over a lazy dog"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- toolz — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| test_step |
Ran ran on a synthesized input |
nala-cub/prost/src/baselines/zs_runner.py code served (permissive licence) · get_code("2d47fac6ddc5af35") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
import torch.nn.functional as F
from einops import rearrange, reduce, repeat
torch.manual_seed(0)
# Define a minimal model that supports both generative and non-generative modes
class MinimalModel(nn.Module):
def __init__(self, vocab_size=256, hidden_size=4, model_type='bert'):
super().__init__()
self.embedding = nn.Embedding(vocab_size, hidden_size)
self.lm_head = nn.Linear(hidden_size, vocab_size)
self.config = type('Config', (), {'model_type': model_type})()
self.vocab_size = vocab_size
self.hidden_size = hidden_size
def forward(self, input_ids, attention_mask=None):
x = self.embedding(input_ids)
logits = self.lm_head(x)
return type('Output', (), {'logits': logits})()
def generate(self, input_ids):
batch_size = input_ids.shape[0]
return torch.randint(0, self.vocab_size, (batch_size, 1))
# Test case 1: Causal model (generative=False, is_causal=True)
batch_size = 2
seq_len = 5
vocab_size = 256
hidden_size = 4
model_causal = MinimalModel(vocab_size=vocab_size, hidden_size=hidden_size, model_type='gpt')
input_ids_causal = torch.randint(0, vocab_size, (batch_size, seq_len))
attention_mask_causal = torch.ones((batch_size, seq_len))
model_input_causal = {
'input_ids': input_ids_causal,
'attention_mask': attention_mask_causal
}
output_causal = {}
# Test case 2: Non-causal model with T5 (is_causal=False, model_type='t5')
model_t5 = MinimalModel(vocab_size=vocab_size, hidden_size=hidden_size, model_type='t5')
# T5 mode expects logits with shape (batch, 3, vocab_size)
class T5MinimalModel(nn.Module):
def __init__(self, vocab_size=256, hidden_size=4):
super().__init__()
self.embedding = nn.Embedding(vocab_size, hidden_size)
self.lm_head = nn.Linear(hidden_size, vocab_size)
self.config = type('Config', (), {'model_type': 't5'})()
def forward(self, input_ids, attention_mask=None):
x = self.embedding(input_ids)
# Always output 3 tokens for T5
x = x.unsqueeze(1).expand(-1, 3, -1, -1)
x = rearrange(x, 'b t s h -> (b t s) h')
logits = self.lm_head(x)
logits = rearrange(logits, '(b t s) v -> b t v', b=input_ids.shape[0], t=3)
return type('Output', (), {'logits': logits})()
model_t5 = T5MinimalModel(vocab_size=vocab_size, hidden_size=hidden_size)
input_ids_t5 = torch.randint(0, vocab_size, (batch_size, 1))
attention_mask_t5 = torch.ones((batch_size, 1))
model_input_t5 = {
'input_ids': input_ids_t5,
'attention_mask': attention_mask_t5
}
num_options = 3
option_input_ids_t5 = torch.randint(0, vocab_size, (batch_size, num_options))
output_t5 = {
'option_input_ids': option_input_ids_t5,
'example_idx': torch.arange(batch_size),
'target_tok_idx': torch.zeros(batch_size, dtype=torch.long)
}
# Test case 3: Non-causal model with non-T5 (is_causal=False, model_type='bert')
model_bert = MinimalModel(vocab_size=vocab_size, hidden_size=hidden_size, model_type='bert')
seq_len_bert = 4
input_ids_bert = torch.randint(0, vocab_size, (batch_size, seq_len_bert))
attention_mask_bert = torch.ones((batch_size, seq_len_bert))
model_input_bert = {
'input_ids': input_ids_bert,
'attention_mask': attention_mask_bert
}
option_input_ids_bert = torch.randint(0, vocab_size, (batch_size, num_options))
output_bert = {
'option_input_ids': option_input_ids_bert,
'example_idx': torch.arange(batch_size),
'target_tok_idx': torch.randint(0, seq_len_bert, (batch_size,))
}
# Build ARGS for the curried function - using causal case as primary
ARGS = {
'model': model_causal,
'is_generative': False,
'is_causal': True,
'output': output_causal,
'model_input': model_input_causal
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- absl, einops, toolz, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| try_get_index |
Ran ran on a synthesized input |
nala-cub/prost/src/baselines/input_pipeline.py code served (permissive licence) · get_code("1ebbf7526d810f87") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from absl import logging
torch.manual_seed(0)
# The function try_get_index takes:
# - tokens: a sequence that supports .index() method (list of strings)
# - stoken: a string token to search for
# - s: start index (default 0)
# - e: end index (default -1)
ARGS = {
"tokens": ["hello", "world", "foo", "bar", "baz"],
"stoken": "world",
"s": 0,
"e": -1,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- absl, numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| build_examples_from_config |
Not yet run unverified (harvested, no recorded run) |
nala-cub/prost/src/prost/prost.py code served (permissive licence) · get_code("9ad5fc60ea43e8f1") How it ran |
| create_examples_t5 |
Not yet run unverified (harvested, no recorded run) |
nala-cub/prost/src/baselines/input_pipeline.py code served (permissive licence) · get_code("774ce0489997b06c") How it ran |
| hf_auto_configure |
Not yet run unverified (harvested, no recorded run) |
nala-cub/prost/src/baselines/configs.py code served (permissive licence) · get_code("b8695e1d11efba89") How it ran |