| build_input_for_seq2seq_model |
Ran ran; our contract draft was wrong, not the code |
hltchkust/knowexpert/src/data_utils/utils.py code served (permissive licence) · get_code("3d02a7d666422095") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 3, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# The function expects:
# - labels: a tensor with at least 2 dimensions, where dim 1 is the sequence length
# - label_masks: a tensor with the same shape as labels
# The function uses .size(1) to get sequence length and .narrow() to slice along dimension 1
batch_size = 2
seqlen = 4
labels = torch.randint(0, 10, (batch_size, seqlen))
label_masks = torch.ones((batch_size, seqlen), dtype=torch.float32)
ARGS = {
"labels": labels,
"label_masks": label_masks,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 3, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| distinct_1 |
Ran ran, honoured the contract we drafted |
hltchkust/knowexpert/evaluation.py code served (permissive licence) · get_code("721918dca3d398e7") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import collections
def get_ngrams(text, n):
"""
Returns all ngrams that are in the text.
Note: this function does NOT lowercase text. If you want to lowercase, you should
do so before calling this function.
Inputs:
text: string, space-separated
n: int
Returns:
list of strings (each is a ngram, space-separated)
"""
tokens = text.split()
return [' '.join(tokens[i:i + n]) for i in range(len(tokens) - (n - 1))]
def get_ngram_counter(text, n):
"""
Returns a counter, indicating how many times each n-gram appeared in text.
Note: this function does NOT lowercase text. If you want to lowercase, you should
do so before calling this function.
Input:
text: is a string, with tokens space-separated.
Returns:
counter: mapping from each n-gram (a space-separated string) appearing in text,
to the number of times it appears
"""
ngrams = get_ngrams(text, n)
counter = collections.Counter()
counter.update(ngrams)
return counter
def _distinct_n(sample, n):
"""
Returns (total number of unique ngrams in story_text) / (total number of ngrams in story_text, including duplicates).
Text is lowercased before counting ngrams.
Returns None if there are no ngrams
"""
ngram_counter = get_ngram_counter(sample.strip().lower(), n)
if sum(ngram_counter.values()) == 0:
print('Warning: encountered a story with no {}-grams'.format(n))
print(sample.strip().lower())
print('ngram_counter: ', ngram_counter)
return None
return len(ngram_counter) / sum(ngram_counter.values())
def distinct_1(sample):
return _distinct_n(sample, 1)
# Setup code to construct valid arguments
sample_text = "the quick brown fox jumps over the lazy dog the quick brown"
ARGS = {
"sample": sample_text
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| getDataLoader |
Ran ran on a synthesized input |
hltchkust/knowexpert/src/data_utils/data_reader.py code served (permissive licence) · get_code("4cc91bb2f3ff1629") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataLoader. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
from torch.utils.data import TensorDataset
torch.manual_seed(0)
# Create a simple dataset with small tensors
# TensorDataset expects tensors and will return (x, y) tuples when iterated
x_data = torch.randn(2, 4) # 2 samples, 4 features
y_data = torch.randint(0, 2, (2,)) # 2 labels
dataset = TensorDataset(x_data, y_data)
ARGS = {
"dataset": dataset,
"bsz": 2,
"test": False
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DataLoader — a DataLoader output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataLoader output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_data_from_batch |
Ran ran; our contract draft was wrong, not the code |
hltchkust/knowexpert/src/data_utils/dialog_reader.py code served (permissive licence) · get_code("d8546d79a094a6b0") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Create batch dictionary with required keys
batch_size = 2
seq_len = 4
context_dim = 3
batch = {
'chosen_sentence': torch.randint(0, 100, (batch_size, seq_len)),
'chosen_sentence_mask': torch.ones(batch_size, seq_len),
'title': torch.randint(0, 100, (batch_size, seq_len)),
'title_mask': torch.ones(batch_size, seq_len),
'context': torch.randint(0, 100, (batch_size, seq_len)),
'context_mask': torch.ones(batch_size, seq_len),
'response': torch.randint(0, 100, (batch_size, seq_len)),
'response_mask': torch.ones(batch_size, seq_len),
'token_type': torch.randint(0, 2, (batch_size, seq_len)),
}
ARGS = {
'batch': batch,
'model_type': 'decoder_only'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_ngram_counter |
Ran ran; our contract draft was wrong, not the code |
hltchkust/knowexpert/evaluation.py code served (permissive licence) · get_code("b19a8662a8cd2ea8") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Counter. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import collections
def get_ngrams(text, n):
"""
Returns all ngrams that are in the text.
Note: this function does NOT lowercase text. If you want to lowercase, you should
do so before calling this function.
Inputs:
text: string, space-separated
n: int
Returns:
list of strings (each is a ngram, space-separated)
"""
tokens = text.split()
return [' '.join(tokens[i:i + n]) for i in range(len(tokens) - (n - 1))]
def get_ngram_counter(text, n):
"""
Returns a counter, indicating how many times each n-gram appeared in text.
Note: this function does NOT lowercase text. If you want to lowercase, you should
do so before calling this function.
Input:
text: is a string, with tokens space-separated.
Returns:
counter: mapping from each n-gram (a space-separated string) appearing in text,
to the number of times it appears
"""
ngrams = get_ngrams(text, n)
counter = collections.Counter()
counter.update(ngrams)
return counter
ARGS = {
"text": "the quick brown fox jumps over the lazy dog the quick fox",
"n": 2
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Counter — a Counter output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a Counter output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_ngrams |
Ran ran; our contract draft was wrong, not the code |
hltchkust/knowexpert/evaluation.py code served (permissive licence) · get_code("8dced10689d4eef1") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
# Define ARGS dict with valid arguments for get_ngrams function
ARGS = {
"text": "the quick brown fox jumps over the lazy dog",
"n": 2
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load |
Ran ran; our contract draft was wrong, not the code |
hltchkust/knowexpert/src/data_utils/utils.py code served (permissive licence) · get_code("1c4021c8a4208e22") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import pickle
import tempfile
import os
# Create a temporary file with some pickled data
temp_dir = tempfile.gettempdir()
temp_filename = os.path.join(temp_dir, "test_pickle_load.pkl")
# Create test data and pickle it
test_data = {"key": "value", "number": 42, "list": [1, 2, 3]}
with open(temp_filename, 'wb') as f:
pickle.dump(test_data, f)
ARGS = {
'filename': temp_filename,
'mode': 'rb'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_cmu_episodes |
Ran ran; our contract draft was wrong, not the code |
hltchkust/knowexpert/src/data_utils/cmu_dog_reader.py code served (permissive licence) · get_code("eb1b7ecf101a0251") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import os
import tempfile
from pathlib import Path
# Create a temporary directory structure with the required files
temp_dir = tempfile.mkdtemp()
datasets_dir = os.path.join(temp_dir, 'datasets-CMU_DoG')
os.makedirs(datasets_dir, exist_ok=True)
# Create sample data files for 'train' mode
mode = 'train'
# Sample source data (context with 3 parts separated by < SEP >)
src_data = [
"Hello < SEP > How are you < SEP > I'm fine",
"What 's up < SEP > Not much < SEP > Just "chilling"",
]
# Sample target data (response)
tgt_data = [
"I'm doing great!",
"Nothing much here either.",
]
# Sample knowledge data (knowledge with 3 parts separated by < SEP >)
kn_data = [
"fact1 < SEP > fact2 < SEP > fact3",
"knowledge1 < SEP > knowledge2 < SEP > knowledge3",
]
# Write files
src_file = os.path.join(datasets_dir, f'src-{mode}-tokenized.txt')
tgt_file = os.path.join(datasets_dir, f'tgt-{mode}-tokenized.txt')
kn_file = os.path.join(datasets_dir, f'knl-{mode}-tokenized.txt')
with open(src_file, 'w') as f:
f.write('\n'.join(src_data))
with open(tgt_file, 'w') as f:
f.write('\n'.join(tgt_data))
with open(kn_file, 'w') as f:
f.write('\n'.join(kn_data))
ARGS = {
'data_dir': temp_dir,
'mode': 'train',
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| pad_sents |
Ran ran; our contract draft was wrong, not the code |
hltchkust/knowexpert/src/data_utils/utils.py code served (permissive licence) · get_code("71dac7a11a85c231") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# sents parameter: a list of sentences, where each sentence is a list of integers (token ids)
# We'll create a few sentences of different lengths
sents = [
[1, 2, 3, 4, 5],
[10, 11, 12],
[20, 21, 22, 23, 24, 25, 26]
]
ARGS = {
'sents': sents,
'pad_token': 0,
'max_len': 512
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| preprocess_wow_episodes |
Ran ran; our contract draft was wrong, not the code |
hltchkust/knowexpert/src/data_utils/data_reader.py code served (permissive licence) · get_code("169dee0029e8d444") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import math
from tqdm import tqdm
def preprocess_wow_episodes(episodes, mode, history_in_context, max_episode_length=1, cal_time=False):
new_episodes = []
for episode_num, episode in enumerate(tqdm(episodes, desc='Preprocess episodes', ncols=100)):
for example_num, example in enumerate(episode):
new_examples = {'context': [], 'response': '', 'title': '', 'topic': '', 'checked_sentence': '', 'knowledge': [], 'sample_id': example_num, 'episode_num': episode_num}
history = []
if example_num != 0 and history_in_context:
max_pair_length = math.ceil(max_episode_length / 2)
start_idx = max(0, example_num - max_pair_length)
for num in range(start_idx, example_num):
if max_episode_length % 2 == 0 or num != start_idx:
history.append(episode[num]['text'].lower().strip())
history.append(episode[num]['labels'][0].lower().strip() if mode == 'train' else episode[num]['eval_labels'][0].lower().strip())
assert len(history) == max_episode_length
context = history + [example['text'].lower().strip()]
if mode == 'train':
response = example['labels'][0]
else:
response = example['eval_labels'][0]
title = example['title']
checked_sentence = example['checked_sentence']
topic = example['chosen_topic']
knowledges = example['knowledge'].rstrip().split('\n')
new_examples['context'] = context
new_examples['response'] = response.lower()
new_examples['title'] = title.lower()
new_examples['topic'] = topic
new_examples['checked_sentence'] = checked_sentence.lower()
new_examples['knowledge'] = knowledges
new_episodes.append(new_examples)
if 'test' in mode and cal_time:
print('Testing the efficiency of the model!')
with open(f'Efficiency/history_{mode}_ids.json', 'r') as f:
filter_ids = json.load(f)
temp = []
for idx in range(len(new_episodes)):
if idx in filter_ids:
temp.append(new_episodes[idx])
new_episodes = temp
print(f'There are {len(new_episodes)} samples in the final testing list.')
return new_episodes
example_train = {
'text': 'Hello there',
'labels': ['Hi how are you'],
'eval_labels': ['Hi there'],
'title': 'Greeting',
'checked_sentence': 'A polite greeting',
'chosen_topic': 'conversation',
'knowledge': 'fact1\nfact2\nfact3'
}
example_train_2 = {
'text': 'How are you doing',
'labels': ['I am doing great'],
'eval_labels': ['Doing well'],
'title': 'Status',
'checked_sentence': 'Asking about status',
'chosen_topic': 'health',
'knowledge': 'fact4\nfact5'
}
example_test = {
'text': 'What is your name',
'labels': ['My name is Bot'],
'eval_labels': ['I am an AI'],
'title': 'Identity',
'checked_sentence': 'Asking for identity',
'chosen_topic': 'identity',
'knowledge': 'fact6'
}
episode_train = [example_train, example_train_2]
episode_test = [example_test]
episodes_train = [episode_train]
episodes_test = [episode_test]
ARGS = {
'episodes': episodes_train,
'mode': 'train',
'history_in_context': False,
'max_episode_length': 1,
'cal_time': False
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| prediction_step |
Not yet run unverified (harvested, no recorded run) |
hltchkust/knowexpert/inference_task.py code served (permissive licence) · get_code("5c4e4584b5aaf31c") How it ran |
| prepare_inputs_for_generation |
Not yet run unverified (harvested, no recorded run) |
hltchkust/knowexpert/inference_task.py code served (permissive licence) · get_code("67735cd057723d31") How it ran |