| collate |
Ran ran; our contract draft was wrong, not the code |
TurkuNLP/semantic-sim/embed_data.py code served (permissive licence) · get_code("6ed105130bd872bf") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Create sample dataset items that match what collate expects
itemlist = [
{
'enc': [1, 2, 3],
'attention_mask': [1, 1, 1],
'spec_token_mask': [0, 0, 1],
'token_type_id': [0, 0, 0],
'line_idx': 0,
'txt': 'sample text 1'
},
{
'enc': [4, 5],
'attention_mask': [1, 1],
'spec_token_mask': [1, 0],
'token_type_id': [0, 1],
'line_idx': 1,
'txt': 'sample text 2'
}
]
ARGS = {
'itemlist': itemlist
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| embed_batch |
Ran ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT) |
TurkuNLP/semantic-sim/embed.py code served (permissive licence) · get_code("c97754bcbf6fdea1") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
# Create a minimal BERT-like model that returns last_hidden_state
class MinimalBertModel(nn.Module):
def __init__(self, hidden_size=4):
super().__init__()
self.hidden_size = hidden_size
self.linear = nn.Linear(hidden_size, hidden_size)
def forward(self, input_ids, attention_mask, token_type_ids):
# Return a minimal output with last_hidden_state attribute
batch_size = input_ids.shape[0]
seq_length = input_ids.shape[1]
last_hidden_state = torch.randn(batch_size, seq_length, self.hidden_size)
# Create a simple namespace-like object to hold last_hidden_state
class Output:
pass
output = Output()
output.last_hidden_state = last_hidden_state
return output
# Create a small batch with required keys
batch_size = 2
seq_length = 5
hidden_dim = 4
batch = {
'enc': torch.randint(0, 1000, (batch_size, seq_length)),
'attention_mask': torch.ones(batch_size, seq_length, dtype=torch.long),
'token_type_id': torch.zeros(batch_size, seq_length, dtype=torch.long),
'spec_token_mask': torch.zeros(batch_size, seq_length, dtype=torch.long),
}
# Create model
bert_model = MinimalBertModel(hidden_size=hidden_dim)
ARGS = {
'batch': batch,
'bert_model': bert_model,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| embed_batch |
Ran ran, honoured the contract we drafted |
TurkuNLP/semantic-sim/embed_sbert.py code served (permissive licence) · get_code("ffa9d49520a1a132") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
class MockSBERTModel:
def encode(self, texts):
# Return numpy array of embeddings
# texts can be a string or list of strings
if isinstance(texts, str):
texts = [texts]
# Return embeddings of shape (batch_size, embedding_dim)
return np.random.randn(len(texts), 4).astype(np.float32)
batch = {
'txt': ['hello world', 'test sentence']
}
sbert_model = MockSBERTModel()
ARGS = {
'batch': batch,
'sbert_model': sbert_model
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| grouper |
Ran ran; our contract draft was wrong, not the code |
TurkuNLP/semantic-sim/interactive_index.py code served (permissive licence) · get_code("2d0224d483d91857") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned zip_longest. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import itertools
def grouper(iterable, n, fillvalue=None):
"""Collect data into non-overlapping fixed-length chunks or blocks"""
args = [iter(iterable)] * n
return itertools.zip_longest(*args, fillvalue=fillvalue)
# Setup code
ARGS = {
"iterable": [1, 2, 3, 4, 5, 6, 7, 8],
"n": 3,
"fillvalue": None
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type zip_longest — a zip_longest output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a zip_longest output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| line_about_ok |
Ran ran; our contract draft was wrong, not the code |
TurkuNLP/semantic-sim/clean_data.py code served (permissive licence) · get_code("c5b52464dd6c62a5") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def line_about_ok(s):
n_rough_words = len(s.split())
if n_rough_words < 3 or n_rough_words > 50:
return (False, 'too few words')
n_punct_and_junk = len(re.sub('[a-zA-ZäöåÅÄÖ0-9:"`\'(""\',.!?() -]+', '', s))
if n_punct_and_junk / len(s) > 0.1:
return (False, 'too many punct')
n_num = len(re.sub('[^0-9]+', ' ', s).split())
if n_num > 5 and n_num / len(s) > 0.2:
return (False, 'too many nums')
return (True, None)
# Construct test arguments
ARGS = {
's': 'The quick brown fox jumps over the lazy dog today'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| line_clean |
Ran ran; our contract draft was wrong, not the code |
TurkuNLP/semantic-sim/clean_data.py code served (permissive licence) · get_code("24df7eb1a19e250b") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def remove_initial_dash(s):
return re.sub('^\\s*-+\\s*', '', s)
def line_about_ok(s):
n_rough_words = len(s.split())
if n_rough_words < 3 or n_rough_words > 50:
return (False, 'too few words')
n_punct_and_junk = len(re.sub('[a-zA-ZäöåÅÄÖ0-9:"`\'(""\',.!?() -]+', '', s))
if n_punct_and_junk / len(s) > 0.1:
return (False, 'too many punct')
n_num = len(re.sub('[^0-9]+', ' ', s).split())
if n_num > 5 and n_num / len(s) > 0.2:
return (False, 'too many nums')
return (True, None)
def line_clean(line):
line = line.strip()
if not line:
return None
line = remove_initial_dash(line)
ok, reason = line_about_ok(line)
if ok:
return (True, 'ok', line.strip())
else:
return (False, reason, line.strip())
ARGS = {
'line': 'This is a valid test line with enough words for processing'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| main_cls |
Ran ran on a synthesized input |
TurkuNLP/semantic-sim/plot.py code served (permissive licence) · get_code("7979f5424cb0defe") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
ARGS = {
'cls': '1<'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| open_possibly_gz_file |
Ran ran; our contract draft was wrong, not the code |
TurkuNLP/semantic-sim/id2txt.py code served (permissive licence) · get_code("43c3943e2bfa8ea9") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned TextIOWrapper. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import gzip
import os
import tempfile
# Create a temporary directory and files for testing
temp_dir = tempfile.mkdtemp()
# Create a plain text file
plain_file = os.path.join(temp_dir, "test.txt")
with open(plain_file, 'w') as f:
f.write("test content")
# Create a gzipped file
gz_file = os.path.join(temp_dir, "test.gz")
with gzip.open(gz_file, 'wt') as f:
f.write("gzipped content")
# Create a file that exists as both plain and .gz version
plain_dual = os.path.join(temp_dir, "dual")
with open(plain_dual, 'w') as f:
f.write("dual plain")
# Test cases: None, string path to .gz file, string path to plain file,
# string path where .gz exists, and a file-like object
import io
file_like = io.StringIO("file-like content")
ARGS = {
"f_or_name": plain_file
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type TextIOWrapper — a TextIOWrapper output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| pad_with_zero |
Ran ran, honoured the contract we drafted |
TurkuNLP/semantic-sim/embed_data.py code served (permissive licence) · get_code("b15b3e24b6e9ffbd") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family int, shape (3, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# pad_with_zero takes a single parameter 'vals' which is expected to be
# an iterable of sequences that can be converted to LongTensors
# The function converts each to LongTensor and pads them to the same length
# Create a list of lists with different lengths to demonstrate padding
vals = [
[1, 2, 3],
[4, 5],
[6, 7, 8, 9]
]
ARGS = {
"vals": vals
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 2, dtype family int, shape (3, 4) — shape (3, 4) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| random_sample_of_batches |
Ran ran, honoured the contract we drafted |
TurkuNLP/semantic-sim/create_faiss_index.py code served (permissive licence) · get_code("61a1aa1e32f835b2") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family float, shape (8, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import tempfile
import pickle
import os
torch.manual_seed(0)
# Create temporary batch files with pickled data
temp_dir = tempfile.mkdtemp()
batch_files = []
# Create 3 small batch files
for i in range(3):
batch_file = os.path.join(temp_dir, f"batch_{i}.pkl")
batch_files.append(batch_file)
with open(batch_file, 'wb') as f:
# Write 2 batches per file
for j in range(2):
sent_idx = i * 2 + j
embedding_batch = torch.randn(2, 4) # batch_size=2, embedding_dim=4
pickle.dump((sent_idx, embedding_batch), f)
ARGS = {
'batch_files': batch_files,
'proportion': 0.5
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 2, dtype family float, shape (8, 4) — shape (8, 4) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| read_results |
Ran ran; our contract draft was wrong, not the code |
TurkuNLP/semantic-sim/plot.py code served (permissive licence) · get_code("cc60092f64f71a73") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import collections
def read_results(inp):
counter = {}
for line in inp:
line = line.strip()
cls, position, sent_L, sent_R = line.split('\t')
if cls.startswith('4>') and '-rev' in cls:
continue
if cls.startswith('4<') and '-rev' not in cls:
continue
if position == 'None':
position = None
else:
position = int(position)
coarse_cls = main_cls(cls)
if position is None:
pos = 'NA'
else:
for cutoff in (1, 10, 100, 1000, 2048):
if position - 1 < cutoff:
pos = str(cutoff)
break
else:
assert False
counter.setdefault(coarse_cls, []).append(pos)
for k, pos_list in counter.items():
counter[k] = collections.Counter(pos_list)
result = {}
for k, pos_list in counter.items():
tot = sum((v for k, v in pos_list.items()))
res_list = []
for r in ('1', '10', '100', '1000', '2048', 'NA'):
res_list.append(pos_list[r] / tot * 100)
result[k] = res_list
return result
def main_cls(cls):
coarse = cls[0]
assert coarse.isnumeric(), cls
if len(cls) > 1 and cls[1] in ('<', '>'):
coarse += '<>'
return coarse
inp_lines = [
"1\t5\tleft_sent\tright_sent",
"2>\t50\tleft_sent\tright_sent",
"3<\t500\tleft_sent\tright_sent",
"4>\t1500\tleft_sent\tright_sent",
"4<-rev\t100\tleft_sent\tright_sent",
"5\tNone\tleft_sent\tright_sent",
]
ARGS = {
"inp": inp_lines
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| remove_initial_dash |
Ran ran; our contract draft was wrong, not the code |
TurkuNLP/semantic-sim/clean_data.py code served (permissive licence) · get_code("77382c479342ef23") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def remove_initial_dash(s):
return re.sub('^\\s*-+\\s*', '', s)
ARGS = {
's': ' --- hello world'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| s2hash |
Ran ran; our contract draft was wrong, not the code |
TurkuNLP/semantic-sim/index_sentences.py code served (permissive licence) · get_code("0a1348b9e265f50a") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import hashlib
def s2hash(idx_sent):
idx, sent = idx_sent
return (idx, hashlib.sha1(sent.encode('utf-8')).hexdigest()[:15])
# The function expects a tuple (idx, sent) where:
# - idx: an index (integer or similar)
# - sent: a string that will be encoded to UTF-8 and hashed
ARGS = {
'idx_sent': (0, "hello world")
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| open_possibly_gz_file |
Not yet run unverified (harvested, no recorded run) |
TurkuNLP/semantic-sim/interactive_index.py code served (permissive licence) · get_code("033b3990e98095d9") How it ran |