| cook_refs |
Ran ran; our contract draft was wrong, not the code |
microsoft/MSMARCO-Question-Answering/Evaluation/bleu_scorer.py code served (permissive licence) · get_code("7586e1d32f7325f7") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from collections import defaultdict
ARGS = {
'refs': [
'the quick brown fox jumps over the lazy dog',
'a quick brown fox leaps over a lazy dog'
],
'eff': 'shortest',
'n': 4
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| cook_test |
Ran ran; our contract draft was wrong, not the code |
microsoft/MSMARCO-Question-Answering/Evaluation/bleu_scorer.py code served (permissive licence) · get_code("019ae8bc6f98133f") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from collections import defaultdict
def precook(s, n=4, out=False):
"""Takes a string as input and returns an object that can be given to
either cook_refs or cook_test. This is optional: cook_refs and cook_test
can take string arguments as well."""
words = s.split()
counts = defaultdict(int)
for k in range(1, n + 1):
for i in range(len(words) - k + 1):
ngram = tuple(words[i:i + k])
counts[ngram] += 1
return (len(words), counts)
def cook_test(test, xxx_todo_changeme, eff=None, n=4):
"""Takes a test sentence and returns an object that
encapsulates everything that BLEU needs to know about it."""
reflen, refmaxcounts = xxx_todo_changeme
testlen, counts = precook(test, n, True)
result = {}
if eff == 'closest':
result['reflen'] = min(((abs(l - testlen), l) for l in reflen))[1]
else:
result['reflen'] = reflen
result['testlen'] = testlen
result['guess'] = [max(0, testlen - k + 1) for k in range(1, n + 1)]
result['correct'] = [0] * n
for ngram, count in counts.items():
result['correct'][len(ngram) - 1] += min(refmaxcounts.get(ngram, 0), count)
return result
# Construct test string
test_string = "the quick brown fox jumps over the lazy dog"
# Construct reference data (precook output)
ref_text = "the quick brown fox jumps over a lazy dog"
ref_length, ref_counts = precook(ref_text, n=4)
# Build xxx_todo_changeme parameter
# It expects (reflen, refmaxcounts)
# reflen should be iterable of lengths, refmaxcounts should be a defaultdict
xxx_todo_changeme_value = ([ref_length], ref_counts)
ARGS = {
'test': test_string,
'xxx_todo_changeme': xxx_todo_changeme_value,
'eff': None,
'n': 4
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| dedup |
Ran ran on a synthesized input |
microsoft/OpenKP/evaluate.py code served (permissive licence) · get_code("2a9560b1d7891c67") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 2, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# The function dedup takes a kp_list and removes duplicates while preserving order.
# It checks if each kp is hashable and can be added to a set.
# Let's create a list of hashable keypoints (tuples of floats work well as they're hashable)
kp_list = [
(1.0, 2.0, 3.0, 4.0),
(5.0, 6.0, 7.0, 8.0),
(1.0, 2.0, 3.0, 4.0), # duplicate
(9.0, 10.0, 11.0, 12.0),
(5.0, 6.0, 7.0, 8.0), # duplicate
]
ARGS = {
"kp_list": kp_list
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 2, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| generateOuput |
Ran ran on a synthesized input |
dfcf93/MSMARCO/ConversationalSearch/generateQueryEmbeddings.py code served (permissive licence) · get_code("537a8f2a5574b5d6") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# Construct responses list with required structure
responses = [
{
'query': 'what is python',
'vector': [0.1, 0.2, 0.3, 0.4]
},
{
'query': 'machine learning basics',
'vector': [0.5, 0.6, 0.7, 0.8]
}
]
ARGS = {
'responses': responses
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| getScoreEM |
Ran ran on a synthesized input |
microsoft/OpenKP/agreement.py code served (permissive licence) · get_code("5bbdf43246de0b42") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Analyzing the function:
# - candidate: iterable of labels that can be converted to strings and compared with ==
# - gold: iterable of labels that can be converted to strings and compared with ==
# - The function compares candidate and gold labels for equality
# - It returns a score as sum(bestMatch.values()) / len(gold)
# - Both parameters are used in for loops and string conversion
# Create simple label candidates and gold standard
candidate = ['label_A', 'label_B', 'label_C']
gold = ['label_A', 'label_B']
ARGS = {
'candidate': candidate,
'gold': gold
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| getScoreUnigram |
Ran ran on a synthesized input |
microsoft/OpenKP/agreement.py code served (permissive licence) · get_code("1789c669f2709136") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
# Analyzing the function:
# - candidate and gold are iterable collections of items
# - The function performs set operations on these items (using - operator)
# - It calculates len() on items and uses len() on the collections
# - Items need to support subtraction (set-like behavior)
# Using frozenset as the item type since it supports:
# - len()
# - set difference operator (-)
# - hashing (needed for dict keys via str())
candidate = [
frozenset([1, 2, 3]),
frozenset([2, 3, 4]),
]
gold = [
frozenset([1, 2]),
frozenset([2, 3, 4, 5]),
]
ARGS = {
"candidate": candidate,
"gold": gold,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_candidates |
Ran ran on a synthesized input |
amenra/ranx/ranx/fusion/bordafuse.py code served (permissive licence) · get_code("ccb4d73aaca976aa") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned List. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from numba.typed import List as TypedList
from numba.types import unicode_type
# Construct runs parameter
# runs is expected to be a list of dictionaries
# Each dictionary maps query_ids (keys) to nested dictionaries of doc_ids
run1 = {
"q1": {"doc_1": 0.9, "doc_2": 0.8, "doc_3": 0.7},
"q2": {"doc_4": 0.95, "doc_5": 0.85},
"q3": {"doc_1": 0.75, "doc_6": 0.65}
}
run2 = {
"q1": {"doc_2": 0.88, "doc_7": 0.72},
"q2": {"doc_4": 0.92, "doc_8": 0.80},
"q3": {"doc_9": 0.78}
}
runs = [run1, run2]
ARGS = {
"runs": runs
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numba, numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type List — a List output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_file_kind |
Ran ran on a synthesized input |
AmenRa/rank_eval/ranx/data_structures/qrels.py code served (permissive licence) · get_code("77f6de2e73ca8660") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import os
def get_file_kind(path: str='qrels.json', kind: str=None) -> str:
if kind is None:
kind = os.path.splitext(path)[1][1:]
kind = 'trec' if kind == 'txt' else kind
kind = 'parquet' if kind == 'parq' else kind
assert kind in {'json', 'trec', 'gz', 'parquet'}, "Error `kind` must be 'json', 'trec', 'gz', or 'parquet'"
return kind
ARGS = {
'path': 'qrels.json',
'kind': None
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_metric_label |
Ran ran on a synthesized input |
AmenRa/rank_eval/ranx/ranxhub.py code served (permissive licence) · get_code("ec8c6814aa969fd2") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
# The function takes a single parameter 'm' which is a string metric name
# The function checks if '-l' is in m, splits on '@', and looks up the metric in metric_labels dict
ARGS = {
'm': 'ndcg@10'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_stats |
Ran ran on a synthesized input |
microsoft/MSMARCO-Question-Answering/Utils/get_stats_about_length.py code served (permissive licence) · get_code("9346371b965cc92e") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
# The function expects a histogram parameter which is a dict-like object
# where keys are values and dict[value] gives the frequency/count
# The function iterates over keys and accesses histogram[value]
histogram = {
1.0: 5, # value 1.0 appears 5 times
2.0: 3, # value 2.0 appears 3 times
3.0: 2, # value 3.0 appears 2 times
}
ARGS = {
"histogram": histogram
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_url |
Ran ran on a synthesized input |
AmenRa/rank_eval/ranx/io.py code served (permissive licence) · get_code("76ec290b848f6d2d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
# Analyze the function:
# - get_url takes a single parameter 'id' of type str
# - It uses the id in an f-string with quote() to construct a URL
# - The function simply returns a formatted string
ARGS = {
'id': 'my_test_id_123'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| loadMSMARCO |
Ran ran on a synthesized input |
dfcf93/MSMARCO/ConversationalSearch/generateQuerysets.py code served (permissive licence) · get_code("19ad04a1d7cd706f") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned set. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import os
# Create temporary files with MSMARCO-like format
# The function expects files where each line has tab-separated values
# and the second column (index 1) contains query text
temp_dir = tempfile.mkdtemp()
# Create first temporary file
file1 = os.path.join(temp_dir, "file1.tsv")
with open(file1, 'w') as f:
f.write("doc_id_1\tquery one\tpassage one\n")
f.write("doc_id_2\tquery two\tpassage two\n")
f.write("doc_id_3\tquery one\tpassage three\n")
# Create second temporary file
file2 = os.path.join(temp_dir, "file2.tsv")
with open(file2, 'w') as f:
f.write("doc_id_4\tquery three\tpassage four\n")
f.write("doc_id_5\tquery two\tpassage five\n")
ARGS = {
"files": [file1, file2]
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type set — a set output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a set output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| loadNQ |
Ran ran on a synthesized input |
dfcf93/MSMARCO/ConversationalSearch/generateQuerysets.py code served (permissive licence) · get_code("f35b559c8eea08c7") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned set. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import json
import os
# Create a temporary directory with test files
temp_dir = tempfile.mkdtemp()
# Create test files with NQ-format data
test_data_1 = [
{'question_text': 'What is the capital of France?'},
{'question_text': 'Who wrote Romeo and Juliet?'},
{'question_text': 'What is the capital of France?'}, # duplicate
]
test_data_2 = [
{'question_text': 'What is the largest planet?'},
{'question_text': 'Who wrote Romeo and Juliet?'}, # duplicate from file 1
]
# Write test files
with open(os.path.join(temp_dir, 'file1.jsonl'), 'w') as f:
for item in test_data_1:
f.write(json.dumps(item) + '\n')
with open(os.path.join(temp_dir, 'file2.jsonl'), 'w') as f:
for item in test_data_2:
f.write(json.dumps(item) + '\n')
ARGS = {
'dirPath': temp_dir
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type set — a set output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a set output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| loadPassages |
Ran ran on a synthesized input |
microsoft/MSMARCO-Passage-Ranking/generateBERTPassageEmbedding.py code served (permissive licence) · get_code("4aae25f3615c1cb4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import os
# Create a temporary file with passage data
temp_file = tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.txt')
temp_file.write("1\tHello World! This is a test passage.\n")
temp_file.write("2\tAnother PASSAGE with SPECIAL@#$%^&*()characters and multiple spaces.\n")
temp_file.write("3\tthird passage - lowercase and punctuation!!!\n")
temp_file.close()
ARGS = {
'filename': temp_file.name
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| loadQueries |
Ran ran on a synthesized input |
dfcf93/MSMARCO/ConversationalSearch/generateArtificialSessions.py code served (permissive licence) · get_code("31374e7d57fb5faa") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned set. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import os
# Create a temporary file with sample query data
temp_file = tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.txt')
temp_filename = temp_file.name
# Write some sample queries to the file
sample_queries = [
"query one",
"query two",
"query three",
"another query"
]
for query in sample_queries:
temp_file.write(query + '\n')
temp_file.close()
# Define the ARGS dictionary
ARGS = {
'filename': temp_filename
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type set — a set output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a set output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| loadQueries |
Ran ran on a synthesized input |
dfcf93/MSMARCO/ConversationalSearch/generateQueryEmbeddings.py code served (permissive licence) · get_code("f65721714fdad34b") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import os
# Create a temporary file with sample query data
temp_file = tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.txt')
temp_file.write("query one\n")
temp_file.write("query two\n")
temp_file.write("query three\n")
temp_file.close()
filename = temp_file.name
ARGS = {
'filename': filename
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| loadSessions |
Ran ran on a synthesized input |
dfcf93/MSMARCO/ConversationalSearch/generateArtificialSessions.py code served (permissive licence) · get_code("daf613df77edb1b0") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import os
# Create a temporary file with tab-separated session data
temp_file = tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.txt')
temp_file.write("session1\tuser1\taction1\n")
temp_file.write("session2\tuser2\taction2\taction3\n")
temp_file.write("session3\tuser3\n")
temp_file.close()
ARGS = {
'filename': temp_file.name
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list — a list output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| loadVectors |
Ran ran on a synthesized input |
dfcf93/MSMARCO/ConversationalSearch/generateArtificialSessions.py code served (permissive licence) · get_code("695af7f11d756cdd") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
import tempfile
import os
# Create a temporary file with test data
temp_file = tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.txt')
temp_filename = temp_file.name
# Write test data: query \t space-separated vectors
test_data = [
"query1\t0.1 0.2 0.3 0.4",
"query2\t0.5 0.6 0.7 0.8",
"query3\t0.9 1.0 1.1 1.2",
"query4\t1.3 1.4 1.5 1.6"
]
for line in test_data:
temp_file.write(line + '\n')
temp_file.close()
# Define the arguments
realQueries = ["query1", "query3"]
artificialQueries = ["query2", "query4"]
ARGS = {
"filename": temp_filename,
"realQueries": realQueries,
"artificialQueries": artificialQueries
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_candidate_from_stream |
Ran ran; our contract draft was wrong, not the code |
microsoft/MSMARCO-Passage-Ranking/ms_marco_eval.py code served (permissive licence) · get_code("44dabd5275dfdd74") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import io
# Create a stream with candidate passage data
# Format: query_id\tpassage_id\trank
stream_content = """1\t101\t1
1\t102\t2
1\t103\t3
2\t201\t1
2\t202\t2
3\t301\t1
"""
# Convert string to a file-like object
f = io.StringIO(stream_content)
ARGS = {
"f": f
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_reference_from_stream |
Ran ran; our contract draft was wrong, not the code |
microsoft/MSMARCO-Passage-Ranking/ms_marco_eval.py code served (permissive licence) · get_code("b09cad901d138056") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import io
# The function reads from a stream (file-like object)
# It expects lines with tab-separated values where:
# - l[0] is query_id (int)
# - l[2] is passage_id (int)
stream_content = """1\tpassage_0\t101
1\tpassage_1\t102
2\tpassage_0\t201
3\tpassage_2\t302
3\tpassage_1\t301
"""
f = io.StringIO(stream_content)
ARGS = {
"f": f
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| normalize_answer |
Ran ran on a synthesized input |
microsoft/OpenKP/evaluate.py code served (permissive licence) · get_code("ae96a7f4fe58827f") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
import string
def normalize_answer(s):
def remove_articles(text):
return re.sub('\\b(a|an|the)\\b', ' ', text)
def white_space_fix(text):
return ' '.join(text.split())
def remove_punc(text):
exclude = set(string.punctuation)
return ''.join((ch for ch in text if ch not in exclude))
def lower(text):
return text.lower()
return ' '.join([lower(x) for x in s]).rstrip()
# The function takes a single parameter 's' which is iterable (used in a for loop)
# It should be a sequence of strings that will be lowercased, joined, and processed
s = "The quick brown fox jumps over the lazy dog! What a beautiful day, isn't it?"
ARGS = {
"s": s
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| precook |
Ran ran; our contract draft was wrong, not the code |
microsoft/MSMARCO-Question-Answering/Evaluation/bleu_scorer.py code served (permissive licence) · get_code("ec4448373d72d636") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Sonnet 4.5, prompt python_fixture_v1) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
ARGS = {
"s": "the quick brown fox jumps over the lazy dog",
"n": 4,
"out": False
}
- Setup block drafted by
- a language model, Claude Sonnet 4.5, prompt python_fixture_v1
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| process |
Ran ran on a synthesized input |
microsoft/MSMARCO-Passage-Ranking/generateBERTPassageEmbedding.py code served (permissive licence) · get_code("ab7871c7af226530") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# Analyze the function:
# - passageIDs: iterated over with len(), indexed with [i]
# - response: iterated over with len(), indexed with [i][j]
# Both need to support nested indexing and len()
passageIDs = ['passage_1', 'passage_2']
response = [
[0.95, 0.87, 0.72],
[0.54, 0.61, 0.89]
]
ARGS = {
'passageIDs': passageIDs,
'response': response
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| process |
Ran ran on a synthesized input |
dfcf93/MSMARCO/ConversationalSearch/generateQueryEmbeddingsBERT.py code served (permissive licence) · get_code("90cfc8bb296d7674") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# queryPack should be a list of items (strings or numbers that can be formatted)
# response should be a list of lists, where each inner list contains items that can be formatted
queryPack = ['query1', 'query2']
response = [['result1', 'result2'], ['result3', 'result4', 'result5']]
ARGS = {
'queryPack': queryPack,
'response': response
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| remove_empty |
Ran ran on a synthesized input |
microsoft/MSMARCO-Passage-Ranking/getstats.py code served (permissive licence) · get_code("6f73ec516845e330") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# The function remove_empty takes a_list and filters out empty items
# It checks len(item) > 0 for each item, so items need to be length-checkable
# Create a list with mixed empty and non-empty items
ARGS = {
"a_list": [
[], # empty list
[1, 2, 3], # non-empty list
torch.tensor([1, 2, 3, 4]), # non-empty tensor
[], # empty list
np.array([5, 6]), # non-empty numpy array
"", # empty string
"hello", # non-empty string
]
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list — a list output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| remove_empty |
Ran ran on a synthesized input |
microsoft/OpenKP/evaluate.py code served (permissive licence) · get_code("113dbc7ab532d6a3") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
import string
def normalize_answer(s):
def remove_articles(text):
return re.sub('\\b(a|an|the)\\b', ' ', text)
def white_space_fix(text):
return ' '.join(text.split())
def remove_punc(text):
exclude = set(string.punctuation)
return ''.join((ch for ch in text if ch not in exclude))
def lower(text):
return text.lower()
return ' '.join([lower(x) for x in s]).rstrip()
def remove_empty(a_list):
new_list = []
for i in a_list:
if len(i) > 0:
if len(i[0]) > 0:
new_list.append(normalize_answer(i))
return new_list
# Analyzing the function:
# - remove_empty takes a_list as parameter
# - It iterates through items in a_list
# - For each item i, it checks len(i) > 0
# - Then it checks len(i[0]) > 0, so i must be indexable and i[0] exists
# - If both conditions pass, it calls normalize_answer(i)
# - normalize_answer expects a string-like object that can be iterated
# - It applies string operations (lower, replace, join, etc.)
# So a_list should be a list of items where:
# - Each item has len > 0
# - Each item has a first element with len > 0
# - Each item should ultimately be a string or sequence of strings
# Construct test data: list of non-empty strings/sequences
a_list_value = [
"The quick brown fox",
"An example string with articles",
"A simple test.",
]
# Edge cases: include some empty strings to test filtering
a_list_with_mixed = [
"Hello world",
"", # This will be filtered out (len == 0)
"The test string",
]
# Build the ARGS dict
ARGS = {
"a_list": a_list_with_mixed
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| remove_punctuation |
Ran ran on a synthesized input |
microsoft/MSMARCO-Question-Answering/Evaluation/eval_exp.py code served (permissive licence) · get_code("0a2e4a7e57b0e25a") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import sys
import unicodedata
tbl = dict.fromkeys((i for i in range(sys.maxunicode) if unicodedata.category(chr(i)).startswith('P')))
def remove_punctuation(text):
return text.translate(tbl)
# Create a sample text string with punctuation
sample_text = "Hello, World! This is a test. It has punctuation: @#$%."
ARGS = {
"text": sample_text
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| whether_ynq |
Ran ran on a synthesized input |
microsoft/MSMARCO-Question-Answering/Utils/exploredata.py code served (permissive licence) · get_code("5b99609663c1da5c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
def whether_ynq(query):
YSNO_MAP = set(['is', 'can', 'does', 'are', 'do', 'should', 'did', 'will', 'was', 'has', 'could', 'were', 'would'])
tokens = query.split(' ')
head = tokens[0]
if head in YSNO_MAP and 'or' not in set(tokens):
return True
else:
return False
ARGS = {
'query': 'is this a question'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_path |
Not yet run unverified (harvested, no recorded run) |
AmenRa/rank_eval/ranx/io.py code served (permissive licence) · get_code("cdbcc024a7240d33") How it ran |
| get_tmp_path |
Not yet run unverified (harvested, no recorded run) |
AmenRa/rank_eval/ranx/io.py code served (permissive licence) · get_code("036a6050bf6f5374") How it ran |
| loadJudgements |
Not yet run unverified (harvested, no recorded run) |
microsoft/OpenKP/agreement.py code served (permissive licence) · get_code("2e000671b39ad111") How it ran |
| loadQuora |
Not yet run unverified (harvested, no recorded run) |
dfcf93/MSMARCO/ConversationalSearch/generateQuerysets.py code served (permissive licence) · get_code("16d3a24b2a035ad7") How it ran |
| load_reference |
Not yet run unverified (harvested, no recorded run) |
microsoft/MSMARCO-Passage-Ranking/ms_marco_eval.py code served (permissive licence) · get_code("52e247dd5467b4bb") How it ran |