| compute_ground_truth |
Ran ran; our contract draft was wrong, not the code |
jznmsl/siren/03_analysis_scripts/compute_ground_truth.py code served (permissive licence) · get_code("c6409fc87ba95a34") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Set random seed for reproducibility
np.random.seed(42)
rng = np.random.RandomState(42)
# Create a small binary score matrix [M, K]
# M = 10 samples, K = 4 classes
scores = np.random.randint(0, 2, size=(10, 4)).astype(np.float32)
# rho: fraction of data to use for development set
rho = 0.6
# tau: temperature parameter for softmax
tau = 0.5
# n_mc: number of Monte Carlo iterations
n_mc = 5
# rng: numpy RandomState for reproducibility
rng = np.random.RandomState(42)
ARGS = {
"scores": scores,
"rho": rho,
"tau": tau,
"n_mc": n_mc,
"rng": rng,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| compute_ground_truth |
Ran ran, honoured the contract we drafted |
jznmsl/siren/03_analysis_scripts/exp3_optimism.py code served (permissive licence) · get_code("657c4fbeeb6dac34") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
from scipy.special import expit
# Create a random number generator
rng = np.random.RandomState(42)
# Define arguments for compute_ground_truth
ARGS = {
"q_0": 0.5,
"diff_low": -1.0,
"diff_high": 1.0,
"N_gt": 100,
"rng": rng
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, scipy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_answer |
Ran ran; our contract draft was wrong, not the code |
jznmsl/siren/03_analysis_scripts/study_e_multimodel_api.py code served (permissive licence) · get_code("99767d2dbf24d8cf") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def extract_answer(text):
if not text:
return '?'
if '</think>' in text:
text = text.split('</think>')[-1]
text = text.strip()
if text and text[0] in 'ABCDEFGHIJ':
rest = text[1:].strip()
if not rest or rest[0] in '.\n,) ':
return text[0]
m = re.search('(?:answer|Answer|ANSWER)\\s*(?:is|:)\\s*([A-J])', text)
if m:
return m.group(1)
matches = re.findall('\\b([A-J])\\b', text)
return matches[-1] if matches else '?'
ARGS = {
'text': 'The answer is: B'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_answer |
Ran ran; our contract draft was wrong, not the code |
jznmsl/siren/03_analysis_scripts/study_e_multimodel_law.py code served (permissive licence) · get_code("16e4d0149abd853f") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import argparse
def extract_answer(text):
if not text:
return '?'
if '</think>' in text:
text = text.split('</think>')[-1]
text = text.strip()
if text and text[0] in 'ABCDEFGHIJ':
rest = text[1:].strip()
if not rest or rest[0] in '.\n,) ':
return text[0]
m = re.search('(?:answer|Answer|ANSWER)\\s*(?:is|:)\\s*([A-J])', text)
if m:
return m.group(1)
matches = re.findall('\\b([A-J])\\b', text)
return matches[-1] if matches else '?'
import re
ARGS = {
'text': 'The answer is: B'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_answer |
Ran ran; our contract draft was wrong, not the code |
jznmsl/siren/03_analysis_scripts/build_tensor.py code served (permissive licence) · get_code("bec005b22dd07b60") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
VALID = set('ABCDEFGHIJ')
def extract_answer(text):
if '</think>' in text:
text = text.split('</think>')[-1]
text = text.strip()
if text and text[0] in VALID:
return text[0]
m = re.search('(?:answer|Answer|ANSWER)\\s*(?:is|:)\\s*([A-J])', text)
if m:
return m.group(1)
matches = re.findall('\\b([A-J])\\b', text)
if matches:
return matches[-1]
for ch in reversed(text):
if ch in VALID:
return ch
return '?'
# Test cases for different code paths
test_cases = [
"A is the answer",
"</think>The answer is B",
"The final choice: C",
"Looking at options D and E, E is best",
"No valid answer here",
"F",
]
ARGS = {
"text": "Answer: G"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_letter |
Ran ran; our contract draft was wrong, not the code |
jznmsl/siren/03_analysis_scripts/study_e_dspy_randsearch_law.py code served (permissive licence) · get_code("ab86878635cab737") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def extract_letter(s):
if not s:
return '?'
s = str(s).strip()
if '</think>' in s:
s = s.split('</think>')[-1].strip()
if s and s[0] in 'ABCDEFGHIJ':
rest = s[1:].strip()
if not rest or rest[0] in '.\n,) ':
return s[0]
m = re.search('(?:answer|Answer|ANSWER)\\s*(?:is|:)\\s*([A-J])', s)
if m:
return m.group(1)
matches = re.findall('\\b([A-J])\\b', s)
return matches[-1] if matches else '?'
# Test case 1: Simple string with leading letter
test1 = "A. This is the answer"
# Test case 2: String with answer keyword
test2 = "The answer is B"
# Test case 3: String with think tags
test3 = "<think>some reasoning</think>C) correct"
# Test case 4: Empty string
test4 = ""
# Test case 5: String with multiple letters
test5 = "Between A and D, the answer is D"
# Test case 6: No valid letter
test6 = "No letter here"
# Test case 7: None input (will be converted to string)
test7 = None
# Test case 8: Answer with colon
test8 = "Answer: F"
ARGS = {
"s": test1
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_letter |
Ran ran; our contract draft was wrong, not the code |
jznmsl/siren/03_analysis_scripts/study_e_miprov2_law.py code served (permissive licence) · get_code("34e5242449c228b1") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def extract_letter(s):
"""Extract a single A-J letter from model output."""
if not s:
return '?'
s = str(s).strip()
if '</think>' in s:
s = s.split('</think>')[-1].strip()
if s and s[0] in 'ABCDEFGHIJ':
rest = s[1:].strip()
if not rest or rest[0] in '.\n,) ':
return s[0]
m = re.search('(?:answer|Answer|ANSWER)\\s*(?:is|:)\\s*([A-J])', s)
if m:
return m.group(1)
matches = re.findall('\\b([A-J])\\b', s)
return matches[-1] if matches else '?'
ARGS = {
's': 'Answer is: B'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| fit_rasch |
Ran ran; our contract draft was wrong, not the code |
jznmsl/siren/03_analysis_scripts/study_h_prompteval_law.py code served (permissive licence) · get_code("a76e46d8778190cc") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Create small test data for the Rasch model
# Y_full: binary response matrix (I items × J persons)
# mask: binary mask indicating which entries are observed
np.random.seed(0)
I, J = 5, 4 # 5 items, 4 persons
Y_full = np.random.randint(0, 2, size=(I, J))
mask = np.ones((I, J), dtype=bool)
# Mask out a few entries to test the masking logic
mask[0, 0] = False
mask[2, 3] = False
ARGS = {
"Y_full": Y_full,
"mask": mask,
"n_iter": 10,
"lr": 0.5,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, scipy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| format_options |
Ran ran; our contract draft was wrong, not the code |
jznmsl/siren/03_analysis_scripts/build_tensor.py code served (permissive licence) · get_code("20ef5b27be91837f") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# Analyzing the function:
# - format_options takes a single parameter 'options'
# - It iterates over enumerate(options), so options must be iterable
# - Each element is used as a string in an f-string (opt)
# - The function returns a formatted string with letters A-J as prefixes
# Valid test case: a list of strings (option descriptions)
options = [
"First option",
"Second option",
"Third option",
"Fourth option",
"Fifth option"
]
ARGS = {
"options": options
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| format_options |
Ran ran; our contract draft was wrong, not the code |
jznmsl/siren/03_analysis_scripts/study_e_dspy_randsearch_law.py code served (permissive licence) · get_code("7f9f073eb2c03fd6") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# The function takes 'options' as a parameter
# It iterates through enumerate(options) and formats each element
# So options needs to be an iterable of strings (or string-convertible objects)
options = [
"Option A",
"Option B",
"Option C",
"Option D"
]
ARGS = {
"options": options
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| format_question |
Ran ran; our contract draft was wrong, not the code |
jznmsl/siren/03_analysis_scripts/study_e_multimodel_api.py code served (permissive licence) · get_code("7dc3245651ee6f5e") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
ARGS = {
'item': {
'question': 'What is the capital of France?',
'options': ['London', 'Paris', 'Berlin', 'Madrid']
}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| generate_scores |
Ran ran; our contract draft was wrong, not the code |
jznmsl/siren/03_analysis_scripts/exp1_coverage.py code served (permissive licence) · get_code("42d8f6af2837c5f6") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 2, dtype family float, shape (5, 6) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Set random seed for reproducibility
rng = np.random.default_rng(seed=42)
# Define parameters for generate_scores
M = 5 # number of items
qualities = np.array([0.5, -0.3, 1.2, -0.8, 0.1, -1.5]) # K=6 quality values
diff_low = -2.0
diff_high = 2.0
ARGS = {
"M": M,
"qualities": qualities,
"rng": rng,
"diff_low": diff_low,
"diff_high": diff_high,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, scipy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 2, dtype family float, shape (5, 6) — shape (5, 6) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| generate_scores |
Ran ran; our contract draft was wrong, not the code |
jznmsl/siren/03_analysis_scripts/exp2_near_tie.py code served (permissive licence) · get_code("63018ddc189ff1cd") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 2, dtype family float, shape (2, 3) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Set random seed for reproducibility
np.random.seed(42)
# Create a random number generator
rng = np.random.default_rng(seed=42)
# Create qualities array: qualities should be a 1D array of quality scores
# Using shape (3,) to have a distinct size from M
qualities = np.array([0.5, -0.3, 1.2])
# M: number of items/difficulties
M = 2
# Optional parameters for difficulty range
diff_low = -2.0
diff_high = 2.0
ARGS = {
"M": M,
"qualities": qualities,
"rng": rng,
"diff_low": diff_low,
"diff_high": diff_high,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, scipy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 2, dtype family float, shape (2, 3) — shape (2, 3) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| generate_scores |
Ran ran; our contract draft was wrong, not the code |
jznmsl/siren/03_analysis_scripts/exp3_optimism.py code served (permissive licence) · get_code("9c912271312561f5") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 2, dtype family float, shape (5, 3) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
from scipy.special import expit
# Set random seed for reproducibility
np.random.seed(42)
# Create a random number generator
rng = np.random.RandomState(42)
# Define parameters
M = 5 # number of items
K = 3 # number of artifacts
q_0 = 1.5 # quality parameter
difficulties = np.array([0.2, 0.5, 1.0, 1.5, 2.0]) # M difficulties
ARGS = {
"M": M,
"K": K,
"q_0": q_0,
"rng": rng,
"difficulties": difficulties,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, scipy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 2, dtype family float, shape (5, 3) — shape (5, 3) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| generate_splits |
Ran ran; our contract draft was wrong, not the code |
jznmsl/siren/03_analysis_scripts/exp2_near_tie.py code served (permissive licence) · get_code("39d80562b160e690") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Set up random generator for reproducibility
rng = np.random.Generator(np.random.PCG64(seed=42))
# Define arguments for generate_splits
M = 10 # Total number of samples
R = 3 # Number of splits to create
rho = 0.7 # Proportion for dev split (70% dev, 30% eval)
ARGS = {
"M": M,
"R": R,
"rho": rho,
"rng": rng
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list — a list output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_true_answer |
Ran ran; our contract draft was wrong, not the code |
jznmsl/siren/03_analysis_scripts/build_tensor.py code served (permissive licence) · get_code("25605f30a28c4300") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
# Analyze the function: get_true_answer takes one parameter 'item'
# The function expects 'item' to be a dict-like object with:
# - 'answer' key (can be int or string)
# - 'answer_index' key (can be int)
# Create test items to verify the function works
item1 = {
'answer': 5, # integer answer
'answer_index': None
}
item2 = {
'answer': 'B', # string answer
'answer_index': None
}
item3 = {
'answer': ' c ', # string with whitespace
'answer_index': None
}
item4 = {
'answer_index': 3 # uses answer_index
}
item5 = {
'answer': 'invalid_answer', # returns as-is if not valid
'answer_index': None
}
ARGS = {
'item': item1
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_multimodel |
Ran ran; our contract draft was wrong, not the code |
jznmsl/siren/03_analysis_scripts/compare_models.py code served (permissive licence) · get_code("0a1998c1fc760741") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
from pathlib import Path
# Create a temporary directory to serve as 'd'
temp_dir = tempfile.mkdtemp()
d = Path(temp_dir)
# Create the study_e_results.json file with valid data
results_data = {
'model': 'test_model_v1',
'results': {
'batch_1': {
'ground_truth': [0, 1, 0, 1],
'predictions': [0.1, 0.9, 0.2, 0.8]
},
'batch_2': {
'ground_truth': [1, 0, 1],
'predictions': [0.95, 0.1, 0.85]
},
'batch_3': {
'predictions': [0.5, 0.5]
}
}
}
# Write the JSON file
results_file = d / 'study_e_results.json'
with open(results_file, 'w') as fh:
json.dump(results_data, fh)
ARGS = {
'd': d
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| make_qualities |
Ran ran, honoured the contract we drafted |
jznmsl/siren/03_analysis_scripts/exp1_coverage.py code served (permissive licence) · get_code("529cdb87aecf6052") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (5,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
ARGS = {
"K": 5,
"q_base": 0.5,
"delta_q": 1.0,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (5,) — shape (5,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| prompteval_point |
Ran ran; our contract draft was wrong, not the code |
jznmsl/siren/03_analysis_scripts/study_h_prompteval_law.py code served (permissive licence) · get_code("7b48b187da83fc0a") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Create a small synthetic Y matrix (I=5 prompts, J=4 items)
# Y contains binary ratings in [0, 1]
np.random.seed(42)
I, J = 5, 4
Y = np.random.binomial(1, 0.6, size=(I, J)).astype(np.float32)
# budget_cells: number of cells to observe in the mask
# Should be less than I*J = 20
budget_cells = 8
# seed: random seed for reproducibility
seed = 0
ARGS = {
"Y": Y,
"budget_cells": budget_cells,
"seed": seed,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, scipy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| softmax |
Ran ran, violated the contract we drafted |
jznmsl/siren/03_analysis_scripts/compute_ground_truth.py code served (permissive licence) · get_code("d3064adf95e8f16e") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Construct valid arguments for softmax function
# The function takes x (array-like) and tau (temperature parameter)
# and computes softmax with temperature scaling
# x should be a numpy array or array-like
x = np.array([1.0, 2.0, 3.0, 4.0])
# tau should be a positive scalar (temperature parameter)
tau = 1.0
ARGS = {
"x": x,
"tau": tau
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| softmax |
Ran ran, violated the contract we drafted |
jznmsl/siren/03_analysis_scripts/exp1_coverage.py code served (permissive licence) · get_code("8b1d8ac976df7a27") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (5,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Create a small 1-D numpy array with K=5 elements
x = np.array([1.0, 2.0, 3.0, 0.5, 1.5])
# Temperature parameter
tau = 0.5
# Construct the arguments dictionary
ARGS = {
"x": x,
"tau": tau,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (5,) — shape (5,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| softmax |
Ran ran, violated the contract we drafted |
jznmsl/siren/03_analysis_scripts/exp2_near_tie.py code served (permissive licence) · get_code("f7b74dbfee2cac12") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Create a small numpy array for the softmax function
x = np.array([1.0, 2.0, 3.0, 0.5])
# Create a temperature parameter
tau = 1.0
# Construct the arguments dictionary
ARGS = {
"x": x,
"tau": tau
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| softmax |
Ran ran, violated the contract we drafted |
jznmsl/siren/03_analysis_scripts/exp3_optimism.py code served (permissive licence) · get_code("72c3fa12241ebe8f") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Construct arguments for the softmax function
# The function takes:
# - x: a numpy array that will be divided, have max subtracted, exponentiated
# - tau: a temperature parameter (scalar) for scaling
# Create a small test array
x = np.array([1.0, 2.0, 3.0, 4.0])
tau = 1.0
ARGS = {
"x": x,
"tau": tau,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| two_way_balanced_sample |
Ran ran, violated the contract we drafted |
jznmsl/siren/03_analysis_scripts/study_h_prompteval_law.py code served (permissive licence) · get_code("c5507d6d47ba2f2c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 2, dtype family float, shape (5, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Set random seed for reproducibility
np.random.seed(0)
# Create a random number generator
rng = np.random.RandomState(42)
# Define parameters for two_way_balanced_sample
I = 5 # number of rows
J = 4 # number of columns
budget_cells = 12 # budget for number of True cells (less than I*J=20)
# Construct ARGS dictionary
ARGS = {
"I": I,
"J": J,
"budget_cells": budget_cells,
"rng": rng
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 2, dtype family float, shape (5, 4) — shape (5, 4) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|