| build_assign_nuggetizer_kwargs |
Ran ran on a synthesized input |
castorini/nuggetizer/src/nuggetizer/cli/execution.py code served (permissive licence) · get_code("b072bca5935fc26d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from dataclasses import dataclass
from typing import Any, Literal
ExecutionMode = Literal['sync', 'async']
@dataclass(frozen=True)
class AssignExecutionConfig:
model: str
execution_mode: ExecutionMode = 'sync'
log_level: int = 0
use_azure_openai: bool = False
use_openrouter: bool = False
reasoning_effort: str | None = None
include_trace: bool = False
include_reasoning: bool = False
redact_prompts: bool = False
quiet: bool = False
output: str = 'json'
def build_assign_nuggetizer_kwargs(config: AssignExecutionConfig) -> dict[str, Any]:
return {'assigner_model': config.model, 'log_level': config.log_level, 'use_azure_openai': config.use_azure_openai, 'use_openrouter': config.use_openrouter, 'store_trace': config.include_trace, 'store_reasoning': config.include_reasoning, 'reasoning_effort': config.reasoning_effort}
config = AssignExecutionConfig(
model='gpt-4',
execution_mode='sync',
log_level=1,
use_azure_openai=False,
use_openrouter=False,
reasoning_effort=None,
include_trace=False,
include_reasoning=False,
redact_prompts=False,
quiet=False,
output='json'
)
ARGS = {
'config': config
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| build_create_nuggetizer_kwargs |
Ran ran on a synthesized input |
castorini/nuggetizer/src/nuggetizer/cli/execution.py code served (permissive licence) · get_code("c775437a8c8fe07d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from dataclasses import dataclass
from typing import Literal
ExecutionMode = Literal['sync', 'async']
@dataclass(frozen=True)
class CreateExecutionConfig:
model: str
creator_model: str | None = None
scorer_model: str | None = None
window_size: int | None = None
max_nuggets: int | None = None
min_judgment: int = 2
execution_mode: ExecutionMode = 'sync'
log_level: int = 0
use_azure_openai: bool = False
use_openrouter: bool = False
reasoning_effort: str | None = None
include_trace: bool = False
include_reasoning: bool = False
redact_prompts: bool = False
quiet: bool = False
output: str = 'json'
# Test case 1: minimal config with only model specified
config_minimal = CreateExecutionConfig(
model='gpt-4'
)
# Test case 2: config with creator and scorer models
config_with_models = CreateExecutionConfig(
model='gpt-4',
creator_model='gpt-3.5-turbo',
scorer_model='gpt-4',
window_size=512,
max_nuggets=10,
log_level=1,
use_azure_openai=True,
use_openrouter=False,
reasoning_effort='high',
include_trace=True,
include_reasoning=True
)
# Test case 3: config without creator/scorer but with window and max_nuggets
config_with_params = CreateExecutionConfig(
model='claude-3-opus',
window_size=256,
max_nuggets=5,
log_level=2
)
ARGS = {
'config': config_minimal
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| collect_nonempty_reasoning_traces |
Ran ran on a synthesized input |
castorini/nuggetizer/src/nuggetizer/cli/adapters.py code served (permissive licence) · get_code("53b36b0cccd27107") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from collections.abc import Sequence
ARGS = {
"traces": [
"reasoning step 1",
None,
" reasoning step 2 ",
"",
"reasoning step 3",
None,
" ",
]
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| compute_elo |
Ran ran, honoured the contract we drafted |
castorini/ragnarok/src/ragnarok/api/elo.py code served (permissive licence) · get_code("d76974dd615c64e1") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
# The function compute_elo takes three parameters:
# - sa: int (score for player a, appears to be 0 or 1 for loss/win)
# - ra: int (rating for player a, defaults to INIT_RATING=1000)
# - rb: int (rating for player b, defaults to INIT_RATING=1000)
#
# The function body only does arithmetic operations on these integers,
# so we just need valid int values.
ARGS = {
"sa": 1, # Player A's score (0 for loss, 1 for win)
"ra": 1000, # Player A's current rating
"rb": 1200, # Player B's current rating
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| detect_command |
Ran ran on a synthesized input |
castorini/nuggetizer/src/nuggetizer/cli/main_support.py code served (permissive licence) · get_code("145f2c77b2912062") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
from collections.abc import Sequence
torch.manual_seed(0)
# The function detect_command takes argv: Sequence[str]
# It iterates through tokens in argv and returns the first one found in KNOWN_COMMANDS
# If none found, returns 'unknown'
# Test case 1: argv containing a known command
argv_with_command = ['--help', 'create', '--verbose']
# Test case 2: argv with no known commands
argv_without_command = ['--help', '--verbose', '--output', 'file.txt']
# Test case 3: argv with command at the start
argv_command_first = ['serve', '--port', '8080']
# Test case 4: empty argv
argv_empty = []
# Test case 5: argv with only known command
argv_only_command = ['metrics']
# Use one of these as the main test case
ARGS = {
'argv': argv_with_command
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_run_id |
Ran ran on a synthesized input |
castorini/nuggetizer/src/nuggetizer/cli/io.py code served (permissive licence) · get_code("8fa7a665d31a83dc") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
from pathlib import Path
# The function extracts a run identifier from a file path using Path.stem
# which returns the filename without extension
ARGS = {
"file_path": "/some/path/to/experiment_run_001.txt"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| make_data_artifact |
Ran ran; our contract draft was wrong, not the code |
castorini/ragnarok/src/ragnarok/cli/adapters.py code served (permissive licence) · get_code("dd91db53416878f5") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# The function make_data_artifact simply takes a name and data,
# and returns a dict with those values plus a 'kind' field.
# No special attributes or methods are called on the data object,
# so we can pass any type of data.
# Create diverse data examples
tensor_data = torch.randn(2, 4)
numpy_data = np.array([[1, 2, 3], [4, 5, 6]])
list_data = [1, 2, 3, 4, 5]
dict_data = {'key1': 'value1', 'key2': 'value2'}
ARGS = {
'name': 'my_dataset',
'data': tensor_data
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| make_file_artifact |
Ran ran; our contract draft was wrong, not the code |
castorini/ragnarok/src/ragnarok/cli/adapters.py code served (permissive licence) · get_code("7544006c2fb2eb8d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
from pathlib import Path
ARGS = {
'name': 'test_artifact',
'path': '/tmp/test_file.txt'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| prepare_output_path |
Ran ran; our contract draft was wrong, not the code |
castorini/ragnarok/src/ragnarok/cli/dispatch.py code served (permissive licence) · get_code("4a89617557cd8d60") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import argparse
from pathlib import Path
import tempfile
import os
# Define a minimal CLIError class since it's referenced in the function
class CLIError(Exception):
def __init__(self, message, exit_code=1, status=None, error_code=None, command=None, details=None):
self.message = message
self.exit_code = exit_code
self.status = status
self.error_code = error_code
self.command = command
self.details = details
super().__init__(message)
# Define EXIT_CODES dict since it's referenced
EXIT_CODES = {
'invalid_arguments': 1,
'validation_error': 2,
}
# Create a temporary directory for output
temp_dir = tempfile.mkdtemp()
output_file_path = os.path.join(temp_dir, "output.txt")
# Create args namespace with output_file and no resume/overwrite/fail_if_exists flags
args = argparse.Namespace(
output_file=output_file_path,
resume=False,
overwrite=False,
fail_if_exists=False
)
ARGS = {
'args': args,
'command': 'test_command'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| read_direct_payload |
Ran ran; our contract draft was wrong, not the code |
castorini/ragnarok/src/ragnarok/cli/dispatch.py code served (permissive licence) · get_code("f3bfd9301b14fe26") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import argparse
import json
import sys
from io import StringIO
from typing import Any, cast
# Define the CLIError class that the function references
class CLIError(Exception):
def __init__(self, message, exit_code=1, status='error', error_code='unknown', command='test', details=None):
self.message = message
self.exit_code = exit_code
self.status = status
self.error_code = error_code
self.command = command
self.details = details or {}
super().__init__(self.message)
# Define EXIT_CODES dict that the function references
EXIT_CODES = {
'invalid_arguments': 2,
'success': 0,
}
# Create a valid args namespace with stdin=True and proper JSON input
namespace = argparse.Namespace(
stdin=True,
input_json=None,
command='test_command'
)
# Set up stdin with valid JSON payload
test_payload = {'key': 'value', 'nested': {'data': 123}}
original_stdin = sys.stdin
sys.stdin = StringIO(json.dumps(test_payload))
ARGS = {
'args': namespace
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| read_jsonl |
Ran ran on a synthesized input |
castorini/nuggetizer/src/nuggetizer/cli/io.py code served (permissive licence) · get_code("719779e74d1ac660") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
import os
# Create a temporary JSONL file for testing
temp_dir = tempfile.mkdtemp()
temp_file = os.path.join(temp_dir, "test.jsonl")
# Write some test data to the JSONL file
test_data = [
{"id": 1, "name": "Alice", "value": 42},
{"id": 2, "name": "Bob", "value": 3.14},
{"id": 3, "name": "Charlie", "value": None}
]
with open(temp_file, 'w', encoding='utf-8') as f:
for item in test_data:
f.write(json.dumps(item) + '\n')
# Define the arguments for read_jsonl
ARGS = {
"file_path": temp_file
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| read_results_from_file |
Ran ran; our contract draft was wrong, not the code |
castorini/ragnarok/src/ragnarok/data.py code served (permissive licence) · get_code("e653a76dcbab5bab") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
import os
# Create a temporary JSON file with valid data structure
temp_dir = tempfile.mkdtemp()
json_file_path = os.path.join(temp_dir, "results.json")
# Create sample data matching the expected structure
sample_data = [
{
"topic": "What is machine learning?",
"topic_id": 1,
"references": ["doc1", "doc2"],
"answer": [
{
"text": "Machine learning is a subset of AI.",
"citations": [0]
},
{
"text": "It enables systems to learn from data.",
"citations": [1]
}
]
},
{
"topic": "How does deep learning work?",
"topic_id": 2,
"references": ["doc3", "doc4"],
"answer": [
{
"text": "Deep learning uses neural networks.",
"citations": [0, 1]
}
]
}
]
# Write the sample data to the temp JSON file
with open(json_file_path, 'w', encoding='utf-8') as f:
json.dump(sample_data, f)
# Define ARGS dictionary with the file_path parameter
ARGS = {
"file_path": json_file_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| resolve_write_policy |
Ran ran; our contract draft was wrong, not the code |
castorini/ragnarok/src/ragnarok/cli/dispatch.py code served (permissive licence) · get_code("33cd494e3b5331cf") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import argparse
ARGS = {
'args': argparse.Namespace(
resume=False,
overwrite=False,
fail_if_exists=False
)
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| result_to_dict |
Ran ran; our contract draft was wrong, not the code |
castorini/ragnarok/src/ragnarok/data.py code served (permissive licence) · get_code("4a11e6b65475e860") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
from dataclasses import dataclass, field
from typing import Any
@dataclass
class Query:
text: str
qid: str | int
@dataclass
class Candidate:
docid: str | int
score: float
doc: dict[str, Any]
@dataclass
class CitedSentence:
text: str
citations: list[int] = field(default_factory=list)
@dataclass
class RAGExecInfo:
prompt: Any
response: Any
input_token_count: int
output_token_count: int
reasoning: str | None = None
candidates: list[Candidate] = field(default_factory=list)
@dataclass
class Result:
query: Query
references: list[str | int] = field(default_factory=list)
answer: list[CitedSentence] = field(default_factory=list)
rag_exec_summary: RAGExecInfo | None = None
# Construct minimal valid arguments
query = Query(text="What is machine learning?", qid=42)
answer_sentences = [
CitedSentence(text="Machine learning is a field of AI.", citations=[0, 1]),
CitedSentence(text="It involves training models on data.", citations=[1, 2]),
]
rag_exec_info = RAGExecInfo(
prompt="Answer the question: What is machine learning?",
response="Machine learning is a field of AI. It involves training models on data.",
input_token_count=15,
output_token_count=25,
reasoning="Retrieved relevant documents about ML concepts",
candidates=[
Candidate(docid="doc1", score=0.95, doc={"title": "ML Basics", "text": "Introduction to ML"}),
Candidate(docid="doc2", score=0.87, doc={"title": "ML Advanced", "text": "Advanced ML techniques"}),
]
)
result = Result(
query=query,
references=["doc1", "doc2", "doc3"],
answer=answer_sentences,
rag_exec_summary=rag_exec_info
)
ARGS = {
"result": result,
"run_id": "run_001",
"include_trace": True,
"redact_prompts": False,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| setup_logging |
Ran ran on a synthesized input |
castorini/nuggetizer/src/nuggetizer/cli/logging_utils.py code served (permissive licence) · get_code("b74163bc3735c7ee") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Logger. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import logging
ARGS = {
'log_level': 1,
'quiet': False,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Logger — a Logger output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a Logger output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| unwrap_direct_create_payload |
Ran ran on a synthesized input |
castorini/nuggetizer/src/nuggetizer/cli/normalize.py code served (permissive licence) · get_code("074a68a9b4bcba9f") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from typing import Any
# Create a valid payload that matches the schema_version and structure expected
# The function looks for artifacts with 'query' and 'candidates' keys, or a 'judgments' artifact
payload = {
'schema_version': 'castorini.cli.v1',
'artifacts': [
{
'name': 'search_results',
'data': {
'query': 'what is machine learning',
'candidates': [
{'id': '1', 'text': 'Machine learning is a subset of AI'},
{'id': '2', 'text': 'ML involves training models on data'}
]
}
}
]
}
ARGS = {
'payload': payload
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| unwrap_generation_record |
Ran ran on a synthesized input |
castorini/nuggetizer/src/nuggetizer/cli/normalize.py code served (permissive licence) · get_code("74873c74cd75260c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
# Construct valid arguments for unwrap_generation_record
ARGS = {
'payload': {
'schema_version': 'castorini.cli.v1',
'artifacts': [
{
'name': 'generation-results',
'data': {
'query': 'test query',
'generated_text': 'test output',
'score': 0.95
}
}
]
}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| unwrap_nugget_record |
Ran ran on a synthesized input |
castorini/nuggetizer/src/nuggetizer/cli/normalize.py code served (permissive licence) · get_code("fd5baa7869c054bc") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
# Build a valid payload that satisfies unwrap_nugget_record
# The function expects a payload dict with:
# - schema_version: 'castorini.cli.v1'
# - artifacts: list of dicts where one has name='create-result'
# The artifact's data/value should be a dict or a list with exactly one dict
payload = {
'schema_version': 'castorini.cli.v1',
'artifacts': [
{
'name': 'create-result',
'data': {
'id': 'test-id',
'text': 'test nugget content',
'metadata': {'key': 'value'}
}
}
]
}
ARGS = {
'payload': payload
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| wants_json |
Ran ran, violated the contract we drafted |
castorini/ragnarok/src/ragnarok/cli/errors.py code served (permissive licence) · get_code("c53a0ca89658e42c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from collections.abc import Sequence
def wants_json(argv: Sequence[str]) -> bool:
for index, token in enumerate(argv):
if token == '--output' and index + 1 < len(argv):
return argv[index + 1] == 'json'
return False
# Test case 1: argv with --output json
argv_with_json = ['program', '--output', 'json', '--verbose']
ARGS = {
'argv': argv_with_json
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| detect_command |
Not yet run unverified (harvested, no recorded run) |
castorini/ragnarok/src/ragnarok/cli/errors.py code served (permissive licence) · get_code("258e9e13fcbf5b84") How it ran |
| get_processed_values |
Not yet run unverified (harvested, no recorded run) |
castorini/nuggetizer/src/nuggetizer/cli/io.py code served (permissive licence) · get_code("7f2886141fa690bf") How it ran |
| validate_generate_batch_file |
Not yet run unverified (harvested, no recorded run) |
castorini/ragnarok/src/ragnarok/cli/introspection.py code served (permissive licence) · get_code("2016f3275f584d17") How it ran |
| validate_generate_payload |
Not yet run unverified (harvested, no recorded run) |
castorini/ragnarok/src/ragnarok/cli/introspection.py code served (permissive licence) · get_code("03080e064ef748c8") How it ran |