| apply_argmax |
Ran ran on a synthesized input |
mlcommons/inference/vision/medical_imaging/3d-unet-kits19/inference_utils.py code served (permissive licence) · get_code("3867e52e00cf507d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 4, dtype family int, shape (1, 1, 4, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Create a small test image with shape (bs=1, channel=3, height=4, width=4)
# This represents a sub-volume inference result with 3 channels
test_image = np.random.randn(1, 3, 4, 4).astype(np.float32)
ARGS = {
"image": test_image
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 4, dtype family int, shape (1, 1, 4, 4) — shape (1, 1, 4, 4) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| apply_norm_map |
Ran ran on a synthesized input |
mlcommons/inference/vision/medical_imaging/3d-unet-kits19/inference_utils.py code served (permissive licence) · get_code("3cea92bceb1a2c9b") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 4, dtype family float, shape (2, 4, 3, 3) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# Create small tensors for image and norm_map
# The function performs element-wise division: image /= norm_map
# Both need to be broadcastable and have compatible shapes
image = torch.randn(2, 4, 3, 3) # batch=2, channels=4, height=3, width=3
norm_map = torch.randn(2, 4, 3, 3) + 1.0 # same shape, offset to avoid near-zero values
ARGS = {
"image": image,
"norm_map": norm_map,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 4, dtype family float, shape (2, 4, 3, 3) — shape (2, 4, 3, 3) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| calculate_retrieval_metrics |
Ran ran on a synthesized input |
mlcommons/inference/e2e-rag/accuracy_eval.py code served (permissive licence) · get_code("49b7d9bb406a744e") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from typing import Dict, List
torch.manual_seed(0)
# Setup for calculate_retrieval_metrics
# The function takes two parameters:
# 1. retrieved_urls: List[str] - a list of retrieved URL strings
# 2. expected_urls: List[str] - a list of expected/ground-truth URL strings
# Create realistic but minimal test data
retrieved_urls = [
"https://example.com/page1",
"https://example.com/page2",
"https://example.com/page3",
"https://example.com/page4"
]
expected_urls = [
"https://example.com/page2",
"https://example.com/page3",
"https://example.com/page5"
]
ARGS = {
"retrieved_urls": retrieved_urls,
"expected_urls": expected_urls
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| calculate_retrieval_metrics |
Ran ran on a synthesized input |
mlcommons/inference/e2e-rag/evaluation.py code served (permissive licence) · get_code("ad3ffba7eb3bd285") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# Construct arguments for calculate_retrieval_metrics
expected_urls = [
"https://example.com/page1",
"https://example.com/page2",
"https://example.com/page3",
]
retrieved_urls = [
"https://example.com/page2",
"https://example.com/page4",
"https://example.com/page1",
"https://example.com/page5",
"https://example.com/page3",
]
k_values = [1, 3, 5, 10]
ARGS = {
"expected_urls": expected_urls,
"retrieved_urls": retrieved_urls,
"k_values": k_values,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| call_judge |
Ran ran on a synthesized input |
mlcommons/inference/e2e-rag/accuracy_eval.py code served (permissive licence) · get_code("7a7d20420db85d84") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import os
import re
from typing import Dict, List
# Set up mock environment
os.environ['OPENROUTER_API_KEY'] = 'sk-or-v1-test-key-12345'
# Define arguments for call_judge function
ARGS = {
'question': 'What is the capital of France?',
'ground_truth': 'Paris',
'llm_answer': 'Paris',
'service_url': 'http://127.0.0.1:8123/v1/chat/completions',
'model_name': 'gpt-oss-20b',
'api_key': 'sk-or-v1-test-key-12345'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- requests — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| call_judge |
Ran ran on a synthesized input |
mlcommons/inference/e2e-rag/evaluate.py code served (permissive licence) · get_code("1ce3b34b77863636") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import requests
from unittest.mock import Mock, MagicMock
import json
# Create a mock requests.Session
session = Mock(spec=requests.Session)
# Create a mock response object
mock_response = Mock()
mock_response.json.return_value = {
'choices': [
{
'message': {
'content': json.dumps({
'score': 1,
'explanation': 'The model answer correctly answers the question.'
})
}
}
]
}
mock_response.raise_for_status.return_value = None
# Configure the session mock to return our mock response
session.post.return_value = mock_response
# Define the arguments
ARGS = {
'session': session,
'service_url': 'https://api.openrouter.ai/api/v1/chat/completions',
'model': 'gpt-3.5-turbo',
'question': 'What is the capital of France?',
'gold': 'Paris',
'pred': 'Paris'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- requests — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| compute_md5 |
Ran ran on a synthesized input |
mlcommons/inference/e2e-rag/datasetup_accuracy_eval.py code served (permissive licence) · get_code("007cfc74e0e0649b") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import os
# Create a temporary file with some content
temp_file = tempfile.NamedTemporaryFile(delete=False, mode='wb')
temp_file.write(b"Hello, this is test content for MD5 hashing.\n" * 100)
temp_file.close()
ARGS = {
"file_path": temp_file.name
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| evaluate_results |
Ran ran on a synthesized input |
mlcommons/inference/e2e-rag/accuracy_eval.py code served (permissive licence) · get_code("54a637337a4e3cb3") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import pandas as pd
import tempfile
import os
# Create a minimal TSV dataset file
tsv_content = """Prompt Answer wikipedia_link_1 wikipedia_link_2
What is the capital of France? Paris https://en.wikipedia.org/wiki/Paris https://en.wikipedia.org/wiki/France
What is the largest planet? Jupiter https://en.wikipedia.org/wiki/Jupiter https://en.wikipedia.org/wiki/Solar_System
"""
# Write to a temporary file
temp_dir = tempfile.mkdtemp()
dataset_path = os.path.join(temp_dir, 'frames_dataset.tsv')
with open(dataset_path, 'w') as f:
f.write(tsv_content)
# Create results dict mapping query_id to result_dict
# The function expects results to have queries that match the dataset
results = {
'query_1': {
'query': 'What is the capital of France?',
'llm_answer': 'Paris',
'retrieved_urls': ['https://en.wikipedia.org/wiki/Paris', 'https://en.wikipedia.org/wiki/France']
},
'query_2': {
'query': 'What is the largest planet?',
'llm_answer': 'Jupiter is the largest planet in our solar system',
'retrieved_urls': ['https://en.wikipedia.org/wiki/Jupiter']
}
}
ARGS = {
'results': results,
'dataset_path': dataset_path,
'num_workers': 2,
'judge_service_url': 'http://127.0.0.1:8123/v1/chat/completions',
'judge_model': 'gpt-oss-20b'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- pandas, requests — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| evaluate_retrieval_query |
Ran ran on a synthesized input |
mlcommons/inference/e2e-rag/evaluation.py code served (permissive licence) · get_code("305c35ca9e02614f") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
from typing import List, Dict, Any, Optional, Tuple, Union, Callable
class MockDocument:
"""Minimal Document class matching what the function expects"""
def __init__(self, page_content: str, metadata: Dict[str, Any]):
self.page_content = page_content
self.metadata = metadata
class MockRerankerModel(nn.Module):
"""Minimal reranker model"""
def __init__(self):
super().__init__()
self.linear = nn.Linear(10, 1)
def forward(self, x):
return self.linear(x)
class MockRAGDB:
"""Minimal RAG database matching function expectations"""
def __init__(self, reranker=True):
self._reranker_model = MockRerankerModel() if reranker else None
def lookup(self, query: str, k: int = 50) -> List[MockDocument]:
"""Return mock retrieval results"""
results = [
MockDocument(
page_content=f"Content for document {i}",
metadata={"original_url": f"https://example.com/doc{i}"}
)
for i in range(min(k, 5))
]
return results
def rerank(self, query: str, passages: List[str]) -> List[Tuple[str, float]]:
"""Return reranked passages with scores"""
scored = [(passage, 0.9 - i * 0.1) for i, passage in enumerate(passages)]
return sorted(scored, key=lambda x: x[1], reverse=True)
# Build arguments for evaluate_retrieval_query
torch.manual_seed(0)
rag_db = MockRAGDB(reranker=True)
query = "What is machine learning?"
expected_urls = [
"https://example.com/doc0",
"https://example.com/doc1",
"https://example.com/doc3",
]
ARGS = {
"rag_db": rag_db,
"query": query,
"expected_urls": expected_urls,
"top_k_retriever": 50,
"top_k_reranking": 10,
"verbose": True,
"no_rerank": False,
"retrieval_strategy": "fixed_k",
"print_results": False,
"return_results": False,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- retrieve, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_wikipedia_links |
Ran ran on a synthesized input |
mlcommons/inference/e2e-rag/download_docs.py code served (permissive licence) · get_code("757290e048b0c991") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
import ast
# Analyze the function to understand what it expects:
# - extract_wikipedia_links takes one parameter: `item`
# - It checks if item is a string, and if so, uses ast.literal_eval to convert it
# - Then it iterates over `item` as a list
# - For each entry in the list, it finds Wikipedia URLs using regex
# - The regex pattern matches Wikipedia URLs in the format: https://en.wikipedia.org/wiki/...
# - URLs can contain commas (legitimately), and the function handles Markdown link syntax
# Construct a valid `item` argument:
# Option 1: A list of strings containing Wikipedia URLs
# Option 2: A string representation of a list (which gets literal_eval'd)
# Let's create test data with both markdown links and plain URLs with commas
item_value = [
"Check out https://en.wikipedia.org/wiki/Quincy,_Massachusetts and https://en.wikipedia.org/wiki/John_Doe",
"[Wikipedia Link](https://en.wikipedia.org/wiki/Python_(programming_language))",
"https://en.wikipedia.org/wiki/Tick,_Tick..._Boom!",
]
ARGS = {
"item": item_value
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| fix_malformed_url |
Ran ran on a synthesized input |
mlcommons/inference/e2e-rag/download_docs.py code served (permissive licence) · get_code("4ceeb0dd1f0e41e8") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def fix_malformed_url(url: str) -> str:
if '#:~:text=' in url:
url = url.split('#:~:text=')[0]
if re.search('%25[0-9A-Fa-f]{2}', url):
url = re.sub('%25([0-9A-Fa-f]{2})', lambda m: '%' + m.group(1), url)
open_parens = url.count('(')
close_parens = url.count(')')
if open_parens > close_parens:
missing_closes = open_parens - close_parens
url += ')' * missing_closes
return url
ARGS = {
"url": "https://example.com/page(test#:~:text=hello%25world"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_config |
Ran ran; our contract draft was wrong, not the code |
mlcommons/inference/text_to_video/wan-2.2-t2v-a14b/run_inference.py code served (permissive licence) · get_code("44c6d1dcec31f489") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import yaml
import tempfile
import os
# Create a temporary YAML configuration file
temp_dir = tempfile.mkdtemp()
config_file = os.path.join(temp_dir, "config.yaml")
config_content = """
model:
type: transformer
hidden_size: 256
num_layers: 2
training:
learning_rate: 0.001
batch_size: 32
epochs: 10
data:
path: /path/to/data
split: 0.8
"""
with open(config_file, 'w') as f:
f.write(config_content)
ARGS = {
"config_path": config_file
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- yaml — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_prompts |
Ran ran on a synthesized input |
mlcommons/inference/text_to_video/wan-2.2-t2v-a14b/run_inference.py code served (permissive licence) · get_code("7b3e0f2610f104fb") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import os
# Create a temporary file with sample prompts
temp_dir = tempfile.mkdtemp()
dataset_path = os.path.join(temp_dir, "prompts.txt")
with open(dataset_path, 'w') as f:
f.write("What is machine learning?\n")
f.write("Explain neural networks\n")
f.write("\n")
f.write("How does backpropagation work?\n")
ARGS = {
"dataset_path": dataset_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_results |
Ran ran on a synthesized input |
mlcommons/inference/e2e-rag/evaluate.py code served (permissive licence) · get_code("a9ede96a0650926d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import json
import pickle
from pathlib import Path
import pandas as pd
# Create a temporary directory for test files
tmpdir = tempfile.mkdtemp()
# Setup for pickle file path
pkl_path = Path(tmpdir) / "results.pkl"
df = pd.DataFrame([
{'query': 'What is 2+2?', 'llm_answer': '4', 'success': True},
{'query': 'What is the capital of France?', 'llm_answer': 'Paris', 'success': True},
{'query': 'Bad query', 'llm_answer': 'Error', 'success': False},
])
with open(pkl_path, 'wb') as f:
pickle.dump(df, f)
# Setup for JSON file path
json_path = Path(tmpdir) / "results.json"
json_data = {
'results': [
{'prompt': 'What is AI?', 'llm_answer': 'Artificial Intelligence'},
{'prompt': 'What is ML?', 'llm_answer': 'Machine Learning'},
{'prompt': None, 'llm_answer': 'No prompt'},
]
}
json_path.write_text(json.dumps(json_data), encoding='utf-8')
# Define ARGS for both test cases
ARGS = {
'path': pkl_path # Can also test with json_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| parse_accuracy_log |
Ran ran on a synthesized input |
mlcommons/inference/e2e-rag/datasetup_accuracy_eval.py code served (permissive licence) · get_code("97cde6f95685bb1a") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
import os
# Create a temporary JSON file with valid accuracy log data
temp_dir = tempfile.mkdtemp()
log_file = os.path.join(temp_dir, "mlperf_log_accuracy.json")
# Create sample accuracy log in JSON array format
sample_data = [
{
"qsl_idx": 0,
"data": "48656c6c6f",
"seq_id": 1
},
{
"qsl_idx": 1,
"data": "576f726c64",
"seq_id": 2
},
{
"qsl_idx": 2,
"data": [65, 66, 67],
"seq_id": 3
}
]
with open(log_file, 'w') as f:
json.dump(sample_data, f)
ARGS = {
"log_path": log_file
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| postprocess |
Ran ran on a synthesized input |
mlcommons/inference/lon/network_SUT.py code served (permissive licence) · get_code("f5c0874dcb5d50cd") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# The function takes a single parameter 'query' and returns it unchanged
# Based on the function body, 'query' can be any object
# We'll use a simple dict as a representative query object
query_obj = {
"text": "sample query",
"id": 123,
"data": torch.randn(2, 4)
}
ARGS = {
"query": query_obj
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| preprocess |
Ran ran on a synthesized input |
mlcommons/inference/lon/network_SUT.py code served (permissive licence) · get_code("e05733aa3409d7b2") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# The function preprocess takes a single parameter 'query'
# and simply returns it unchanged. We need to provide a valid argument.
# Since the function doesn't specify the type, we can pass various types.
# A common use case would be passing a string, dict, tensor, or list.
# Let's use a simple string as the query.
query = "test query string"
ARGS = {
"query": query
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| validate_database |
Ran ran on a synthesized input |
mlcommons/inference/e2e-rag/datasetup_accuracy_eval.py code served (permissive licence) · get_code("7d14cc5656b8c82f") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from pathlib import Path
import tempfile
import os
torch.manual_seed(0)
# Create a minimal stand-in for VectorDB with the required attributes and methods
class MockFaissIndex:
def __init__(self, d=768):
self.d = d
self.ntotal = 5
class MockVectorStore:
def __init__(self):
self.index = MockFaissIndex(d=768)
self.index_to_docstore_id = {0: "doc0", 1: "doc1", 2: "doc2", 3: "doc3", 4: "doc4"}
self.docstore = {
"doc0": "Document 0 content",
"doc1": "Document 1 content",
"doc2": "Document 2 content",
"doc3": "Document 3 content",
"doc4": "Document 4 content"
}
class MockVectorDB:
def __init__(self, retriever_model=None, device='cpu', embedding_device='cpu', load_embeddings=False):
self.retriever_model = retriever_model
self.device = device
self.embedding_device = embedding_device
self._vector_store = MockVectorStore()
def from_serialized(self, database_path):
pass
def lookup(self, query, k=1):
# Return k mock results
return [
{"id": 0, "content": "result 0", "score": 0.95},
{"id": 1, "content": "result 1", "score": 0.87},
{"id": 2, "content": "result 2", "score": 0.79},
][:k]
# Patch the retrieve module import
import sys
from unittest.mock import MagicMock
retrieve_module = MagicMock()
retrieve_module.VectorDB = MockVectorDB
sys.modules['retrieve'] = retrieve_module
# Create temporary database path
with tempfile.TemporaryDirectory() as tmpdir:
database_path = os.path.join(tmpdir, "test_database.db")
# Create a dummy file to represent the database
with open(database_path, 'w') as f:
f.write("mock database content")
retriever_model_path = "mock_retriever_model"
ARGS = {
'database_path': database_path if os.path.exists(database_path) else str(Path(tempfile.gettempdir()) / "test_database.db"),
'retriever_model': retriever_model_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, retrieve, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| dnn_model |
Not yet run unverified (harvested, no recorded run) |
mlcommons/inference/lon/network_SUT.py code served (permissive licence) · get_code("2d9995b9375a7f1f") How it ran |
| download_frames_dataset |
Not yet run unverified (harvested, no recorded run) |
mlcommons/inference/e2e-rag/download_docs.py code served (permissive licence) · get_code("d9fde455e8c45d1d") How it ran |
| gaussian_kernel |
Not yet run unverified (harvested, no recorded run) |
mlcommons/inference/vision/medical_imaging/3d-unet-kits19/inference_utils.py code served (permissive licence) · get_code("dd5f36b69e453200") How it ran |