| acc_text |
Ran ran; our contract draft was wrong, not the code |
VisionOPD/Vision-OPD/eval/cal_acc.py code served (permissive licence) · get_code("2fb398873e75361f") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
ARGS = {
'correct': 42,
'total': 100
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| block_influence |
Ran ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT) |
icip-cas/ShortX/shortx/shortgpt/metrics.py code served (permissive licence) · get_code("5949cb967d9e2f26") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 1, dtype family float, shape (6,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Create small tensors for block_influence function
# in_hidden: shape (batch, seq_len, hidden_dim)
# out_hidden: shape (batch, seq_len, hidden_dim)
batch_size = 2
seq_len = 3
hidden_dim = 4
in_hidden = torch.randn(batch_size, seq_len, hidden_dim)
out_hidden = torch.randn(batch_size, seq_len, hidden_dim)
ARGS = {
"in_hidden": in_hidden,
"out_hidden": out_hidden,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 1, dtype family float, shape (6,) — shape (6,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_first_option |
Ran ran; our contract draft was wrong, not the code |
VisionOPD/Vision-OPD/eval/judge_qwenlm.py code served (permissive licence) · get_code("7375ac1003e75160") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def extract_first_option(text):
if not text:
return ''
match = re.search('\\(([A-Z])\\)', text)
if match:
return match.group(1)
match = re.search('([A-Z])[\\.\\)\\s]', text)
if match:
return match.group(1)
match = re.search('([A-Z])', text)
if match:
return match.group(1)
return ''
# Test cases to construct valid arguments
ARGS = {
'text': 'Choose option (A) from the list'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_mcq_option |
Ran ran; our contract draft was wrong, not the code |
VisionOPD/Vision-OPD/eval/judge_qwenlm.py code served (permissive licence) · get_code("c08ee5d994b5b602") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def extract_mcq_option(answer):
if not isinstance(answer, str) or not answer:
return ''
text = answer.strip()
pattern = '^[ (\\[]*([A-F])(?:(?=$)|[\\.\\)\\]]|(?:[\\:\\-]\\s+))'
match = re.match(pattern, text)
if match:
return match.group(1)
return ''
# Setup test cases for extract_mcq_option
ARGS = {
'answer': 'A) This is the correct answer'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| generate_config_from_args |
Ran ran; our contract draft was wrong, not the code |
VisionOPD/Vision-OPD/verl/model_merger/base_model_merger.py pointer only (licence: Apache-2.0); the stored copy get_code serves, from LeapLabTHU/OPDVR, is pointer only (licence: NONE) · get_code("8896f4e2bafe8abc") How it ranThe body of this sample is withheld: this paper's link allows the body inline (Apache-2.0), but get_code serves the stored copy, first harvested from LeapLabTHU/OPDVR, and GitHub detected no licence in that repository when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded. |
| identity_forward |
Ran ran, honoured the contract we drafted |
icip-cas/ShortX/shortx/utils/patch_strategies.py code served (permissive licence) · get_code("6650c706f3db2556") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 3, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
# Create a minimal layer object
class DummyLayer(nn.Module):
def __init__(self):
super().__init__()
self.linear = nn.Linear(4, 4)
def forward(self, hidden_states):
return self.linear(hidden_states)
layer = DummyLayer()
# Create tiny hidden states tensor (batch_size=2, hidden_dim=4)
hidden_states = torch.randn(2, 4)
ARGS = {
"layer": layer,
"hidden_states": hidden_states,
"args": (),
"kwargs": {}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 3, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| is_correct |
Ran ran, violated the contract we drafted |
VisionOPD/Vision-OPD/eval/cal_acc.py code served (permissive licence) · get_code("3d659f15a1dfc162") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
ARGS = {
'item': {
'judge': 'yes'
}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| l2_distance |
Ran ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT) |
icip-cas/ShortX/shortx/shortgpt/metrics.py code served (permissive licence) · get_code("1215305d10fb1d13") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 1, dtype family float, shape (6,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Create small tensors for in_hidden and out_hidden
# Shape: (batch_size=2, seq_len=3, hidden_dim=4)
in_hidden = torch.randn(2, 3, 4)
out_hidden = torch.randn(2, 3, 4)
ARGS = {
"in_hidden": in_hidden,
"out_hidden": out_hidden,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 1, dtype family float, shape (6,) — shape (6,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| llama_attention_only_forward |
Ran ran; our contract draft was wrong, not the code |
icip-cas/ShortX/shortx/shortgpt/adapters/llama.py code served (permissive licence) · get_code("7766ed6fe1ace2d2") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 4, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
class MockAttention(nn.Module):
def __init__(self, hidden_size):
super().__init__()
self.hidden_size = hidden_size
def forward(self, hidden_states, attention_mask=None, position_ids=None,
past_key_value=None, use_cache=None, cache_position=None,
position_embeddings=None, **kwargs):
batch_size, seq_len, hidden_size = hidden_states.shape
attn_output = torch.randn(batch_size, seq_len, hidden_size)
return attn_output, None
class MockLayerNorm(nn.Module):
def __init__(self, hidden_size):
super().__init__()
self.weight = nn.Parameter(torch.ones(hidden_size))
self.bias = nn.Parameter(torch.zeros(hidden_size))
def forward(self, x):
return x
class MockLlamaLayer(nn.Module):
def __init__(self, hidden_size):
super().__init__()
self.input_layernorm = MockLayerNorm(hidden_size)
self.self_attn = MockAttention(hidden_size)
hidden_size = 4
batch_size = 2
seq_len = 3
layer = MockLlamaLayer(hidden_size)
hidden_states = torch.randn(batch_size, seq_len, hidden_size)
attention_mask = torch.ones(batch_size, seq_len)
position_ids = torch.arange(seq_len).unsqueeze(0).expand(batch_size, -1)
past_key_value = None
use_cache = False
cache_position = None
position_embeddings = torch.randn(seq_len, hidden_size)
ARGS = {
'layer': layer,
'hidden_states': hidden_states,
'attention_mask': attention_mask,
'position_ids': position_ids,
'past_key_value': past_key_value,
'use_cache': use_cache,
'cache_position': cache_position,
'position_embeddings': position_embeddings,
'args': (),
'kwargs': {}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 4, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| llama_ffn_only_forward |
Ran ran; our contract draft was wrong, not the code |
icip-cas/ShortX/shortx/shortgpt/adapters/llama.py code served (permissive licence) · get_code("2f1ea70ec01e35ff") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
class LlamaFFN(nn.Module):
def __init__(self, hidden_size):
super().__init__()
self.hidden_size = hidden_size
self.w1 = nn.Linear(hidden_size, hidden_size * 4)
self.w2 = nn.Linear(hidden_size * 4, hidden_size)
self.w3 = nn.Linear(hidden_size, hidden_size * 4)
def forward(self, x):
return self.w2(nn.functional.silu(self.w1(x)) * self.w3(x))
class LlamaLayer(nn.Module):
def __init__(self, hidden_size):
super().__init__()
self.post_attention_layernorm = nn.LayerNorm(hidden_size)
self.mlp = LlamaFFN(hidden_size)
hidden_size = 4
batch_size = 2
layer = LlamaLayer(hidden_size)
hidden_states = torch.randn(batch_size, hidden_size)
ARGS = {
"layer": layer,
"hidden_states": hidden_states,
"args": (),
"kwargs": {}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| llama_last_attn_ffn_only_forward |
Ran ran; our contract draft was wrong, not the code |
icip-cas/ShortX/shortx/shortgpt/adapters/llama.py code served (permissive licence) · get_code("51bc546d3017eaa6") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
# Build minimal layer components
class MinimalAttention(nn.Module):
def __init__(self, hidden_size):
super().__init__()
self.hidden_size = hidden_size
def forward(self, hidden_states, attention_mask=None, position_ids=None,
past_key_value=None, use_cache=None, cache_position=None,
position_embeddings=None, **kwargs):
# Return hidden_states unchanged and None for past_key_value
return hidden_states, None
class MinimalMLP(nn.Module):
def __init__(self, hidden_size):
super().__init__()
self.linear = nn.Linear(hidden_size, hidden_size)
def forward(self, hidden_states):
return self.linear(hidden_states)
class MinimalLlamaLayer(nn.Module):
def __init__(self, hidden_size):
super().__init__()
self.hidden_size = hidden_size
self.input_layernorm = nn.LayerNorm(hidden_size)
self.post_attention_layernorm = nn.LayerNorm(hidden_size)
self.self_attn = MinimalAttention(hidden_size)
self.mlp = MinimalMLP(hidden_size)
# Create the layer instance
hidden_size = 4
seq_length = 3
batch_size = 2
llama_layer = MinimalLlamaLayer(hidden_size)
# Create input tensors
hidden_states = torch.randn(batch_size, seq_length, hidden_size)
attention_mask = torch.ones(batch_size, seq_length, dtype=torch.long)
position_ids = torch.arange(seq_length, dtype=torch.long).unsqueeze(0).expand(batch_size, -1)
past_key_value = None
use_cache = False
cache_position = torch.arange(seq_length, dtype=torch.long)
position_embeddings = torch.randn(seq_length, hidden_size)
ARGS = {
"self": llama_layer,
"hidden_states": hidden_states,
"attention_mask": attention_mask,
"position_ids": position_ids,
"past_key_value": past_key_value,
"use_cache": use_cache,
"cache_position": cache_position,
"position_embeddings": position_embeddings,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| make_sample_uid |
Ran ran; our contract draft was wrong, not the code |
VisionOPD/Vision-OPD/eval/infer.py code served (permissive licence) · get_code("16c462aa165ced48") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import hashlib
import json
def make_sample_uid(item, benchmark):
for key in ('sample_uid', 'uid', 'index', 'question_id', 'id'):
value = item.get(key)
if value is not None and str(value) != '':
return f'{benchmark}:{key}:{value}'
stable_obj = {'benchmark': benchmark, 'images': item.get('images') or [], 'query': item.get('query', '')}
raw = json.dumps(stable_obj, ensure_ascii=False, sort_keys=True)
return 'sha1:' + hashlib.sha1(raw.encode('utf-8')).hexdigest()
# Test case 1: item with sample_uid
item_with_uid = {
'sample_uid': '12345',
'query': 'test query',
'images': ['img1.jpg', 'img2.jpg']
}
benchmark_name = 'test_benchmark'
ARGS = {
'item': item_with_uid,
'benchmark': benchmark_name
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| pick_image_path |
Ran ran; our contract draft was wrong, not the code |
VisionOPD/Vision-OPD/eval/infer.py code served (permissive licence) · get_code("ebfbc91b7ed393e0") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# The function pick_image_path takes an item parameter
# It calls item.get('images') which expects a dict with a 'get' method
# It returns the first element of the result (or empty string if None)
# Create a valid item dict
item_with_images = {
'images': ['path/to/image1.jpg', 'path/to/image2.jpg']
}
ARGS = {
'item': item_with_images
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| prepare_zoombench |
Ran ran; our contract draft was wrong, not the code |
VisionOPD/Vision-OPD/eval/prepare_data.py code served (permissive licence) · get_code("d9c5a589cd427f50") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from pathlib import Path
import tempfile
from unittest.mock import patch, MagicMock
import pyarrow as pa
import pyarrow.parquet as pq
# Create a temporary directory for the test
temp_dir = tempfile.mkdtemp()
out_dir = Path(temp_dir)
# Create mock data structure that matches what the function expects
mock_data = {
'id': [1, 2],
'query': ['What is this?', 'Describe the image'],
'response': ['This is a test', 'A test image'],
'image': [
{'bytes': b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00'}, # minimal JPEG header
{'bytes': b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00'}
],
'crop_image': [
{'bytes': b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00'},
None
]
}
# Create a PyArrow table
table = pa.table(mock_data)
# Write the table to a parquet file
local_dir = out_dir / 'ZoomBench_data' / 'data'
local_dir.mkdir(parents=True, exist_ok=True)
parquet_path = local_dir / 'test.parquet'
pq.write_table(table, str(parquet_path))
# Mock the snapshot_download function to avoid network calls
def mock_snapshot_download(repo_id, repo_type=None, local_dir=None):
return local_dir
ARGS = {
'out_dir': out_dir
}
# Patch snapshot_download to use our mock
import huggingface_hub
original_snapshot_download = huggingface_hub.snapshot_download
huggingface_hub.snapshot_download = mock_snapshot_download
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- huggingface_hub, pyarrow — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| process_multi_modal_inputs_for_minicpmo |
Ran ran; our contract draft was wrong, not the code |
VisionOPD/Vision-OPD/verl/utils/dataset/vision_utils.py code served (permissive licence) · get_code("ee4a16e546e12473") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Construct tiny tensors for the function parameters
batch_size = 2
seq_length = 5
vocab_size = 100
# input_ids: token IDs for the batch
input_ids = torch.randint(0, vocab_size, (batch_size, seq_length))
# attention_mask: binary mask with left padding (argmax will find first 1)
attention_mask = torch.tensor([
[0, 0, 1, 1, 1],
[0, 1, 1, 1, 1]
], dtype=torch.long)
# position_ids: positional embeddings
position_ids = torch.arange(seq_length).unsqueeze(0).expand(batch_size, -1)
# cu_seqlens: cumulative sequence lengths for flash attention
cu_seqlens = torch.tensor([0, seq_length, 2 * seq_length], dtype=torch.long)
# multi_modal_inputs: dictionary with image-related data
# image_bound: list of tensors with image token boundaries per sample
image_bound_0 = torch.tensor([[1, 3], [4, 6]], dtype=torch.long) # 2 images in first sample
image_bound_1 = torch.tensor([[2, 4]], dtype=torch.long) # 1 image in second sample
# pixel_values: list of lists of image tensors
pixel_values_0 = [torch.randn(3, 8, 8), torch.randn(3, 8, 8)] # 2 images for first sample
pixel_values_1 = [torch.randn(3, 8, 8)] # 1 image for second sample
# tgt_sizes: list of target sizes per sample
tgt_sizes_0 = torch.tensor([8, 8], dtype=torch.long) # size info for first image
tgt_sizes_1 = torch.tensor([8, 8], dtype=torch.long) # size info for second image
multi_modal_inputs = {
'image_bound': [image_bound_0, image_bound_1],
'pixel_values': [pixel_values_0, pixel_values_1],
'tgt_sizes': [tgt_sizes_0, tgt_sizes_1]
}
ARGS = {
'input_ids': input_ids,
'attention_mask': attention_mask,
'position_ids': position_ids,
'cu_seqlens': cu_seqlens,
'multi_modal_inputs': multi_modal_inputs
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| qwen_attention_only_forward |
Ran ran; our contract draft was wrong, not the code |
icip-cas/ShortX/shortx/shortgpt/adapters/qwen.py code served (permissive licence) · get_code("cf3900721c85d810") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 3, dtype family float, shape (2, 3, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
class MockLayerNorm(nn.Module):
def __init__(self, hidden_size):
super().__init__()
self.weight = nn.Parameter(torch.ones(hidden_size))
self.bias = nn.Parameter(torch.zeros(hidden_size))
def forward(self, x):
return x * self.weight + self.bias
class MockSelfAttention(nn.Module):
def __init__(self, hidden_size):
super().__init__()
self.hidden_size = hidden_size
def forward(self, hidden_states, attention_mask=None, position_ids=None,
past_key_value=None, use_cache=None, cache_position=None,
position_embeddings=None, **kwargs):
batch_size, seq_len, hidden_size = hidden_states.shape
attn_output = torch.randn(batch_size, seq_len, hidden_size)
return attn_output, None
class MockQwenLayer(nn.Module):
def __init__(self, hidden_size):
super().__init__()
self.input_layernorm = MockLayerNorm(hidden_size)
self.self_attn = MockSelfAttention(hidden_size)
hidden_size = 4
batch_size = 2
seq_len = 3
layer = MockQwenLayer(hidden_size)
hidden_states = torch.randn(batch_size, seq_len, hidden_size)
attention_mask = torch.ones(batch_size, 1, seq_len, seq_len)
position_ids = torch.arange(seq_len).unsqueeze(0).expand(batch_size, -1)
past_key_value = None
use_cache = False
cache_position = torch.arange(seq_len)
position_embeddings = torch.randn(seq_len, hidden_size)
ARGS = {
"layer": layer,
"hidden_states": hidden_states,
"attention_mask": attention_mask,
"position_ids": position_ids,
"past_key_value": past_key_value,
"use_cache": use_cache,
"cache_position": cache_position,
"position_embeddings": position_embeddings,
"args": (),
"kwargs": {}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 3, dtype family float, shape (2, 3, 4) — shape (2, 3, 4) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| qwen_ffn_only_forward |
Ran ran; our contract draft was wrong, not the code |
icip-cas/ShortX/shortx/shortgpt/adapters/qwen.py code served (permissive licence) · get_code("361a3caeaaaf446b") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 3, dtype family float, shape (2, 3, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
class SimpleLayerNorm(nn.Module):
def __init__(self, hidden_size):
super().__init__()
self.weight = nn.Parameter(torch.ones(hidden_size))
self.bias = nn.Parameter(torch.zeros(hidden_size))
def forward(self, x):
mean = x.mean(-1, keepdim=True)
std = x.std(-1, keepdim=True)
return (x - mean) / (std + 1e-6) * self.weight + self.bias
class SimpleMLP(nn.Module):
def __init__(self, hidden_size, intermediate_size):
super().__init__()
self.gate_proj = nn.Linear(hidden_size, intermediate_size)
self.up_proj = nn.Linear(hidden_size, intermediate_size)
self.down_proj = nn.Linear(intermediate_size, hidden_size)
def forward(self, x):
gate = torch.sigmoid(self.gate_proj(x))
up = self.up_proj(x)
hidden = gate * up
return self.down_proj(hidden)
class QwenLayer(nn.Module):
def __init__(self, hidden_size=4, intermediate_size=8):
super().__init__()
self.post_attention_layernorm = SimpleLayerNorm(hidden_size)
self.mlp = SimpleMLP(hidden_size, intermediate_size)
def forward(self, hidden_states):
return hidden_states
hidden_size = 4
layer = QwenLayer(hidden_size=hidden_size, intermediate_size=8)
hidden_states = torch.randn(2, 3, hidden_size)
ARGS = {
'layer': layer,
'hidden_states': hidden_states,
'args': (),
'kwargs': {}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 3, dtype family float, shape (2, 3, 4) — shape (2, 3, 4) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| qwen_last_attn_ffn_only_forward |
Ran ran; our contract draft was wrong, not the code |
icip-cas/ShortX/shortx/shortgpt/adapters/qwen.py code served (permissive licence) · get_code("5649b66daf070ead") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 3, dtype family float, shape (2, 5, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
# Build minimal nn.Module for self_attn
class MockAttention(nn.Module):
def forward(self, hidden_states, attention_mask, position_ids, past_key_value, use_cache, cache_position, position_embeddings, **kwargs):
# Return hidden_states unchanged and None for second return value
return hidden_states, None
# Build minimal nn.Module for input_layernorm
class MockLayerNorm(nn.Module):
def forward(self, x):
return x
# Build minimal nn.Module for post_attention_layernorm
class MockPostAttentionLayerNorm(nn.Module):
def forward(self, x):
return x
# Build minimal nn.Module for mlp
class MockMLP(nn.Module):
def forward(self, x):
return x
# Build the parent module that contains self_attn, input_layernorm, post_attention_layernorm, mlp
class MockDecoderLayer(nn.Module):
def __init__(self):
super().__init__()
self.input_layernorm = MockLayerNorm()
self.self_attn = MockAttention()
self.post_attention_layernorm = MockPostAttentionLayerNorm()
self.mlp = MockMLP()
# Create an instance to use as 'self'
mock_layer = MockDecoderLayer()
# Construct arguments
batch_size = 2
seq_len = 5
hidden_dim = 4
num_heads = 2
hidden_states = torch.randn(batch_size, seq_len, hidden_dim)
attention_mask = torch.ones(batch_size, 1, seq_len, seq_len)
position_ids = torch.arange(seq_len).unsqueeze(0).expand(batch_size, -1)
past_key_value = None
use_cache = False
cache_position = torch.arange(seq_len)
position_embeddings = torch.randn(seq_len, hidden_dim)
ARGS = {
"self": mock_layer,
"hidden_states": hidden_states,
"attention_mask": attention_mask,
"position_ids": position_ids,
"past_key_value": past_key_value,
"use_cache": use_cache,
"cache_position": cache_position,
"position_embeddings": position_embeddings,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 3, dtype family float, shape (2, 5, 4) — shape (2, 5, 4) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| resolve_benchmark_json |
Ran ran; our contract draft was wrong, not the code |
VisionOPD/Vision-OPD/eval/prepare_data.py code served (permissive licence) · get_code("225a4721e03c0bbb") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import sys
ARGS = {
'benchmark': 'zoombench'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| should_retry_existing_record |
Ran ran, violated the contract we drafted |
VisionOPD/Vision-OPD/eval/infer.py code served (permissive licence) · get_code("36cd5e00420a29f4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
# Test case 1: Normal dict with valid model_answer
item_valid = {
'model_answer': 'This is a valid response'
}
# Test case 2: Dict with empty string model_answer
item_empty = {
'model_answer': ' '
}
# Test case 3: Dict with API_ERROR
item_api_error = {
'model_answer': '[API_ERROR] Something went wrong'
}
# Test case 4: Dict with FUTURE_ERROR
item_future_error = {
'model_answer': '[FUTURE_ERROR] Future error occurred'
}
# Test case 5: Dict with non-string model_answer
item_non_string = {
'model_answer': 12345
}
# Test case 6: Dict without model_answer key
item_missing = {
'other_field': 'value'
}
# Primary test case - a dict with model_answer key
ARGS = {
'item': {
'model_answer': 'This is a test response'
}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| variance_change |
Ran ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT) |
icip-cas/ShortX/shortx/shortgpt/metrics.py code served (permissive licence) · get_code("a1a3b6eb1f6a9fa4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 1, dtype family float, shape (6,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Create small tensors with shape (batch=2, seq_len=3, hidden_dim=4)
# in_hidden and out_hidden are used to compute variance across features
in_hidden = torch.randn(2, 3, 4)
out_hidden = torch.randn(2, 3, 4)
ARGS = {
"in_hidden": in_hidden,
"out_hidden": out_hidden,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 1, dtype family float, shape (6,) — shape (6,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| calc_hrbench |
Not yet run unverified (harvested, no recorded run) |
VisionOPD/Vision-OPD/eval/cal_acc.py code served (permissive licence) · get_code("106f8da519c5a301") How it ran |
| pope_extract |
Not yet run unverified (harvested, no recorded run) |
VisionOPD/Vision-OPD/eval/judge_qwenlm.py code served (permissive licence) · get_code("5a8d7a90cea59181") How it ran |
| prepare_vstar |
Not yet run unverified (harvested, no recorded run) |
VisionOPD/Vision-OPD/eval/prepare_data.py code served (permissive licence) · get_code("d982d36f7304ff08") How it ran |