| auto_eval_by_claude |
Ran ran; our contract draft was wrong, not the code |
wi-pi/webspeval_code/judge/eval_binary_with_gt_text_claude.py code served (permissive licence) · get_code("8eca7eb89de5f8fe") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import os
import json
import tempfile
import base64
from unittest.mock import Mock, MagicMock
# Create a temporary directory structure for testing
temp_dir = tempfile.mkdtemp()
process_dir = temp_dir
# Create interact_messages.json
interact_messages = [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "Now given a task: Click the button labeled 'Submit' on the form. Please interact with the system to complete this task."
},
{
"role": "assistant",
"content": "I will help you click the Submit button."
},
{
"role": "user",
"content": "The action has been completed."
}
]
with open(os.path.join(process_dir, 'interact_messages.json'), 'w') as f:
json.dump(interact_messages, f)
# Create thoughts.json
thoughts_data = [
{
"iteration": 1,
"thought": "I need to locate the Submit button on the form",
"action": "Looking for button element with text 'Submit'"
},
{
"iteration": 2,
"thought": "Found the Submit button, now I'll click it",
"action": "Clicking on the Submit button at coordinates (200, 150)"
},
{
"iteration": 3,
"thought": "Form submitted successfully",
"action": "Waiting for confirmation"
}
]
with open(os.path.join(process_dir, 'thoughts.json'), 'w') as f:
json.dump(thoughts_data, f)
# Create dummy screenshot files
screenshot_data = b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x01\x00\x00\x00\x01\x08\x06\x00\x00\x00\x1f\x15\xc4\x89\x00\x00\x00\nIDATx\x9cc\x00\x01\x00\x00\x05\x00\x01\r\n-\xb4\x00\x00\x00\x00IEND\xaeB`\x82'
for i in [1, 2, 3]:
screenshot_path = os.path.join(process_dir, f'screenshot{i}.png')
with open(screenshot_path, 'wb') as f:
f.write(screenshot_data)
# Create mock Claude client
claude_client = Mock()
claude_client.messages = Mock()
claude_client.messages.create = MagicMock(return_value=Mock(
content=[Mock(text="The agent successfully completed the task by clicking the Submit button. The evaluation shows that all steps were followed correctly.")]
))
# Define ground truth steps
ground_truth_steps = [
"Navigate to the form page",
"Locate the Submit button",
"Click the Submit button",
"Verify form submission confirmation"
]
ARGS = {
'process_dir': process_dir,
'claude_client': claude_client,
'api_model': 'claude-3-5-sonnet-20241022',
'system_prompt': 'You are an AI assistant evaluating the correctness of agent actions in completing web-based tasks.',
'ground_truth_steps': ground_truth_steps,
'has_timeout': False,
'temperature': 1.0
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| check_for_api_errors |
Ran ran; our contract draft was wrong, not the code |
wi-pi/webspeval_code/judge/eval_binary_with_gt_text_claude.py code served (permissive licence) · get_code("2881b7afbbf74c18") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import os
import tempfile
# Create a temporary directory with an agent.log file
temp_dir = tempfile.mkdtemp()
log_file_path = os.path.join(temp_dir, 'agent.log')
# Write test content to the log file
with open(log_file_path, 'w') as f:
f.write("Some agent execution log\n")
f.write("Processing task...\n")
f.write("Task execution time limit (180s) exceeded\n")
f.write("API call error occurred\n")
ARGS = {
'process_dir': temp_dir
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| encode_image_base64 |
Ran ran; our contract draft was wrong, not the code |
wi-pi/webspeval_code/judge/eval_binary_with_gt_text_claude.py code served (permissive licence) · get_code("66d017c4b55d2321") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import base64
import tempfile
import os
# Create a temporary image file for testing
temp_dir = tempfile.mkdtemp()
image_path = os.path.join(temp_dir, "test_image.png")
# Create a minimal valid PNG file (1x1 pixel)
# This is the smallest valid PNG in bytes
png_bytes = bytes([
0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A, # PNG signature
0x00, 0x00, 0x00, 0x0D, 0x49, 0x48, 0x44, 0x52, # IHDR chunk
0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, # 1x1 dimensions
0x08, 0x02, 0x00, 0x00, 0x00, 0x90, 0x77, 0x53, # bit depth, color type
0xDE, 0x00, 0x00, 0x00, 0x0C, 0x49, 0x44, 0x41, # CRC, IDAT chunk
0x54, 0x08, 0x99, 0x01, 0x01, 0x00, 0x00, 0xFE, # compressed data
0xFF, 0x00, 0x00, 0x00, 0x02, 0x00, 0x01, 0xE5, # more data
0x27, 0xDE, 0xFC, 0x00, 0x00, 0x00, 0x00, 0x49, # CRC
0x45, 0x4E, 0x44, 0xAE, 0x42, 0x60, 0x82 # IEND chunk
])
with open(image_path, 'wb') as f:
f.write(png_bytes)
ARGS = {
'image_path': image_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| fetch_browser_info |
Ran ran; our contract draft was wrong, not the code |
wi-pi/webspeval_code/src/utils_webarena.py code served (permissive licence) · get_code("282e8a7fe50932e3") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from typing import Any
from unittest.mock import Mock, MagicMock
# Create a mock browser object that implements the required methods
mock_browser = Mock()
# Mock the execute_cdp_cmd method to return a DOM snapshot structure
mock_browser.execute_cdp_cmd.return_value = {
'documents': [
{
'layout': {
'bounds': [
[10, 20, 1920, 1080],
[50, 60, 500, 300],
[100, 150, 800, 600]
]
}
}
]
}
# Mock the get_window_size method
mock_browser.get_window_size.return_value = {'width': 1920, 'height': 1080}
# Mock the execute_script method to return appropriate values
def execute_script_side_effect(script):
if 'pageYOffset' in script:
return 0.0
elif 'pageXOffset' in script:
return 0.0
elif 'screen.width' in script:
return 1920.0
elif 'screen.height' in script:
return 1080.0
elif 'devicePixelRatio' in script:
return 1.0
return None
mock_browser.execute_script.side_effect = execute_script_side_effect
ARGS = {
'browser': mock_browser
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| format_msg_claude |
Ran ran; our contract draft was wrong, not the code |
wi-pi/webspeval_code/src/api_utils.py code served (permissive licence) · get_code("a5524a41e66f9df4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import base64
from PIL import Image
import io
# Create a small base64-encoded PNG image for web_img_b64
img = Image.new('RGB', (10, 10), color='red')
img_bytes = io.BytesIO()
img.save(img_bytes, format='PNG')
img_bytes.seek(0)
web_img_b64 = base64.b64encode(img_bytes.read()).decode('utf-8')
# Define ARGS with all parameters
ARGS = {
'it': 1,
'init_msg': "Please help me navigate this webpage.",
'pdf_obs': None,
'warn_obs': "",
'web_img_b64': web_img_b64,
'web_text': "button: Click here\ninput: Search box",
'tabs_info': {
'tab_1': {
'title': 'Google',
'url': 'https://www.google.com',
'is_current': True
},
'tab_2': {
'title': 'GitHub',
'url': 'https://www.github.com',
'is_current': False
}
}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- PIL — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| format_msg_gpt5 |
Ran ran; our contract draft was wrong, not the code |
wi-pi/webspeval_code/src/api_utils.py code served (permissive licence) · get_code("76510fc830c652c8") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import base64
import io
from PIL import Image
# Create a small base64 encoded image for web_img_b64
img = Image.new('RGB', (100, 100), color='red')
img_bytes = io.BytesIO()
img.save(img_bytes, format='PNG')
img_bytes.seek(0)
web_img_b64 = base64.b64encode(img_bytes.read()).decode('utf-8')
# Define arguments for format_msg_gpt5
ARGS = {
'it': 1,
'init_msg': 'You are a helpful assistant. Please help me interact with this webpage.',
'pdf_obs': None,
'warn_obs': '',
'web_img_b64': web_img_b64,
'web_text': '<button id="btn1">Click me</button>\n<input id="input1" type="text">Enter text here</input>\n<div id="div1">Some content</div>',
'tabs_info': {
'tab1': {
'title': 'Google Search',
'url': 'https://www.google.com',
'is_current': True
},
'tab2': {
'title': 'GitHub',
'url': 'https://www.github.com',
'is_current': False
}
}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- PIL — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| format_msg_text_only_gpt5 |
Ran ran; our contract draft was wrong, not the code |
wi-pi/webspeval_code/src/api_utils.py code served (permissive licence) · get_code("cf52e4eb5297bbbc") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
# Setup for format_msg_text_only_gpt5
# Parameter: it (iteration number)
it = 1
# Parameter: init_msg (initial message string)
init_msg = "Please help me navigate this webpage."
# Parameter: pdf_obs (PDF observation string or empty)
pdf_obs = ""
# Parameter: warn_obs (warning observation string)
warn_obs = "Warning: Element not found"
# Parameter: ac_tree (accessibility tree as string)
ac_tree = """
RootWebArea "Example Page"
navigation "Main Navigation"
link "Home" <click>
link "About" <click>
main "Main Content"
heading "Welcome"
button "Submit" <click>
"""
# Parameter: tabs_info (dictionary of open tabs information)
tabs_info = {
"tab_1": {
"title": "Google Search",
"url": "https://www.google.com",
"is_current": True
},
"tab_2": {
"title": "GitHub",
"url": "https://www.github.com",
"is_current": False
}
}
ARGS = {
"it": it,
"init_msg": init_msg,
"pdf_obs": pdf_obs,
"warn_obs": warn_obs,
"ac_tree": ac_tree,
"tabs_info": tabs_info
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_bounding_client_rect |
Ran ran; our contract draft was wrong, not the code |
wi-pi/webspeval_code/src/utils_webarena.py code served (permissive licence) · get_code("040b32129bf67f60") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from unittest.mock import Mock, MagicMock
# Create a mock browser object that implements the required methods
browser = Mock()
# Mock the execute_cdp_cmd method to return appropriate responses
def mock_execute_cdp_cmd(command, params):
if command == 'DOM.resolveNode':
return {
'object': {
'objectId': 'mock_object_id_12345'
}
}
elif command == 'Runtime.callFunctionOn':
return {
'result': {
'value': {
'x': 10.5,
'y': 20.3,
'width': 100.0,
'height': 50.0,
'top': 20.3,
'left': 10.5,
'bottom': 70.3,
'right': 110.5
}
}
}
return {}
browser.execute_cdp_cmd = mock_execute_cdp_cmd
backend_node_id = "42"
ARGS = {
'browser': browser,
'backend_node_id': backend_node_id
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_element_in_viewport_ratio |
Ran ran, honoured the contract we drafted |
wi-pi/webspeval_code/src/utils_webarena.py code served (permissive licence) · get_code("0b27cebef7f8500d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
# Construct BrowserConfig TypedDict
config = {
'win_top_bound': 0.0,
'win_left_bound': 0.0,
'win_width': 1920.0,
'win_height': 1080.0,
'win_right_bound': 1920.0,
'win_lower_bound': 1080.0,
'device_pixel_ratio': 1.0
}
# Define arguments for get_element_in_viewport_ratio
ARGS = {
'elem_left_bound': 100.0,
'elem_top_bound': 200.0,
'width': 300.0,
'height': 150.0,
'config': config
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_tabs_info |
Ran ran; our contract draft was wrong, not the code |
wi-pi/webspeval_code/src/agent/actions.py code served (permissive licence) · get_code("602e1a57b115b9ef") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import logging
from unittest.mock import Mock, MagicMock
# Create a mock driver/task object that simulates Selenium WebDriver behavior
driver_task = Mock()
# Set up window handles
handle1 = "CDwindow-1234567890"
handle2 = "CDwindow-0987654321"
handle3 = "CDwindow-1111111111"
window_handles = [handle1, handle2, handle3]
driver_task.window_handles = window_handles
driver_task.current_window_handle = handle2
# Create a side effect function for switch_to.window to track which window is active
current_window = [handle2]
def switch_window(handle):
current_window[0] = handle
driver_task.switch_to.window = Mock(side_effect=switch_window)
# Mock the properties that change based on which window is active
def get_title():
if current_window[0] == handle1:
return "Google - Search"
elif current_window[0] == handle2:
return "GitHub"
else:
return "Stack Overflow"
def get_current_url():
if current_window[0] == handle1:
return "https://www.google.com/?q=python"
elif current_window[0] == handle2:
return "https://github.com/user/repo"
else:
return "https://stackoverflow.com/questions/tagged/python"
# Use PropertyMock to make title and current_url return different values
type(driver_task).title = property(lambda self: get_title())
type(driver_task).current_url = property(lambda self: get_current_url())
ARGS = {
"driver_task": driver_task
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_json_file |
Ran ran; our contract draft was wrong, not the code |
wi-pi/webspeval_code/src/state_reset_check.py code served (permissive licence) · get_code("53bdbb7cef5156de") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
import os
# Create a temporary JSON file with valid content
temp_dir = tempfile.mkdtemp()
json_file_path = os.path.join(temp_dir, "test.json")
test_data = {
"key1": "value1",
"key2": [1, 2, 3],
"key3": {"nested": "object"}
}
with open(json_file_path, 'w', encoding='utf-8') as f:
json.dump(test_data, f)
ARGS = {
"json_file": json_file_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| switch_to_latest_tab |
Ran ran, violated the contract we drafted |
wi-pi/webspeval_code/src/agent/actions.py code served (permissive licence) · get_code("c0d2dc95bd1e83ea") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import logging
from unittest.mock import Mock, MagicMock
# Create a mock driver_task object that has the necessary attributes and methods
driver_task = Mock()
# Set up window_handles to simulate browser tabs
# First call returns previous handles, second call returns current handles (with a new one added)
driver_task.window_handles = ["handle1", "handle2", "handle3"]
# Create a mock switch_to object with a window method
driver_task.switch_to = Mock()
driver_task.switch_to.window = Mock()
# Create previous_handles list (simulating tabs that existed before)
previous_handles = ["handle1", "handle2"]
ARGS = {
"driver_task": driver_task,
"previous_handles": previous_handles,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| auto_eval_by_gpt5 |
Not yet run unverified (harvested, no recorded run) |
wi-pi/webspeval_code/judge/eval_binary_with_gt_text_gpt5.py code served (permissive licence) · get_code("c8622f34c1208025") How it ran |
| create_fresh_profile |
Not yet run unverified (harvested, no recorded run) |
wi-pi/webspeval_code/src/agent/driver.py code served (permissive licence) · get_code("5205d09c9c182848") How it ran |
| resize_image |
Not yet run unverified (harvested, no recorded run) |
wi-pi/webspeval_code/src/utils.py code served (permissive licence) · get_code("e63c4f0834fe9c79") How it ran |