| build_process_value |
Ran ran; our contract draft was wrong, not the code |
aifinlab/UniFinEval/evaluate_py/result_utils.py code served (permissive licence) · get_code("93a10bdfb8d89468") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
from typing import Dict, Any
torch_seed = 0
ARGS = {
'model_answer': 'The answer is 42. \\boxed{42}',
'model_data': {
'raw_response': {
'choices': [
{
'message': {
'reasoning': 'Let me think about this problem step by step. First, I need to understand what is being asked.',
'reasoning_content': None,
'reasoning_details': None
}
}
]
}
}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| clean_json_text |
Ran ran; our contract draft was wrong, not the code |
aifinlab/UniFinEval/evaluate_py/judge.py code served (permissive licence) · get_code("a97875241ff90c42") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def clean_json_text(text: str) -> str:
"""
从模型输出中提取 JSON 字符串
兼容模型输出 ```json ... ``` 包裹的情况
"""
text = text.strip()
pattern = '```json\\s*(\\{.*?\\})\\s*```|(\\{.*\\})'
match = re.search(pattern, text, re.DOTALL)
if match:
return match.group(1) or match.group(2)
start = text.find('{')
end = text.rfind('}')
if start != -1 and end != -1:
return text[start:end + 1]
return text
# Test case 1: JSON wrapped in ```json markers
test_text_1 = """```json
{"key": "value", "number": 42}
"""
# Test case 2: Plain JSON without markers
test_text_2 = '{"name": "test", "data": [1, 2, 3]}'
# Test case 3: JSON with surrounding text
test_text_3 = 'Here is some text before {"result": "success"} and after'
# Test case 4: Whitespace handling
test_text_4 = """
json
{"status": "ok"}
"""
ARGS = {
"text": test_text_1
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| convert_to_standard_format |
Ran ran; our contract draft was wrong, not the code |
aifinlab/UniFinEval/evaluate_py/data_converter.py code served (permissive licence) · get_code("e8760156d157dfd3") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
# Construct a representative input item with various field formats
item = {
'question_id': 'q_001',
'image_id': 'img_001',
'image_path': '/path/to/image1.jpg, /path/to/image2.jpg',
'image_type': 'photo',
'question_type': 'visual_qa',
'original_image_path': '/original/path1.jpg;/original/path2.jpg',
'image_urls': ['http://example.com/img1.jpg', 'http://example.com/img2.jpg'],
'scenario': 'indoor',
'capability': 'object_recognition',
'difficulty': 'medium',
'source': 'dataset_v1',
'question': {
'round1': 'What is in the image?',
'round2': 'What color is it?'
},
'answer': {
'round1': 'A cat',
'round2': 'Orange'
},
'options': {
'A': 'Dog',
'B': 'Cat',
'C': 'Bird',
'D': 'Fish'
},
'qa_make_process': {
'step1': 'Extract objects',
'step2': 'Generate questions'
},
'extra_field_1': 'some_value',
'extra_field_2': 42
}
ARGS = {
'item': item
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| encode_image |
Ran ran; our contract draft was wrong, not the code |
aifinlab/UniFinEval/evaluate_py/model_api.py code served (permissive licence) · get_code("625809a36981482c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import os
import tempfile
from PIL import Image
import io
# Create a temporary test image
temp_dir = tempfile.mkdtemp()
test_image_path = os.path.join(temp_dir, "test_image.jpg")
# Create a simple test image (800x600 pixels)
img = Image.new('RGB', (800, 600), color=(73, 109, 137))
img.save(test_image_path, format='JPEG', quality=95)
# Set PIL_AVAILABLE flag for the function
PIL_AVAILABLE = True
# Define arguments for encode_image function
ARGS = {
'image_path': test_image_path,
'compress': False,
'max_longest_side': 1600,
'quality': 92
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- PIL — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| evaluate_single_item |
Ran ran; our contract draft was wrong, not the code |
aifinlab/UniFinEval/evaluate_py/evaluator.py code served (permissive licence) · get_code("58ca03390a3351e2") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import os
import json
import logging
from typing import List, Dict, Any, Optional
# Configure logging
logging.basicConfig(level=logging.INFO)
# Mock the EVAL_CONFIG that the function references
EVAL_CONFIG = {
'skip_missing_images': False
}
# Mock the normalize_question_type function that the function calls
def normalize_question_type(raw_type: str) -> str:
"""Mock function to normalize question types"""
type_mapping = {
'单选': '单选题',
'多选': '多选题',
'多轮单选': '多轮单选题',
}
return type_mapping.get(raw_type, raw_type)
# Create minimal valid arguments
item = {
'question_id': 'q_001',
'id': 'item_001',
'question': 'What is 2+2?',
'answer': 'The answer is 4',
'options': {
'A': '2',
'B': '4',
'C': '6',
'D': '8'
},
'image_path': '',
'image_urls': [],
'question_type': '单选',
'is_multi_round': False
}
enabled_models = [
'gpt-4',
'gpt-3.5-turbo',
'claude-3-opus'
]
profiles = [
'profile_expert',
'profile_beginner',
'profile_average'
]
workers = 2
ARGS = {
'item': item,
'enabled_models': enabled_models,
'profiles': profiles,
'workers': workers
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| format_options |
Ran ran; our contract draft was wrong, not the code |
aifinlab/UniFinEval/evaluate_py/prompts.py code served (permissive licence) · get_code("15d98731e128c61f") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from typing import Dict, Optional
# Test case 1: Normal dictionary with options
options_dict = {
"A": "选项A内容",
"B": "选项B内容",
"C": "选项C内容"
}
# Test case 2: None (should return empty string)
options_none = None
# Test case 3: Empty dictionary (should return empty string)
options_empty = {}
# Test case 4: Dictionary with empty string values (should skip them)
options_with_empty = {
"A": "选项A内容",
"B": "",
"C": "选项C内容"
}
# Primary test case - standard dictionary
ARGS = {
"options": {
"A": "选项A内容",
"B": "选项B内容",
"C": "选项C内容"
}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_image_format |
Ran ran; our contract draft was wrong, not the code |
aifinlab/UniFinEval/evaluate_py/model_api.py code served (permissive licence) · get_code("571e6a28dad787d4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
from pathlib import Path
# Test with various image path formats
ARGS = {
'image_path': '/path/to/image.png'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| is_answer_empty |
Ran ran, violated the contract we drafted |
aifinlab/UniFinEval/evaluate_py/result_utils.py code served (permissive licence) · get_code("102ece5a1388ab6c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
from typing import Dict, Any
# Test case 1: Basic module2_item with valid answer
module2_item_valid = {
'model': {
'model_name': 'gpt-4',
'answer': 'This is a valid answer'
},
'model1': {
'model_name': 'gpt-3.5',
'answer': 'Another valid answer'
}
}
# Test case 2: module2_item with empty answer
module2_item_empty = {
'model': {
'model_name': 'gpt-4',
'answer': ''
}
}
# Test case 3: module2_item with dict answer (multi-round)
module2_item_dict_answer = {
'model': {
'model_name': 'gpt-4',
'answer': {
'round_1': 'Response 1',
'round_2': 'Response 2'
}
}
}
# Test case 4: module2_item with None answer
module2_item_none = {
'model': {
'model_name': 'gpt-4',
'answer': None
}
}
# Test case 5: module2_item with whitespace-only answer
module2_item_whitespace = {
'model': {
'model_name': 'gpt-4',
'answer': ' '
}
}
# Test case 6: module2_item with multiple models
module2_item_multi = {
'model': {
'model_name': 'model-a',
'answer': 'valid answer'
},
'model2': {
'model_name': 'model-b',
'answer': ''
},
'model3': {
'model_name': 'model-c',
'answer': {'round1': 'answer1', 'round2': ''}
}
}
ARGS = {
'module2_item': module2_item_valid,
'model_key': 'model',
'model_name': 'gpt-4'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_csv |
Ran ran; our contract draft was wrong, not the code |
aifinlab/UniFinEval/evaluate_py/data_loader.py code served (permissive licence) · get_code("80031dd34b37c131") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import csv
from pathlib import Path
# Create a temporary CSV file for testing
temp_dir = tempfile.mkdtemp()
csv_file_path = Path(temp_dir) / "test_data.csv"
# Write test CSV data
csv_data = [
{
"id": "1",
"image_path": "/path/to/image1.jpg;/path/to/image2.jpg",
"image_urls": "http://example.com/img1.jpg;http://example.com/img2.jpg",
"round": "1",
"text": "sample text",
"options": '{"A": "option_a", "B": "option_b"}',
},
{
"id": "2",
"image_path": "/path/to/image3.jpg",
"image_urls": "http://example.com/img3.jpg",
"round": "2",
"text": "another sample",
"option_key1": "value1",
"option_key2": "value2",
},
{
"id": "3",
"image_path": "null",
"image_urls": None,
"round": "invalid",
"text": "test with nulls",
"options": None,
},
]
with open(csv_file_path, 'w', newline='', encoding='utf-8') as f:
fieldnames = ["id", "image_path", "image_urls", "round", "text", "options", "option_key1", "option_key2"]
writer = csv.DictWriter(f, fieldnames=fieldnames)
writer.writeheader()
for row in csv_data:
writer.writerow(row)
ARGS = {
"file_path": csv_file_path,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_json |
Ran ran; our contract draft was wrong, not the code |
aifinlab/UniFinEval/evaluate_py/data_loader.py code served (permissive licence) · get_code("d068ef448a266fb4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
from pathlib import Path
# Create a temporary directory for test files
temp_dir = tempfile.mkdtemp()
# Test case 1: JSON file with array structure
array_file = Path(temp_dir) / "array.json"
with open(array_file, 'w', encoding='utf-8') as f:
json.dump([{"id": 1, "name": "item1"}, {"id": 2, "name": "item2"}], f)
# Test case 2: JSON file with dict containing 'items' field
items_file = Path(temp_dir) / "items.json"
with open(items_file, 'w', encoding='utf-8') as f:
json.dump({"items": [{"id": 3, "name": "item3"}, {"id": 4, "name": "item4"}]}, f)
# Test case 3: JSON file with plain dict (no 'items' field)
dict_file = Path(temp_dir) / "dict.json"
with open(dict_file, 'w', encoding='utf-8') as f:
json.dump({"id": 5, "name": "item5"}, f)
ARGS = {
"file_path": array_file
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_jsonl |
Ran ran; our contract draft was wrong, not the code |
aifinlab/UniFinEval/evaluate_py/data_loader.py code served (permissive licence) · get_code("fca995540b6041ae") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
from pathlib import Path
# Create a temporary JSONL file for testing
temp_dir = tempfile.mkdtemp()
temp_file = Path(temp_dir) / "test_data.jsonl"
# Write some test data to the JSONL file
test_data = [
{"id": 1, "text": "Hello world", "value": 42},
{"id": 2, "text": "Another line", "value": 100},
{"id": 3, "text": "Third entry", "nested": {"key": "value"}},
]
with open(temp_file, 'w', encoding='utf-8') as f:
for item in test_data:
f.write(json.dumps(item) + '\n')
# Define arguments for the function
ARGS = {
"file_path": temp_file
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| merge_multi_round_items |
Ran ran; our contract draft was wrong, not the code |
aifinlab/UniFinEval/evaluate_py/data_converter.py code served (permissive licence) · get_code("3a416d9bf9a752e6") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
# Setup code for merge_multi_round_items function
ARGS = {
"items": [
# Single-round item (no round field)
{
"id": "q1",
"question_id": "q1",
"question": "What is this?",
"answer": "This is an object",
"image_id": "img1",
"image_urls": ["http://example.com/img1.jpg"],
"question_type": "visual_qa",
"scenario": "indoor",
"capability": "object_detection",
"difficulty": "easy",
"source": "dataset_a"
},
# Multi-round items with same question_id
{
"id": "q2",
"question_id": "q2",
"round": 1,
"question": "What color is it?",
"answer": "It is red",
"image_id": "img2",
"image_urls": "http://example.com/img2.jpg",
},
{
"id": "q2",
"question_id": "q2",
"round": 2,
"question": "How many are there?",
"answer": "There are three",
"image_id": "img2",
"image_urls": ["http://example.com/img2_alt.jpg"],
},
{
"id": "q2",
"question_id": "q2",
"round": 3,
"question": "What is the material?",
"answer": "It is made of plastic",
"image_id": "img2",
},
# Multi-round items with same id field only
{
"id": "q3",
"round": 1,
"question": "First question",
"answer": "First answer",
"image_id": "img3",
"image_urls": "http://example.com/img3.jpg;http://example.com/img3_v2.jpg",
},
{
"id": "q3",
"round": 2,
"question": "Second question",
"answer": "Second answer",
"image_id": "img3",
},
# Item with no id or question_id (should be skipped initially, added back at end)
{
"question": "Orphan question",
"answer": "Orphan answer",
}
]
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| normalize_question_type |
Ran ran; our contract draft was wrong, not the code |
aifinlab/UniFinEval/evaluate_py/prompts.py code served (permissive licence) · get_code("f6d2ed65eee5b187") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
QUESTION_TYPE_MAPPING = {'single_choice': '单选题', 'multiple_choice': '多选题', 'true_false': '判断题', 'essay': '问答题', 'multi_round_single_choice': '多轮单选题', 'multi_round_essay': '多轮问答题', '单选题': '单选题', '多选题': '多选题', '判断题': '判断题', '问答题': '问答题', '多轮单选题': '多轮单选题', '多轮问答题': '多轮问答题', '单选': '单选题', '多选': '多选题', '判断': '判断题', '问答': '问答题', '多轮单选': '多轮单选题', '多轮问答': '多轮问答题', '单选题型': '单选题', '多选题型': '多选题', '判断题型': '判断题', '问答题型': '问答题', '单选题目': '单选题', '多选题目': '多选题', '判断题目': '判断题', '问答题目': '问答题', 'single': '单选题', 'multiple': '多选题', 'true_false_question': '判断题', 'essay_question': '问答题', 'multi_round_single': '多轮单选题', 'multi_round': '多轮问答题'}
def normalize_question_type(question_type: str) -> str:
"""
标准化题型名称(统一转换为中文)
Args:
question_type: 题型名称(支持中英文)
Returns:
中文题型名称,如果不在支持列表中则返回原值
"""
if not question_type:
return ''
question_type = question_type.strip()
if question_type in QUESTION_TYPE_MAPPING:
return QUESTION_TYPE_MAPPING[question_type]
question_type_lower = question_type.lower()
for key, value in QUESTION_TYPE_MAPPING.items():
if key.lower() == question_type_lower:
return value
return question_type
ARGS = {
'question_type': 'single_choice'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| sanitize_messages_for_log |
Ran ran; our contract draft was wrong, not the code |
aifinlab/UniFinEval/evaluate_py/logger.py code served (permissive licence) · get_code("b28da1e86c8bd6f5") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from typing import List, Dict, Any, Optional
torch.manual_seed(0)
# Construct valid arguments for sanitize_messages_for_log
# Test case 1: messages with base64 image data and image_paths provided
messages_with_base64 = [
{
'role': 'user',
'content': [
{'type': 'text', 'text': 'What is in this image?'},
{'type': 'image_url', 'image_url': {'url': 'data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEA...'}}
]
},
{
'role': 'assistant',
'content': [
{'type': 'text', 'text': 'This is an image.'},
{'type': 'image_url', 'image_url': {'url': 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUg...'}}
]
}
]
image_paths_list = ['/path/to/image1.jpg', '/path/to/image2.png']
# Test case 2: messages with regular URLs (not base64)
messages_with_urls = [
{
'role': 'user',
'content': [
{'type': 'text', 'text': 'Check this URL'},
{'type': 'image_url', 'image_url': {'url': 'https://example.com/image.jpg'}}
]
}
]
# Test case 3: empty messages
empty_messages = []
# Test case 4: messages with string content (not list)
messages_string_content = [
{
'role': 'user',
'content': 'This is a simple text message'
}
]
# Test case 5: messages with base64 but no image_paths
messages_base64_no_paths = [
{
'role': 'user',
'content': [
{'type': 'text', 'text': 'Image below'},
{'type': 'image_url', 'image_url': {'url': 'data:image/webp;base64,UklGRiYA...'}}
]
}
]
ARGS = {
'messages': messages_with_base64,
'image_paths': image_paths_list
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| strip_boxed_content |
Ran ran; our contract draft was wrong, not the code |
aifinlab/UniFinEval/evaluate_py/result_utils.py code served (permissive licence) · get_code("2a0691b3aab83548") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def strip_boxed_content(text: str) -> str:
"""
移除文本中的 \\boxed{...} 片段,只保留思考/分析部分。
Args:
text: 原始文本
Returns:
清理后的文本
"""
if not text:
return ''
cleaned = re.sub('\\\\\\\\boxed\\{.*?\\}', '', text, flags=re.DOTALL)
cleaned = re.sub('\\\\boxed\\{.*?\\}', '', cleaned, flags=re.DOTALL)
return cleaned.strip()
ARGS = {
"text": "This is some thinking \\boxed{answer1} and more text \\\\boxed{answer2} with analysis"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| calculate_output_statistics |
Not yet run unverified (harvested, no recorded run) |
aifinlab/UniFinEval/evaluate_py/statistics.py code served (permissive licence) · get_code("df38989e5f169bd9") How it ran |
| calculate_statistics |
Not yet run unverified (harvested, no recorded run) |
aifinlab/UniFinEval/evaluate_py/statistics.py code served (permissive licence) · get_code("197585381b7bb3fd") How it ran |
| compress_image |
Not yet run unverified (harvested, no recorded run) |
aifinlab/UniFinEval/evaluate_py/model_api.py code served (permissive licence) · get_code("6eb801bbd3095e75") How it ran |
| convert_batch |
Not yet run unverified (harvested, no recorded run) |
aifinlab/UniFinEval/evaluate_py/data_converter.py code served (permissive licence) · get_code("ff8af687b81c85d6") How it ran |
| convert_to_module2_format |
Not yet run unverified (harvested, no recorded run) |
aifinlab/UniFinEval/evaluate_py/result_converter.py code served (permissive licence) · get_code("5a452f073e882c48") How it ran |
| get_prompt |
Not yet run unverified (harvested, no recorded run) |
aifinlab/UniFinEval/evaluate_py/prompts.py code served (permissive licence) · get_code("eb48ceb1095ac012") How it ran |