| build_gpt_eval_messages |
Ran ran; our contract draft was wrong, not the code |
SunGL001/OGPSA/eval/HalluQA.py code served (permissive licence) · get_code("4af073ea30776bc7") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# sample dict - contains model_response key
sample = {
'model_response': '根据提供的信息,这个问题的答案是相关的内容。',
'other_field': 'some_value'
}
# ref dict - contains Question and Best Answer keys
ref = {
'Question': '什么是人工智能?',
'Best Answer': '人工智能是计算机科学的一个分支,致力于创建能够执行通常需要人类智能的任务的系统。',
'Best Answer1': '人工智能(AI)是指由人制造出来的机器所表现的智能。',
'Best Answer2': 'AI是模拟人类智能过程的计算机科学技术。',
}
ARGS = {
'sample': sample,
'ref': ref
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| build_gpt_eval_prompt |
Ran ran; our contract draft was wrong, not the code |
SunGL001/OGPSA/eval/AIME2024.py code served (permissive licence) · get_code("89e5642dfceebefd") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
# Simple string-based arguments for the build_gpt_eval_prompt function
problem = "What is the sum of 5 and 3?"
response = "The sum of 5 and 3 is calculated as follows: 5 + 3 = 8. Therefore, the final answer is 8."
correct_answer = 8
ARGS = {
"problem": problem,
"response": response,
"correct_answer": correct_answer
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| build_gpt_eval_prompt |
Ran ran; our contract draft was wrong, not the code |
SunGL001/OGPSA/eval/AdvGLUE.py code served (permissive licence) · get_code("0d72969c11a70528") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
# Define ARGS dict with valid arguments for build_gpt_eval_prompt
ARGS = {
"question": "What is the sentiment of this text?",
"response": "The sentiment is positive because the text contains many positive adjectives and uplifting language. The label is: 1"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| build_gpt_eval_prompt |
Ran ran; our contract draft was wrong, not the code |
SunGL001/OGPSA/eval/GPQA.py code served (permissive licence) · get_code("1c27eb34b1f772b2") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
ARGS = {
"question": "What is the capital of France?",
"response": "The capital of France is Paris, so the answer is B.",
"choices": {
"A": "London",
"B": "Paris",
"C": "Berlin",
"D": "Madrid"
}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| build_gpt_eval_prompt |
Ran ran; our contract draft was wrong, not the code |
SunGL001/OGPSA/eval/HHH.py code served (permissive licence) · get_code("24d523840feeeb28") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# Define test arguments for build_gpt_eval_prompt
question = "What is the capital of France?"
response = "The capital of France is Paris, which is option A."
ARGS = {
"question": question,
"response": response,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| detect_model_type_from_path |
Ran ran; our contract draft was wrong, not the code |
SunGL001/OGPSA/eval/AlpacaEval.py code served (permissive licence) · get_code("7bc66882c50c5fef") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
# The function takes a single string parameter that represents an input file path
# It detects the model type based on whether 'qwen' or 'llama' appears in the path
# Test case 1: Path containing 'qwen'
input_path_qwen = "/models/qwen-7b/model.bin"
# Test case 2: Path containing 'llama'
input_path_llama = "/models/llama-2-7b/weights.pt"
# Test case 3: Path with no model type indicator
input_path_unknown = "/models/custom_model/checkpoint.pth"
# We'll use the qwen path as the main test case
ARGS = {
"input_path": "/models/qwen-7b/model.bin"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_answer |
Ran ran; our contract draft was wrong, not the code |
SunGL001/OGPSA/eval/MMLU.py code served (permissive licence) · get_code("b9ce5a37c260fee5") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def extract_answer(response):
"""
从模型响应中提取答案(A/B/C/D)
优先查找 "Answer: X" 格式,如果没有则查找其他可能的格式
"""
if not response:
return None
patterns = ['Answer:\\s*([A-D])', 'Answer:\\s*([A-D])\\)', 'answer:\\s*([A-D])', 'ANSWER:\\s*([A-D])']
for pattern in patterns:
match = re.search(pattern, response, re.IGNORECASE)
if match:
answer = match.group(1).upper()
if answer in ['A', 'B', 'C', 'D']:
return answer
lines = response.strip().split('\n')
for line in reversed(lines[-5:]):
line = line.strip()
if len(line) == 1 and line.upper() in ['A', 'B', 'C', 'D']:
return line.upper()
match = re.search('\\b([A-D])\\b', line, re.IGNORECASE)
if match:
answer = match.group(1).upper()
if answer in ['A', 'B', 'C', 'D']:
return answer
match = re.search('\\b([A-D])\\b', response, re.IGNORECASE)
if match:
answer = match.group(1).upper()
if answer in ['A', 'B', 'C', 'D']:
return answer
return None
ARGS = {
"response": "The correct Answer: B is the best choice."
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_answer_from_response |
Ran ran; our contract draft was wrong, not the code |
SunGL001/OGPSA/eval/AIME2024.py code served (permissive licence) · get_code("e704f14bfb984d51") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def extract_answer_from_response(response):
"""从模型回答中提取答案(尝试多种模式)"""
if not response:
return None
boxed_pattern = '\\\\boxed\\{(\\d+)\\}'
match = re.search(boxed_pattern, response)
if match:
return match.group(1)
boxed_pattern2 = 'boxed\\{(\\d+)\\}'
match = re.search(boxed_pattern2, response)
if match:
return match.group(1)
final_answer_patterns = ['(?:The\\s+answer\\s+is|Final\\s+answer|Answer|答案是)[:\\s]*(\\d+)', '(?:Therefore|Thus|Hence)[,\\s]+(?:the\\s+answer\\s+is|we\\s+get)[:\\s]*(\\d+)']
for pattern in final_answer_patterns:
match = re.search(pattern, response, re.IGNORECASE)
if match:
return match.group(1)
lines = response.strip().split('\n')
for line in reversed(lines[-5:]):
numbers = re.findall('\\b(\\d+)\\b', line)
if numbers:
return numbers[-1]
return None
# Test cases with different response formats
test_response_1 = "The answer is 42"
test_response_2 = "Therefore, we get: 123"
test_response_3 = "Some calculation...\nFinal result: 789"
test_response_4 = ""
test_response_5 = "boxed{55}"
ARGS = {
"response": test_response_1
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_answer_from_response |
Ran ran; our contract draft was wrong, not the code |
SunGL001/OGPSA/eval/GPQA.py code served (permissive licence) · get_code("996c1c7807871550") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
ANSWER_PATTERN_MULTICHOICE = '(?i)Answer\\s*:\\s*([A-D])'
def extract_answer_from_response(response):
"""从模型回答中提取多选题答案(A、B、C 或 D)"""
if not response:
return None
match = re.search(ANSWER_PATTERN_MULTICHOICE, response)
if match:
answer = match.group(1).upper()
if answer in ['A', 'B', 'C', 'D']:
return answer
patterns = ['(?:答案是|答案|Answer|answer|The answer is|The correct answer is)[:\\s]*([A-D])', '([A-D])(?:\\s*\\)|\\s*\\.|\\s*$|\\s*[。,,])', '选择\\s*([A-D])', '选项\\s*([A-D])']
for pattern in patterns:
match = re.search(pattern, response, re.IGNORECASE)
if match:
answer = match.group(1).upper()
if answer in ['A', 'B', 'C', 'D']:
return answer
lines = response.strip().split('\n')
for line in reversed(lines[-5:]):
match = re.search('\\b([A-D])\\b', line, re.IGNORECASE)
if match:
answer = match.group(1).upper()
if answer in ['A', 'B', 'C', 'D']:
return answer
return None
ARGS = {
"response": "Answer: B"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_label_from_response |
Ran ran; our contract draft was wrong, not the code |
SunGL001/OGPSA/eval/HHH.py code served (permissive licence) · get_code("522a73213ca49776") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def extract_label_from_response(response):
"""从响应中提取标签(A 或 B)"""
if not response:
return None
patterns = ['\\b[Aa]\\b', '\\b[Bb]\\b', 'Choice:\\s*([ABab])', 'Option\\s+([ABab])']
for pattern in patterns:
match = re.search(pattern, response)
if match:
label = match.group(1) if len(match.groups()) > 0 else match.group(0)
return label.upper()
return None
ARGS = {
"response": "The answer is A"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_model_identifier |
Ran ran; our contract draft was wrong, not the code |
SunGL001/OGPSA/eval/AlpacaEval_pre.py code served (permissive licence) · get_code("92c41ca5148de65a") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def extract_model_identifier(response):
"""从 GPT 响应中提取模型标识符(m 或 M)"""
if not response:
return None
response = response.strip()
if response == 'm':
return 'm'
if response == 'M':
return 'M'
match_m = re.search('\\b[m]\\b', response, re.IGNORECASE)
if match_m:
matched_char = match_m.group(0)
if matched_char == 'm':
return 'm'
elif matched_char == 'M':
return 'M'
if 'm' in response.lower():
for i, char in enumerate(response):
if char.lower() == 'm':
return char
return None
# Test cases for the function
test_response_1 = "m"
test_response_2 = "M"
test_response_3 = "The model is m"
test_response_4 = "Choose between M or N"
test_response_5 = ""
test_response_6 = None
test_response_7 = " m "
ARGS = {
"response": "The answer is m or M"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_prompt |
Ran ran; our contract draft was wrong, not the code |
SunGL001/OGPSA/eval/AlpacaEval_pre.py code served (permissive licence) · get_code("fa528f9eb5d2a2af") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Analyzing the function signature and body:
# - instruction: a string used in the prompt construction
# - ans1: a string representing the model response (gets inserted into prompt)
# - ans2: a string representing the baseline output (gets inserted into prompt)
# All three parameters are strings that are concatenated into a larger prompt message.
ARGS = {
'instruction': 'What is the capital of France?',
'ans1': '"Paris is the capital of France."',
'ans2': '"The capital city of France is Paris."'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_token_count |
Ran ran, honoured the contract we drafted |
SunGL001/OGPSA/eval/AlpacaEval.py code served (permissive licence) · get_code("3a11d80fea8ca185") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# Since get_token_count calls _tokenizer.encode(text, add_special_tokens=False)
# we need a minimal tokenizer stand-in that has an encode method
class MockTokenizer:
def encode(self, text, add_special_tokens=False):
# Simple mock: return a list of token ids based on text length
# Each "word" becomes a token
return list(range(len(text.split()) if text else 0))
# Create a mock tokenizer instance
mock_tokenizer = MockTokenizer()
# Build the arguments dictionary for get_token_count
ARGS = {
"text": "hello world this is a test string",
"model_type": "qwen"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch, transformers — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| process_one |
Ran ran; our contract draft was wrong, not the code |
SunGL001/OGPSA/eval/AdvGLUE.py code served (permissive licence) · get_code("b543cea785162e44") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# Construct valid arguments for process_one(idx, d)
# The function expects:
# - idx: an integer index
# - d: a dictionary with keys 'prompt_to_model', 'prompt', 'model_response', 'task', 'label'
idx = 0
d = {
'prompt_to_model': 'What is the answer?',
'prompt': 'Fallback prompt',
'model_response': 'The answer is 1',
'task': 'classification',
'label': 1
}
ARGS = {
'idx': idx,
'd': d
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| process_one |
Ran ran; our contract draft was wrong, not the code |
SunGL001/OGPSA/eval/GPQA_Diamond.py code served (permissive licence) · get_code("f4dc40dbd0a904af") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
ANSWER_PATTERN_MULTICHOICE = '(?i)Answer\\s*:\\s*([A-D])'
def extract_answer(response):
"""从响应中提取答案字母(参考 gpqa_eval.py 的 get_score 函数)"""
if not response:
return None
match = re.search(ANSWER_PATTERN_MULTICHOICE, response)
if match:
return match.group(1).upper()
return None
def process_one(idx, d):
"""单条样本的处理逻辑(线程函数)"""
response = d.get('model_response', '').strip()
correct_answer = d.get('correct_answer_letter', '').strip().upper()
extracted_answer = extract_answer(response)
correct = False
if extracted_answer and correct_answer:
correct = extracted_answer == correct_answer
d['extracted_answer'] = extracted_answer
d['correct_answer'] = correct_answer
d['correct'] = correct
d['score'] = 1 if correct else 0
return (idx, d)
# Setup code to construct valid arguments for process_one
idx = 0
d = {
'model_response': 'The answer is: Answer: B',
'correct_answer_letter': 'B',
}
ARGS = {
'idx': idx,
'd': d,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| process_one |
Ran ran; our contract draft was wrong, not the code |
SunGL001/OGPSA/eval/MMLU.py code served (permissive licence) · get_code("0364cbd97c9ad0de") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
def extract_answer(response):
"""
从模型响应中提取答案(A/B/C/D)
优先查找 "Answer: X" 格式,如果没有则查找其他可能的格式
"""
if not response:
return None
patterns = ['Answer:\\s*([A-D])', 'Answer:\\s*([A-D])\\)', 'answer:\\s*([A-D])', 'ANSWER:\\s*([A-D])']
for pattern in patterns:
match = re.search(pattern, response, re.IGNORECASE)
if match:
answer = match.group(1).upper()
if answer in ['A', 'B', 'C', 'D']:
return answer
lines = response.strip().split('\n')
for line in reversed(lines[-5:]):
line = line.strip()
if len(line) == 1 and line.upper() in ['A', 'B', 'C', 'D']:
return line.upper()
match = re.search('\\b([A-D])\\b', line, re.IGNORECASE)
if match:
answer = match.group(1).upper()
if answer in ['A', 'B', 'C', 'D']:
return answer
match = re.search('\\b([A-D])\\b', response, re.IGNORECASE)
if match:
answer = match.group(1).upper()
if answer in ['A', 'B', 'C', 'D']:
return answer
return None
def process_one(idx, d):
"""单条样本的处理逻辑(线程函数)"""
response = d.get('model_response', '').strip()
label = d.get('label') or d.get('correct_answer_letter')
extracted_label = extract_answer(response)
if extracted_label is None:
extracted_label = 'NONE'
if label:
label = str(label).upper().strip()
correct = extracted_label != 'NONE' and extracted_label == label
else:
correct = False
d['predicted_label'] = extracted_label
d['correct'] = correct
d['score'] = 1 if correct else 0
return (idx, d)
# Setup code
sample_dict = {
'model_response': 'The answer is Answer: B because of the following reasoning.',
'label': 'B',
'correct_answer_letter': None
}
idx_value = 0
ARGS = {
'idx': idx_value,
'd': sample_dict
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_answer |
Not yet run unverified (harvested, no recorded run) |
SunGL001/OGPSA/eval/GPQA_Diamond.py code served (permissive licence) · get_code("a7b81f2f5969d9ab") How it ran |
| load_tokenizer_for_eval |
Not yet run unverified (harvested, no recorded run) |
SunGL001/OGPSA/eval/AlpacaEval.py code served (permissive licence) · get_code("10cae133e44ed5f1") How it ran |