| all_equal |
Ran ran; our contract draft was wrong, not the code |
parallelcodefoundry/ParEval/drivers/util.py code served (permissive licence) · get_code("de2d759b067567af") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# The function all_equal takes an iterable and checks if all values are equal
# by converting to a set and checking if the set has <= 1 element
# Test case 1: All equal elements
iterable_all_equal = [1, 1, 1, 1]
ARGS = {
"iterable": iterable_all_equal
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| already_has_results |
Ran ran, violated the contract we drafted |
parallelcodefoundry/ParEval/drivers/run-all.py code served (permissive licence) · get_code("e7bbb8f061c2443c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
# Construct a prompt dict with results stored as dicts
ARGS = {
'prompt': {
'name': 'test_prompt',
'outputs': [
{'result': 'output1', 'score': 0.95},
{'result': 'output2', 'score': 0.87}
]
}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| any_in |
Ran ran, violated the contract we drafted |
parallelcodefoundry/ParEval/analysis/bin-the-stack.py code served (permissive licence) · get_code("0938e73682f13d8b") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from typing import List
torch.manual_seed(0)
# The function any_in takes:
# - s: str - a string to search in
# - substrs: List[str] - a list of substrings to search for
s = "hello world"
substrs = ["world", "foo", "bar"]
ARGS = {
"s": s,
"substrs": substrs,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numba, numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| buildk |
Ran ran, honoured the contract we drafted |
parallelcodefoundry/ParEval/analysis/metrics.py code served (permissive licence) · get_code("98ceec1f11b8e7a0") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
import pandas as pd
# Create a sample DataFrame with the required columns
data = {
'name': ['model_a', 'model_a', 'model_b', 'model_b', 'model_c', 'model_c'],
'parallelism_model': ['data', 'data', 'model', 'model', 'data', 'model'],
'problem_type': ['classification', 'regression', 'classification', 'regression', 'classification', 'regression'],
'did_build': [1, 0, 1, 1, 0, 1]
}
df = pd.DataFrame(data)
k = 2
ARGS = {
'df': df,
'k': k
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| clean_output |
Ran ran; our contract draft was wrong, not the code |
parallelcodefoundry/ParEval/generate/clean-outputs.py code served (permissive licence) · get_code("b588819cf6250c27") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
# Define ARGS for clean_output function
ARGS = {
"prompt": "def hello():\n print('world')",
"output": "print('hello')"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| efficiencyk |
Ran ran, honoured the contract we drafted |
parallelcodefoundry/ParEval/analysis/metrics-scaling.py code served (permissive licence) · get_code("8d9bdc243671a23f") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
import pandas as pd
# Create a sample dataframe with the required columns
np.random.seed(0)
# Create base data
n_rows = 12
df = pd.DataFrame({
'is_valid': [True] * n_rows,
'n': [2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3],
'name': ['task_a'] * 6 + ['task_b'] * 6,
'parallelism_model': ['model_x'] * 3 + ['model_y'] * 3 + ['model_x'] * 3 + ['model_y'] * 3,
'problem_type': ['type_1'] * 3 + ['type_2'] * 3 + ['type_1'] * 3 + ['type_2'] * 3,
'output_idx': list(range(3)) * 4,
'runtime': np.random.uniform(0.1, 2.0, n_rows),
'best_sequential_runtime': np.random.uniform(0.5, 3.0, n_rows),
})
ARGS = {
'df': df,
'k': 2,
'n': 2,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_correctness_df |
Ran ran; our contract draft was wrong, not the code |
parallelcodefoundry/ParEval/analysis/metrics.py code served (permissive licence) · get_code("e011cbcff09dc44b") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import pandas as pd
# Create a sample DataFrame with the required columns
df = pd.DataFrame({
'name': ['model_a', 'model_a', 'model_a', 'model_b', 'model_b', 'model_b'],
'parallelism_model': ['dp', 'dp', 'tp', 'dp', 'tp', 'tp'],
'output_idx': [0, 0, 1, 0, 0, 1],
'is_valid': [True, True, False, True, True, True],
'problem_type': ['classification', 'classification', 'classification', 'regression', 'regression', 'regression']
})
ARGS = {
'df': df
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_env_var |
Ran ran; our contract draft was wrong, not the code |
parallelcodefoundry/ParEval/generate/generate-gemini.py code served (permissive licence) · get_code("2f1ce38ddef63dc9") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import os
# Set an environment variable for the function to retrieve
os.environ['TEST_VAR'] = 'test_value'
# Define arguments for get_env_var
ARGS = {
'name': 'TEST_VAR'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_function_name |
Ran ran; our contract draft was wrong, not the code |
parallelcodefoundry/ParEval/generate/generate-gemini.py code served (permissive licence) · get_code("fc1387d6dfa3f110") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
GPU_FUNCTION_NAME_PATTERN = re.compile('__global__ void ([a-zA-Z0-9_]+)\\(')
CPU_FUNCTION_NAME_PATTERN = re.compile('\\s*[a-zA-Z_]+ ([a-zA-Z0-9_]+)\\(')
def get_function_name(prompt: str, execution_model: str) -> str:
if execution_model in ['cuda', 'hip']:
match = GPU_FUNCTION_NAME_PATTERN.match(prompt.splitlines()[-1])
else:
match = CPU_FUNCTION_NAME_PATTERN.match(prompt.splitlines()[-1])
if match is None:
raise ValueError(f'Could not find function name in prompt: {prompt}')
return match.group(1)
ARGS = {
'prompt': '__global__ void kernel_function(',
'execution_model': 'cuda'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_max_tokens_per_second |
Ran ran, honoured the contract we drafted |
parallelcodefoundry/ParEval/generate/generate-gemini.py code served (permissive licence) · get_code("60484b445abb73b5") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
from typing import Optional
# Set seed for reproducibility
torch.manual_seed(0)
# The function takes a model parameter (string) and checks if it equals 'gemini-1.0-pro'
# It returns tokens per second or None
# The model parameter is just a string, so we pass a valid model name
ARGS = {
'model': 'gemini-1.0-pro'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_max_tokens_per_second |
Ran ran, honoured the contract we drafted |
parallelcodefoundry/ParEval/generate/generate-openai.py code served (permissive licence) · get_code("85d8ccfa5ba4fb7b") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
from typing import Optional
def get_max_tokens_per_second(model: str) -> Optional[int]:
""" rates limites as of January 2024 """
if model == 'gpt-3.5-turbo-1106':
tokens_per_minute = 60000
return tokens_per_minute / 60
elif model == 'gpt-4-1106-preview':
tokens_per_minute = 150000
return tokens_per_minute / 60
else:
return None
ARGS = {
"model": "gpt-3.5-turbo-1106"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| has_balanced_brackets |
Ran ran, violated the contract we drafted |
parallelcodefoundry/ParEval/generate/clean-outputs.py code served (permissive licence) · get_code("d55502b35fc3f4c1") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
def has_balanced_brackets(s: str) -> bool:
stack = []
for c in s:
if c == '{':
stack.append(c)
elif c == '}':
if len(stack) == 0:
return False
stack.pop()
return len(stack) == 0
ARGS = {
's': '{{}}{}{{{}}}}'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| has_outputs |
Ran ran, violated the contract we drafted |
parallelcodefoundry/ParEval/analysis/create-dataframe.py code served (permissive licence) · get_code("311658ebceb020fe") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Test case 1: Valid prompt with outputs containing dicts
prompt_valid = {
'outputs': [
{'result': 'value1'},
{'result': 'value2'}
]
}
ARGS = {
'prompt': prompt_valid
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_json |
Ran ran; our contract draft was wrong, not the code |
parallelcodefoundry/ParEval/drivers/util.py code served (permissive licence) · get_code("9430b9d58bca314d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
import os
from os import PathLike
# Create a temporary JSON file with some test data
temp_dir = tempfile.mkdtemp()
test_json_path = os.path.join(temp_dir, "test.json")
test_data = {
"key1": "value1",
"key2": 42,
"key3": ["list", "of", "items"],
"key4": {"nested": "dict"}
}
with open(test_json_path, 'w') as f:
json.dump(test_data, f)
ARGS = {
"fpath": test_json_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| nCr |
Ran ran, honoured the contract we drafted |
parallelcodefoundry/ParEval/analysis/metrics-scaling.py code served (permissive licence) · get_code("c1d07cc2f1b9ecf6") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from math import comb
def nCr(n: int, r: int) -> int:
if n < r:
return 1
return comb(n, r)
ARGS = {
"n": 5,
"r": 2
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| read_json |
Ran ran; our contract draft was wrong, not the code |
parallelcodefoundry/ParEval/analysis/bin-the-stack.py code served (permissive licence) · get_code("24736464e5b48a38") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import json
import tempfile
import os
# Create a temporary JSON file
temp_dir = tempfile.mkdtemp()
temp_json_path = os.path.join(temp_dir, "test.json")
# Write a valid JSON file
test_data = {
"key1": "value1",
"key2": [1, 2, 3],
"key3": {"nested": "data"}
}
with open(temp_json_path, 'w') as f:
json.dump(test_data, f)
ARGS = {
"fpath": temp_json_path
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| speedupk |
Ran ran, honoured the contract we drafted |
parallelcodefoundry/ParEval/analysis/metrics-scaling.py code served (permissive licence) · get_code("084d0107c87254df") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import pandas as pd
import numpy as np
# Create a sample DataFrame with the required structure
data = {
'is_valid': [True, True, True, True, True, True, True, True],
'n': [10, 10, 10, 10, 20, 20, 20, 20],
'name': ['task1', 'task1', 'task2', 'task2', 'task1', 'task1', 'task2', 'task2'],
'parallelism_model': ['model_a', 'model_a', 'model_b', 'model_b', 'model_a', 'model_a', 'model_b', 'model_b'],
'output_idx': [0, 1, 0, 1, 0, 1, 0, 1],
'problem_type': ['type_x', 'type_x', 'type_y', 'type_y', 'type_x', 'type_x', 'type_y', 'type_y'],
'best_sequential_runtime': [1.5, 1.5, 2.0, 2.0, 1.5, 1.5, 2.0, 2.0],
'runtime': [0.5, 0.6, 0.7, 0.8, 0.5, 0.6, 0.7, 0.8],
}
df = pd.DataFrame(data)
ARGS = {
'df': df,
'k': 3,
'n': 10,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| uses_mpi |
Ran ran, violated the contract we drafted |
parallelcodefoundry/ParEval/analysis/bin-the-stack.py code served (permissive licence) · get_code("6eee785f978e2422") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
from numba import njit, jit
from typing import List
@njit
def any_in(s: str, substrs: List[str]) -> bool:
for substr in substrs:
if substr in s:
return True
return False
@njit
def uses_mpi(contents: str, language: str) -> bool:
if language in ['C', 'C++']:
return any_in(contents, ['MPI_', '#include <mpi.h>', '#include "mpi.h"'])
elif language == 'FORTRAN':
return any_in(contents, ['MPI_', "include 'mpif.h'", 'include "mpif.h"'])
elif language == 'Python':
return any_in(contents, ['mpi4py'])
return False
ARGS = {
'contents': '#include <mpi.h>\nint main() { MPI_Init(); }',
'language': 'C'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numba, numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| await_input |
Not yet run unverified (harvested, no recorded run) |
parallelcodefoundry/ParEval/drivers/util.py code served (permissive licence) · get_code("9c46438f56d92741") How it ran |