| by_max_acc |
Ran ran on a synthesized input |
sunblaze-ucb/SED/program_synthesis/analysis/organize_results.py code served (permissive licence) · get_code("79c5f129bde8c733") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import pandas as pd
import numpy as np
# Create a sample DataFrame with the required columns
# The function groups by columns that are NOT 'Step', 'Accuracy', 'Exact'
# So we need at least one grouping column plus the three required columns
data = {
'Model': ['ModelA', 'ModelA', 'ModelB', 'ModelB'],
'Dataset': ['DS1', 'DS1', 'DS1', 'DS1'],
'Step': [100, 200, 150, 250],
'Accuracy': [0.85, 0.92, 0.88, 0.95],
'Exact': [0.80, 0.90, 0.85, 0.93]
}
accuracies = pd.DataFrame(data)
ARGS = {
'accuracies': accuracies
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| by_max_step |
Ran ran on a synthesized input |
sunblaze-ucb/SED/program_synthesis/analysis/organize_results.py code served (permissive licence) · get_code("5f53ac6d6dd92573") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import pandas as pd
import numpy as np
# Create a sample DataFrame that mimics the expected structure
# The function groups by all columns except 'Step', 'Accuracy', 'Exact'
# and finds the max Step for each group
data = {
'Step': [1, 2, 3, 1, 2, 3],
'Accuracy': [0.85, 0.87, 0.89, 0.80, 0.82, 0.84],
'Exact': [0.75, 0.77, 0.79, 0.70, 0.72, 0.74],
'Model': ['A', 'A', 'A', 'B', 'B', 'B'],
'Dataset': ['X', 'X', 'X', 'X', 'X', 'X']
}
accuracies = pd.DataFrame(data)
ARGS = {
'accuracies': accuracies
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| evaluate_code |
Ran ran on a synthesized input |
sunblaze-ucb/SED/program_synthesis/datasets/executor.py code served (permissive licence) · get_code("fef08668e5cc872a") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
class ExecutionResult:
def __init__(self, result):
self.result = result
def mock_do_execute(code, arguments, test_input):
"""Mock executor that simulates code execution"""
# For testing purposes, return a result based on test_input
if test_input == "valid":
return ExecutionResult(42)
elif test_input == "mismatch":
return ExecutionResult(99)
else:
return ExecutionResult(None)
# Define test cases
tests = [
{'input': 'valid', 'output': 42},
{'input': 'mismatch', 'output': 100},
{'input': 'none_result', 'output': None},
]
# Define arguments
ARGS = {
'code': 'def test_func(x):\n return x * 2',
'arguments': {'param1': 10, 'param2': [1, 2, 3]},
'tests': tests,
'do_execute': mock_do_execute
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| expand_by_beam |
Ran ran on a synthesized input |
sunblaze-ucb/SED/program_synthesis/models/beam_search.py code served (permissive licence) · get_code("00ffb35e6b02915b") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 3, dtype family float, shape (6, 4, 3) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
torch.manual_seed(0)
# Analyze the function:
# - v: must be a tensor (has .unsqueeze, .repeat, .dim, .view, .shape methods)
# - beam_size: an integer for repetition factor
# Create a small tensor with batch_size=2, some feature dimensions
v = torch.randn(2, 4, 3) # shape: (batch=2, feat1=4, feat2=3)
beam_size = 3
ARGS = {
"v": v,
"beam_size": beam_size,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 3, dtype family float, shape (6, 4, 3) — shape (6, 4, 3) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_arg_parser |
Ran ran on a synthesized input |
sunblaze-ucb/SED/program_synthesis/arguments.py code served (permissive licence) · get_code("881122c4654562fb") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ArgumentParser. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
import argparse
import time
torch.manual_seed(0)
# The function signature is: get_arg_parser(title, mode)
# We need to provide values for both parameters
ARGS = {
'title': 'Test Parser',
'mode': 'train'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ArgumentParser — a ArgumentParser output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a ArgumentParser output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_attn_mask |
Ran ran on a synthesized input |
sunblaze-ucb/SED/program_synthesis/models/base.py code served (permissive licence) · get_code("a09108fab4b8d1b4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family float, shape (3, 5) as a later run recorded it. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# The function expects:
# - seq_lengths: a list of integers representing sequence lengths for each batch
# - cuda: a boolean flag for whether to move to cuda
ARGS = {
"seq_lengths": [3, 5, 2], # List of sequence lengths for batch_size=3
"cuda": False # CPU only
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 2, dtype family float, shape (3, 5) — shape (3, 5) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_exact_match |
Ran ran on a synthesized input |
sunblaze-ucb/SED/program_synthesis/analysis/load_results.py code served (permissive licence) · get_code("7eeeb9e1399f3c5c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import os
import json
import tempfile
# Create a temporary directory and file for testing
temp_dir = tempfile.mkdtemp()
test_file = os.path.join(temp_dir, "test_data.jsonl")
# Create a valid JSONL file with the expected structure
# The function expects lines with 'example' and 'code' keys
test_data = [
{"example": {"code": "def foo(): pass"}, "code": {"code_sequence": "def foo(): pass"}}, # Header line
{"example": {"code": "x = 1"}, "code": {"code_sequence": "x = 1"}}, # Exact match
{"example": {"code": "y = 2"}, "code": {"code_sequence": "y = 3"}}, # Not exact match
{"example": {"code": "z = 4"}, "code": {"code_sequence": "z = 4"}}, # Exact match
]
with open(test_file, 'w') as f:
for item in test_data:
f.write(json.dumps(item) + '\n')
ARGS = {
"path": test_file
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_rev_vocab |
Ran ran on a synthesized input |
sunblaze-ucb/SED/program_synthesis/datasets/data.py code served (permissive licence) · get_code("74ee36a2c213a620") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# The function get_rev_vocab takes a vocab dict and reverses it
# It expects vocab to be a dict with keys (strings) and values (indices/integers)
# It iterates through vocab.items() to build the reverse mapping
vocab = {
"hello": 0,
"world": 1,
"test": 2,
"python": 3
}
ARGS = {
"vocab": vocab
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| hyp_score |
Ran ran on a synthesized input |
sunblaze-ucb/SED/program_synthesis/models/beam_search.py code served (permissive licence) · get_code("5f7285b3d57507d8") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 3, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Create small tensors for the function parameters
batch_size = 2
beam_size = 3
# log_probs: should be shape [batch_size x beam_size]
log_probs = torch.randn(batch_size, beam_size)
# sequence_lengths: should be shape [batch_size x beam_size]
sequence_lengths = torch.randint(1, 20, (batch_size, beam_size)).float()
# penalty_factor: a scalar
penalty_factor = 0.6
ARGS = {
"log_probs": log_probs,
"sequence_lengths": sequence_lengths,
"penalty_factor": penalty_factor,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 3, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| karel_output_ref_mutator_mode |
Ran ran on a synthesized input |
sunblaze-ucb/SED/program_synthesis/datasets/dataset.py code served (permissive licence) · get_code("d5825e840b5f66b7") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
class Args:
def __init__(self, model_type=None, iterative_search_use_overfit_model=None):
self.model_type = model_type
self.iterative_search_use_overfit_model = iterative_search_use_overfit_model
ARGS = {
'args': Args(model_type='karel-lgrl-overfit', iterative_search_use_overfit_model=None)
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| length_penalty |
Ran ran on a synthesized input |
sunblaze-ucb/SED/program_synthesis/models/beam_search.py code served (permissive licence) · get_code("918b15bc296f5115") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# sequence_lengths should be a tensor of shape [batch_size x beam_size x vocab_size]
# according to the docstring, but the function treats it as if it's just a tensor
# of sequence lengths. Looking at the function body, it just does arithmetic on it,
# so we'll create a tensor of shape [batch_size x beam_size]
sequence_lengths = torch.tensor([
[5.0, 8.0, 3.0, 10.0],
[7.0, 4.0, 9.0, 6.0]
], dtype=torch.float32)
# penalty_factor is a scalar that weights the length penalty
penalty_factor = 0.6
ARGS = {
"sequence_lengths": sequence_lengths,
"penalty_factor": penalty_factor,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_vocab |
Ran ran on a synthesized input |
sunblaze-ucb/SED/program_synthesis/datasets/data.py code served (permissive licence) · get_code("3d825f33d60f08a3") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import os
# Create a temporary vocab file with tab-separated key-value pairs
temp_dir = tempfile.mkdtemp()
vocab_file = os.path.join(temp_dir, "vocab.txt")
with open(vocab_file, 'w') as f:
f.write("apple\t0\n")
f.write("banana\t1\n")
f.write("cherry\t2\n")
ARGS = {
"filename": vocab_file,
"mapping": True
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| relpath |
Ran ran on a synthesized input |
sunblaze-ucb/SED/program_synthesis/datasets/dataset.py code served (permissive licence) · get_code("5957df89a5e9f359") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import os
def relpath(path):
return os.path.join(os.path.dirname(__file__), path)
# The function takes a single parameter 'path' which is a string
# It uses os.path.dirname(__file__) to get the directory of the current file
# and os.path.join to combine it with the provided path
ARGS = {
"path": "subdir/file.txt"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| replace_pad_with_end |
Ran ran on a synthesized input |
sunblaze-ucb/SED/program_synthesis/datasets/data.py code served (permissive licence) · get_code("22048f44250fc46a") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family int, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# The function replace_pad_with_end takes a single parameter 'var'
# It expects var to be a tensor that supports:
# - .clone() method
# - indexing with == comparison (var == PAD_TOKEN)
# - assignment via indexing (var[...] = value)
# Create a small tensor with some PAD_TOKEN values
var = torch.tensor([[-1, 0, 2, 3],
[1, -1, 5, -1]], dtype=torch.long)
ARGS = {
'var': var
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 2, dtype family int, shape (2, 4) — shape (2, 4) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| separate_strategies |
Ran ran on a synthesized input |
sunblaze-ucb/SED/program_synthesis/analysis/organize_results.py code served (permissive licence) · get_code("5c5394d7e87f8bab") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import pandas as pd
import numpy as np
# Create a sample DataFrame that matches the expected structure
# The function filters by DataLabel and Model, so we need those columns
# It also uses Accuracy and Exact columns
data = {
'DataLabel': [
'nearai,,greedy,,1',
'nearai,,greedy,,5',
'nearai,,best_first,,10',
'nearai32,,greedy,,1',
'nearai32,,best_first,,5',
'egnps64,,greedy,,1',
'egnps64,,best_first,,25',
'egnpsgood,,greedy,,1',
'egnpsgood,,best_first,,50',
],
'Model': ['model_a', 'model_a', 'model_a', 'model_b', 'model_b', 'model_a', 'model_a', 'model_b', 'model_b'],
'Accuracy': [0.85, 0.87, 0.89, 0.82, 0.84, 0.90, 0.91, 0.88, 0.92],
'Exact': [0.80, 0.82, 0.85, 0.78, 0.80, 0.87, 0.88, 0.85, 0.90],
'Step': [100, 150, 200, 120, 180, 110, 160, 130, 170],
}
df_orig = pd.DataFrame(data)
# Add some additional columns that might be used in groupby operations
# The function does: groupby(list(set(accuracies) - {'Step', 'Accuracy', 'Exact'}))
# So we need at least one more column that could be grouped
df_orig['ExtraColumn'] = ['val1'] * 9
ARGS = {
'df_orig': df_orig
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| test_cardinality |
Ran ran on a synthesized input |
sunblaze-ucb/SED/program_synthesis/datasets/stats.py code served (permissive licence) · get_code("3c24d293db334783") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Construct test cases with various outputs
tests = [
{'output': 'result_1'},
{'output': 'result_2'},
{'output': 'result_1'}, # duplicate output
{'output': 'result_3'},
]
ARGS = {
'tests': tests
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| code_stat |
Not yet run unverified (harvested, no recorded run) |
sunblaze-ucb/SED/program_synthesis/datasets/stats.py code served (permissive licence) · get_code("039979ec200d5319") How it ran |
| lisp_code_stat |
Not yet run unverified (harvested, no recorded run) |
sunblaze-ucb/SED/program_synthesis/datasets/stats.py code served (permissive licence) · get_code("b85637c0fc6f9373") How it ran |
| table_of_accuracies |
Not yet run unverified (harvested, no recorded run) |
sunblaze-ucb/SED/program_synthesis/analysis/load_results.py code served (permissive licence) · get_code("2e635b1a8221f10c") How it ran |