| add_gumbel_noise |
Ran ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT) |
ming053l/C4-dLLM/c4/decode.py code served (permissive licence) · get_code("027b7c000eacca21") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family float, shape (2, 4) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# Create a small logits tensor (batch_size=2, dim=4)
logits = torch.randn(2, 4)
# Temperature parameter - use a non-zero value to test the Gumbel noise addition
temperature = 0.5
ARGS = {
"logits": logits,
"temperature": temperature,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 2, dtype family float, shape (2, 4) — shape (2, 4) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| build_chat_prompt |
Ran ran; our contract draft was wrong, not the code |
ming053l/C4-dLLM/c4/prompts.py code served (permissive licence) · get_code("d74cea188ab3eae6") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
class MockTokenizer:
"""Minimal tokenizer mock that implements apply_chat_template"""
def apply_chat_template(self, messages, add_generation_prompt=False, tokenize=False):
"""Mock implementation that formats messages into a prompt string"""
prompt = ""
for msg in messages:
role = msg.get('role', 'user')
content = msg.get('content', '')
prompt += f"<{role}>{content}</{role}>"
if add_generation_prompt:
prompt += "<assistant>"
if tokenize:
# Return token ids as a simple mock
return [1, 2, 3]
return prompt
torch.manual_seed(0)
tokenizer = MockTokenizer()
user_content = "What is the capital of France?"
ARGS = {
'tokenizer': tokenizer,
'user_content': user_content
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| code_candidate |
Ran ran; our contract draft was wrong, not the code |
ming053l/C4-dLLM/c4/codegen_grade.py code served (permissive licence) · get_code("6c1e050ae921030c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
_CODE_FENCE_RE = re.compile('```(?:python)?\\s*\\n(.*?)```', re.S)
def extract_code(text):
"""Model was asked for a bare function implementation, no fences -- but strip markdown fences
if the model added them anyway (observed occasionally despite the instruction), taking the
first fenced block if present, else the raw text as-is."""
m = _CODE_FENCE_RE.search(text)
return m.group(1) if m else text
def code_candidate(text):
"""CVEE's extracted candidate $a_t$ for humaneval/mbpp: the completed function definition, as a
normalized string, or None when no function has been started yet (which leaves run/changes
paused and fails the confidence gate automatically, exactly like a missing number on GSM8K).
Everything from the buffer's start through the end of the function's body is kept -- imports
and comments the model emits before the `def` are part of the program and must not be dropped.
The body ends at the first non-blank line at column 0 after the definition, Python's own
end-of-block rule; whatever follows (usage examples, asserts, prose, or the not-yet-decoded
tail's provisional garbage) is not part of the answer and is excluded so that a settled
function reads as settled."""
code = extract_code(text)
lines = code.split('\n')
start = next((i for i, l in enumerate(lines) if l.startswith('def ') or l.startswith('async def ')), None)
if start is None:
return None
end = start
for i in range(start + 1, len(lines)):
line = lines[i]
if not line.strip():
continue
if line[0] in ' \t':
end = i
continue
break
cand = '\n'.join(lines[:end + 1]).rstrip()
return cand or None
ARGS = {
'text': '''def add(a, b):
"""Add two numbers."""
return a + b
print(add(1, 2))
'''
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| conf_floor |
Ran ran, violated the contract we drafted |
ming053l/C4-dLLM/c4/policies.py code served (permissive licence) · get_code("4688f8835bbc4c36") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import math
torch_imported = False
try:
import torch
torch.manual_seed(0)
torch_imported = True
except ImportError:
pass
# Construct arguments for conf_floor function
run = 10
changes = 2
conf = 0.95
gamma = 16.0
p_min = 3
tau = 0.9
ARGS = {
"run": run,
"changes": changes,
"conf": conf,
"gamma": gamma,
"p_min": p_min,
"tau": tau,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| confidence_only |
Ran ran, violated the contract we drafted |
ming053l/C4-dLLM/c4/policies.py code served (permissive licence) · get_code("c3f877271843c051") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# The function signature is: confidence_only(conf, tau=0.7, **_)
# It checks: conf is not None and conf >= tau
# So `conf` should be a numeric value (float or tensor) that can be compared to tau
# conf can be:
# - A float/int value
# - A tensor (will be compared with >=)
# - None (returns False)
# Let's test with a float value that's greater than tau
conf_value = 0.85
ARGS = {
"conf": conf_value,
"tau": 0.7,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_boxed |
Ran ran; our contract draft was wrong, not the code |
ming053l/C4-dLLM/c4/extract.py code served (permissive licence) · get_code("414a8b03b41334b1") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
_NUM_RE = re.compile('(-?[$0-9.,]{2,})|(-?[0-9]+)')
_BOXED_RE = re.compile('\\\\boxed\\{((?:[^{}]|\\{[^{}]*\\})*)\\}')
def extract_number(text):
"""Last number-looking substring in `text`, normalized (strip $ and , and trailing .)."""
matches = _NUM_RE.findall(text)
if not matches:
return None
last = next((g for g in matches[-1] if g))
return last.strip().rstrip('.').replace('$', '').replace(',', '')
_BARE_FRAC_RE = re.compile('\\\\frac(\\d)(\\d)(?!\\d)')
def _normalize_math(s):
"""Loose normalization for MATH-style answers: strip whitespace/$, drop \\left \\right, collapse
spaces. Not a full CAS-equivalence checker (out of scope) -- matches the "lenient" grading
philosophy already used for GSM8K/MMLU in this codebase, not a claim of mathematical rigor.
Also inserts braces around MATH-500's own occasional bare-digit \\frac shorthand (e.g. the
ground-truth string "\\frac65" meaning 6/5, confirmed present verbatim in 3/500 MATH-500
answers) so it compares equal to a model's equivalently-valued but properly-braced
"\\frac{6}{5}" -- both denote the identical fraction, so treating them as a mismatch was
exactly the kind of formatting-not-correctness gap this function already exists to avoid."""
s = s.strip()
s = s.replace('\\left', '').replace('\\right', '')
s = s.replace(' ', '')
s = s.replace('\\!', '').replace('\\,', '')
s = s.rstrip('.')
if s.startswith('$') and s.endswith('$'):
s = s[1:-1]
s = _BARE_FRAC_RE.sub('\\\\frac{\\1}{\\2}', s)
return s
def extract_boxed(text):
"""Last \\boxed{...} content; fallback to the last number if no \\boxed{} is present (mirrors
the GSM8K/MMLU lenient-grading philosophy: don't zero out an otherwise-correct answer over
formatting)."""
matches = _BOXED_RE.findall(text)
if matches:
return _normalize_math(matches[-1])
num = extract_number(text)
return _normalize_math(num) if num is not None else None
ARGS = {
"text": "The answer is \\boxed{42}"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_code |
Ran ran; our contract draft was wrong, not the code |
ming053l/C4-dLLM/c4/codegen_grade.py code served (permissive licence) · get_code("1660493478fd2823") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
_CODE_FENCE_RE = re.compile('```(?:python)?\\s*\\n(.*?)```', re.S)
def extract_code(text):
"""Model was asked for a bare function implementation, no fences -- but strip markdown fences
if the model added them anyway (observed occasionally despite the instruction), taking the
first fenced block if present, else the raw text as-is."""
m = _CODE_FENCE_RE.search(text)
return m.group(1) if m else text
# Test case 1: text with markdown fences
text_with_fences = """Here's the code:
def hello():
return "world"
That's it!"""
# Test case 2: text without fences (bare function)
text_without_fences = """def hello():
return "world"
"""
ARGS = {
"text": text_with_fences
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_letter |
Ran ran; our contract draft was wrong, not the code |
ming053l/C4-dLLM/c4/extract.py code served (permissive licence) · get_code("719354fa9daaeb3b") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
_LETTER_RE = re.compile('\\b([A-M])\\b')
def extract_letter(text):
"""First standalone A/B/C/D in `text`."""
m = _LETTER_RE.search(text)
return m.group(1) if m else None
ARGS = {
"text": "This is a test with letter B in it"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| extract_number |
Ran ran; our contract draft was wrong, not the code |
ming053l/C4-dLLM/c4/extract.py code served (permissive licence) · get_code("d9cea69389fc2c87") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
_NUM_RE = re.compile('(-?[$0-9.,]{2,})|(-?[0-9]+)')
def extract_number(text):
"""Last number-looking substring in `text`, normalized (strip $ and , and trailing .)."""
matches = _NUM_RE.findall(text)
if not matches:
return None
last = next((g for g in matches[-1] if g))
return last.strip().rstrip('.').replace('$', '').replace(',', '')
ARGS = {
"text": "The price is $1,234.50 and the quantity is 42"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| generate_full |
Ran ran; our contract draft was wrong, not the code |
ming053l/C4-dLLM/c4/decode.py code served (permissive licence) · get_code("e58520fdde4188ff") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
import numpy as np
torch.manual_seed(0)
# Build a minimal model that satisfies the function's usage
class MinimalModel(nn.Module):
def __init__(self, vocab_size=126340, hidden_dim=8):
super().__init__()
self.embedding = nn.Embedding(vocab_size, hidden_dim)
self.linear = nn.Linear(hidden_dim, vocab_size)
self.device_attr = 'cpu'
@property
def device(self):
return torch.device(self.device_attr)
def forward(self, x):
# x shape: [batch, seq_len]
# returns object with .logits attribute
embedded = self.embedding(x)
logits = self.linear(embedded)
class LogitOutput:
def __init__(self, logits_tensor):
self.logits = logits_tensor
return LogitOutput(logits)
# Create model
model = MinimalModel(vocab_size=126340, hidden_dim=8)
model.eval()
# Create prompt: [batch=1, seq_len=5]
prompt = torch.randint(0, 126336, (1, 5), dtype=torch.long)
# Parameters
steps = 4
gen_length = 8
block_length = 4
temperature = 0.0
remasking = 'low_confidence'
mask_id = 126336
logit_shift = False
constraints = {0: 100, 2: 200} # optional constraints: gen-relative pos -> token_id
ARGS = {
'model': model,
'prompt': prompt,
'steps': steps,
'gen_length': gen_length,
'block_length': block_length,
'temperature': temperature,
'remasking': remasking,
'mask_id': mask_id,
'logit_shift': logit_shift,
'constraints': constraints,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_num_transfer_tokens |
Ran ran, honoured the contract we drafted |
ming053l/C4-dLLM/c4/decode.py code served (permissive licence) · get_code("befea8ff69c3c190") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 2, dtype family int, shape (2, 3) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# mask_index is a 2D boolean or integer tensor where:
# - dim 0 is batch size
# - dim 1 is sequence length
# We need to sum along dim 1, so let's create a mask with some True values
mask_index = torch.tensor([
[True, True, False, True],
[True, False, False, False]
], dtype=torch.bool)
# steps is an integer representing the number of steps to distribute tokens across
steps = 3
ARGS = {
"mask_index": mask_index,
"steps": steps
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type Tensor, ndim 2, dtype family int, shape (2, 3) — shape (2, 3) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| grade_countdown |
Ran ran, violated the contract we drafted |
ming053l/C4-dLLM/c4/planning.py code served (permissive licence) · get_code("cb2a9aa46f28b87c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
_EQN_RE = re.compile('Answer:\\s*([-+*/()\\d\\s.]+?)\\s*=\\s*-?\\d+(?:\\.\\d+)?', re.IGNORECASE)
_EQN_RE_NOEQ = re.compile('Answer:\\s*([-+*/()\\d\\s.]+)')
def grade_countdown(pred_text, gt):
nums, target = (gt['nums'], gt['target'])
m = _EQN_RE.search(pred_text) or _EQN_RE_NOEQ.search(pred_text)
if not m:
return False
expr = m.group(1).strip().rstrip('=').strip()
used = sorted((int(x) for x in re.findall('\\d+', expr)))
if used != sorted(nums):
return False
try:
val = eval(expr, {'__builtins__': {}}, {})
except Exception:
return False
return isinstance(val, (int, float)) and abs(val - target) < 1e-06
ARGS = {
'pred_text': 'Answer: 5 + 3 * 2 = 11',
'gt': {
'nums': [2, 3, 5],
'target': 11
}
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| grade_humaneval |
Ran ran, violated the contract we drafted |
ming053l/C4-dLLM/c4/codegen_grade.py code served (permissive licence) · get_code("96bb76592ee05d4b") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import re
import subprocess
import sys
import tempfile
# Test function that checks if a simple function works
test_code = """
def check(candidate):
assert candidate(5) == 10, f"Expected 10, got {candidate(5)}"
assert candidate(3) == 6, f"Expected 6, got {candidate(3)}"
"""
# Simple candidate function implementation
pred_text = """
def add_five(x):
return x * 2
"""
# Ground truth structure
gt = {
"test": test_code,
"entry_point": "add_five"
}
ARGS = {
"pred_text": pred_text,
"gt": gt
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_sudoku |
Ran ran; our contract draft was wrong, not the code |
ming053l/C4-dLLM/c4/planning.py code served (permissive licence) · get_code("aadbb0ccde54a8c9") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import random
ARGS = {
'n': 2,
'seed': 42,
'skip': 1,
'n_blank': 5
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| taec_adaptive |
Ran ran, violated the contract we drafted |
ming053l/C4-dLLM/c4/policies.py code served (permissive licence) · get_code("dd1a47966def149c") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned bool, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import math
torch_imported = False
try:
import torch
torch.manual_seed(0)
torch_imported = True
except ImportError:
pass
ARGS = {
"run": 5,
"changes": 2,
"progress": 0.5,
"gamma": 16.0,
"p_min": 3,
"min_progress": 0.1,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type bool, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|