| collect_topk_images |
Ran ran; our contract draft was wrong, not the code |
najeebjebreel/lira_analysis/comprehensive_analysis/compose_top_vulnerable.py code served (permissive licence) · get_code("30c1c468615ed8aa") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import shutil
from pathlib import Path
# Create a temporary directory structure that mimics the expected layout
temp_dir = tempfile.mkdtemp()
repo_root = Path(temp_dir) / "repo"
dest = Path(temp_dir) / "dest"
repo_root.mkdir(parents=True, exist_ok=True)
# Create the analysis_results structure
analysis_results = repo_root / 'analysis_results'
analysis_results.mkdir(parents=True, exist_ok=True)
# Create figures subdirectory
figures_root = analysis_results / 'figures'
figures_root.mkdir(parents=True, exist_ok=True)
# Create some subdirectories in analysis_results (not under figures)
subdir1 = analysis_results / 'subdir1'
subdir1.mkdir(parents=True, exist_ok=True)
# Create dummy PNG files that match the pattern 'top*vulnerable*.png'
# These should be OUTSIDE of figures_root to be collected
(subdir1 / 'top_vulnerable_001.png').touch()
(subdir1 / 'top_vulnerable_002.png').touch()
# Create another subdir with matching files
subdir2 = analysis_results / 'subdir2'
subdir2.mkdir(parents=True, exist_ok=True)
(subdir2 / 'top_results_vulnerable_003.png').touch()
# Create files in figures_root that should be skipped (relative_to succeeds)
(figures_root / 'top_vulnerable_skip.png').touch()
ARGS = {
'repo_root': repo_root,
'dest': dest,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| compute_precision |
Ran ran, honoured the contract we drafted |
najeebjebreel/lira_analysis/comprehensive_analysis/metrics.py code served (permissive licence) · get_code("54526585121a623a") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Construct arguments for compute_precision
# The function takes three parameters: tpr, fpr, and prior (with default 0.5)
ARGS = {
"tpr": 0.9, # True positive rate: 90% of members correctly identified
"fpr": 0.01, # False positive rate: 1% of non-members incorrectly flagged
"prior": 0.5, # Prior probability of membership: 50% (balanced dataset)
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| compute_roc_metrics |
Ran ran; our contract draft was wrong, not the code |
najeebjebreel/lira_analysis/comprehensive_analysis/metrics.py code served (permissive licence) · get_code("99a772f122b776db") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Set random seed for reproducibility
np.random.seed(0)
# Create sample attack scores (continuous values, higher = more likely member)
# Using 10 samples for a small but meaningful dataset
scores = np.array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.85, 0.95], dtype=np.float32)
# Create ground truth membership labels (0 = non-member, 1 = member)
# Mix of both classes to avoid degenerate case
labels = np.array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1], dtype=np.int32)
ARGS = {
"scores": scores,
"labels": labels,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, sklearn — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| compute_tpr_at_fpr |
Ran ran; our contract draft was wrong, not the code |
najeebjebreel/lira_analysis/comprehensive_analysis/metrics.py code served (permissive licence) · get_code("cf5ec905e15dd612") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Create sample ROC curve data
# fpr: False positive rates (should be monotonically increasing)
fpr = np.array([0.0, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1.0])
# tpr: True positive rates (should be monotonically increasing)
tpr = np.array([0.0, 0.3, 0.5, 0.65, 0.8, 0.85, 0.95, 1.0])
# thresholds: Decision thresholds (should be monotonically decreasing)
thresholds = np.array([np.inf, 0.9, 0.7, 0.5, 0.3, 0.2, 0.1, 0.0])
# target_fpr: We want to find TPR at FPR <= 0.01
target_fpr = 0.01
ARGS = {
"fpr": fpr,
"tpr": tpr,
"thresholds": thresholds,
"target_fpr": target_fpr,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| create_output_directory |
Ran ran; our contract draft was wrong, not the code |
najeebjebreel/lira_analysis/comprehensive_analysis/run_analysis.py code served (permissive licence) · get_code("ca73aedeabad1c37") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned PosixPath. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from pathlib import Path
import tempfile
# Create a temporary directory to serve as base_dir
base_dir = Path(tempfile.gettempdir()) / "test_output_base"
# Create an exp_path with at least 4 parts to trigger the dataset/model/run_name extraction
# The path structure: .../dataset_name/model_name/run_name
exp_path = Path("/some/experiment/my_dataset/my_model/run_001")
ARGS = {
"base_dir": base_dir,
"exp_path": exp_path,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type PosixPath — a PosixPath output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a PosixPath output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| fpr_to_tag |
Ran ran; our contract draft was wrong, not the code |
najeebjebreel/lira_analysis/comprehensive_analysis/reproducibility_analysis.py code served (permissive licence) · get_code("e214cc0b45e0daee") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
def fpr_to_tag(fpr: float) -> str:
pct = fpr * 100.0
value = f'{pct:.3f}'.rstrip('0').rstrip('.')
return value.replace('.', 'p') + 'pct'
ARGS = {
'fpr': 0.05
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| fpr_to_tag |
Ran ran; our contract draft was wrong, not the code |
najeebjebreel/lira_analysis/comprehensive_analysis/run_analysis.py code served (permissive licence) · get_code("408beb717e5ac9fd") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
# Construct valid arguments for fpr_to_tag
# The function takes a single parameter: fpr (float)
# It converts the float to a percentage, replaces '.' with 'p', and appends 'pct'
ARGS = {
'fpr': 0.05
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| freedman_diaconis_bins |
Ran ran; our contract draft was wrong, not the code |
najeebjebreel/lira_analysis/comprehensive_analysis/plot_benchmark_distribution.py code served (permissive licence) · get_code("7f54c1a69ee2722e") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Create a simple numpy array with varied values for testing
# The function expects a numpy array and will compute bins based on the Freedman-Diaconis rule
values = np.array([1.2, 2.5, 3.1, 4.7, 5.2, 6.8, 7.1, 8.3, 9.5, 10.2,
11.1, 12.4, 13.7, 14.2, 15.8, 16.3, 17.9, 18.5, 19.1, 20.0])
ARGS = {
"values": values,
"max_bins": 80,
"min_bins": 50
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_attack_methods |
Ran ran; our contract draft was wrong, not the code |
najeebjebreel/lira_analysis/attack.py code served (permissive licence) · get_code("970e527fa55fb4bf") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
torch.manual_seed(0)
# The function takes a config dict and extracts attack methods from it
# It accesses config.get('attack', {}).get('method', [])
# and checks if the result is a string or converts it to a list
# Test case 1: config with attack.method as a string
config_string = {
'attack': {
'method': 'fgsm'
}
}
# Test case 2: config with attack.method as a list
config_list = {
'attack': {
'method': ['fgsm', 'pgd', 'cw']
}
}
# Test case 3: config with missing attack key (should return empty list)
config_empty = {}
# Test case 4: config with attack but no method key
config_no_method = {
'attack': {}
}
# Use the first config as the primary test case
ARGS = {
'config': config_string
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_experiment_info |
Ran ran; our contract draft was wrong, not the code |
najeebjebreel/lira_analysis/comprehensive_analysis/analysis_utils.py code served (permissive licence) · get_code("33f9972a2979215a") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import yaml
from pathlib import Path
# Create a temporary directory structure for the experiment
temp_dir = tempfile.TemporaryDirectory()
temp_path = Path(temp_dir.name)
# Create nested directory structure matching expected format
# Format: experiments/dataset/model/config_name
experiment_path = temp_path / "experiments" / "cifar10" / "resnet18" / "experiment_001"
experiment_path.mkdir(parents=True, exist_ok=True)
# Create train_config.yaml
train_config = {
'training': {
'num_shadow_models': 5,
'epochs': 100,
},
'model': {
'architecture': 'resnet18',
},
'dataset': {
'name': 'cifar10',
}
}
train_config_path = experiment_path / 'train_config.yaml'
with open(train_config_path, 'w') as f:
yaml.dump(train_config, f)
# Create attack_config.yaml
attack_config = {
'attack': {
'type': 'membership_inference',
'threshold': 0.5,
}
}
attack_config_path = experiment_path / 'attack_config.yaml'
with open(attack_config_path, 'w') as f:
yaml.dump(attack_config, f)
# Define arguments for get_experiment_info
ARGS = {
'experiment_dir': str(experiment_path)
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- yaml — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_experiment_config |
Ran ran; our contract draft was wrong, not the code |
najeebjebreel/lira_analysis/comprehensive_analysis/analysis_utils.py code served (permissive licence) · get_code("9ee0cfeab3854a74") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
import yaml
from pathlib import Path
# Create temporary directory with test config files
temp_dir = tempfile.mkdtemp()
experiment_dir = Path(temp_dir)
# Create train_config.yaml
train_config = {
'training': {
'num_shadow_models': 5,
'epochs': 10,
'batch_size': 32,
'learning_rate': 0.001
},
'model': {
'architecture': 'resnet18',
'pretrained': False
}
}
train_config_path = experiment_dir / 'train_config.yaml'
with open(train_config_path, 'w') as f:
yaml.dump(train_config, f)
# Create attack_config.yaml
attack_config = {
'attack': {
'type': 'membership_inference',
'threshold': 0.5,
'num_queries': 1000
},
'defense': {
'method': 'differential_privacy',
'epsilon': 1.0
}
}
attack_config_path = experiment_dir / 'attack_config.yaml'
with open(attack_config_path, 'w') as f:
yaml.dump(attack_config, f)
ARGS = {
'experiment_dir': experiment_dir
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- yaml — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_pooled_thresholds |
Ran ran, honoured the contract we drafted |
najeebjebreel/lira_analysis/comprehensive_analysis/threshold_distribution.py code served (permissive licence) · get_code("67d36c79acbea5e2") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (3,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import tempfile
from pathlib import Path
import pandas as pd
import numpy as np
# Create temporary CSV files with threshold data
temp_dir = tempfile.TemporaryDirectory()
temp_path = Path(temp_dir.name)
# Create first CSV file
csv_path_1 = temp_path / "thresholds_1.csv"
df_1 = pd.DataFrame({
'mode': ['target', 'target', 'baseline'],
'attack': ['fgsm', 'fgsm', 'pgd'],
'target_fpr': [0.1, 0.1, 0.1],
'prior': [0.01, 0.01, 0.01],
'threshold': [0.5, 0.6, 0.7]
})
df_1.to_csv(csv_path_1, index=False)
# Create second CSV file
csv_path_2 = temp_path / "thresholds_2.csv"
df_2 = pd.DataFrame({
'mode': ['target', 'target', 'baseline'],
'attack': ['fgsm', 'pgd', 'fgsm'],
'target_fpr': [0.1, 0.2, 0.1],
'prior': [0.01, 0.01, 0.02],
'threshold': [0.55, 0.65, 0.75]
})
df_2.to_csv(csv_path_2, index=False)
ARGS = {
'csv_paths': [csv_path_1, csv_path_2],
'attack': 'fgsm',
'target_fpr': 0.1,
'prior': 0.01
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (3,) — shape (3,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_thresholds_filtered |
Ran ran, honoured the contract we drafted |
najeebjebreel/lira_analysis/comprehensive_analysis/threshold_distribution.py code served (permissive licence) · get_code("a6bc219cba07af29") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (2,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import pandas as pd
import numpy as np
from pathlib import Path
import tempfile
import os
# Create a temporary CSV file with test data
temp_dir = tempfile.mkdtemp()
csv_path = Path(temp_dir) / "thresholds.csv"
# Create a DataFrame with the required columns
df = pd.DataFrame({
'mode': ['target', 'target', 'other', 'target'],
'attack': ['membership', 'membership', 'membership', 'other_attack'],
'target_fpr': [0.01, 0.01, 0.01, 0.01],
'prior': [0.01, 0.01, 0.01, 0.01],
'threshold': [0.5, 0.6, 0.7, 0.8]
})
df.to_csv(csv_path, index=False)
ARGS = {
'csv_path': csv_path,
'attack': 'membership',
'target_fpr': 0.01,
'prior_equals': 0.01,
'mode_equals': 'target'
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family float, shape (2,) — shape (2,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| mean_std |
Ran ran, honoured the contract we drafted |
najeebjebreel/lira_analysis/comprehensive_analysis/rank_stability_utils.py code served (permissive licence) · get_code("2aa1b93743d08c9d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Construct a valid numpy array with a mix of finite and non-finite values
arr = np.array([1.0, 2.5, 3.7, np.nan, 5.2, np.inf, 7.1, -np.inf, 9.0])
ARGS = {
"arr": arr
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| normalize_columns |
Ran ran; our contract draft was wrong, not the code |
najeebjebreel/lira_analysis/comprehensive_analysis/loss_ratio_tpr.py code served (permissive licence) · get_code("b4263be4e1529343") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import pandas as pd
# Create a sample DataFrame with columns that the function will rename
df = pd.DataFrame({
'Loss_Ratio': [0.1, 0.2, 0.3],
'loss_ratio': [0.15, 0.25, 0.35],
'TPR_0.1FPR': [0.5, 0.6, 0.7],
'tpr': [0.55, 0.65, 0.75],
'TPR@0.1%FPR': [0.52, 0.62, 0.72],
'other_column': [1, 2, 3]
})
ARGS = {
'df': df
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- pandas — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| parse_panel |
Ran ran; our contract draft was wrong, not the code |
najeebjebreel/lira_analysis/comprehensive_analysis/plot_benchmark_distribution.py code served (permissive licence) · get_code("08a2335c3fcc3802") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
from pathlib import Path
ARGS = {
"panel_arg": "MyPanel=/tmp/test_path"
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| resolve_config_paths |
Ran ran; our contract draft was wrong, not the code |
najeebjebreel/lira_analysis/comprehensive_analysis/plot_benchmark_distribution.py code served (permissive licence) · get_code("e4629aee855b31c7") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from pathlib import Path
# Create temporary directories for testing
config_dir = Path("/tmp/config")
repo_root_dir = Path("/tmp/repo")
cli_experiments_root_dir = Path("/tmp/experiments")
# Create a minimal config dictionary
config = {
'experiments_root': 'experiments',
'panels': [
{
'title': 'Panel 1',
'path': 'results/panel1.png'
},
{
'title': 'Panel 2',
'path': '/absolute/path/panel2.png'
}
],
'out': 'output/figures/result.pdf'
}
# Define ARGS with all required parameters
ARGS = {
'config': config,
'config_path': config_dir / 'config.yaml',
'repo_root': repo_root_dir,
'cli_experiments_root': cli_experiments_root_dir
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| robust_stats_1d |
Ran ran; our contract draft was wrong, not the code |
najeebjebreel/lira_analysis/comprehensive_analysis/rank_stability_utils.py code served (permissive licence) · get_code("1a7471281a81be03") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# The function robust_stats_1d takes a single parameter: x (np.ndarray)
# It converts x to a float array, filters out non-finite values, and computes statistics
# We need to provide a valid numpy array with some numeric data
x_data = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0])
ARGS = {
'x': x_data
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| safe_spearman |
Ran ran; our contract draft was wrong, not the code |
najeebjebreel/lira_analysis/comprehensive_analysis/rank_stability_utils.py code served (permissive licence) · get_code("34e81e2a77283b09") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
from scipy.stats import spearmanr
def safe_spearman(a: np.ndarray, b: np.ndarray) -> float:
if a.size < 2:
return np.nan
return float(spearmanr(a, b).correlation)
# Setup code
np.random.seed(42)
# Create two numpy arrays with at least 2 elements for valid Spearman correlation
a = np.array([1.5, 2.3, 3.1, 4.8, 5.2])
b = np.array([2.1, 3.5, 3.0, 5.1, 6.0])
ARGS = {
"a": a,
"b": b,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, scipy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| compute_identical_over_thresholds |
Not yet run unverified (harvested, no recorded run) |
najeebjebreel/lira_analysis/comprehensive_analysis/reproducibility_analysis.py code served (permissive licence) · get_code("a412bade5c864edb") How it ran |
| create_output_directory |
Not yet run unverified (harvested, no recorded run) |
najeebjebreel/lira_analysis/comprehensive_analysis/analysis_utils.py code served (permissive licence) · get_code("982573cf8895bdfa") How it ran |
| generate_reproducibility_panels |
Not yet run unverified (harvested, no recorded run) |
najeebjebreel/lira_analysis/comprehensive_analysis/reproducibility_analysis.py code served (permissive licence) · get_code("0128a2f4eb9aceaf") How it ran |
| save_latex_summary |
Not yet run unverified (harvested, no recorded run) |
najeebjebreel/lira_analysis/comprehensive_analysis/run_analysis.py code served (permissive licence) · get_code("096a5b0d17325f3e") How it ran |