| compute_cost |
Ran ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT) |
amazon-research/fair-ordinal-regression/learn_fair_threshold_model/evaluation.py code served (permissive licence) · get_code("77331f760d7f305b") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float32, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Create small ground-truth labels (indices into cost_matrix rows)
true = np.array([0, 1, 0, 2])
# Create small predictions (indices into cost_matrix columns)
pred = np.array([1, 1, 2, 0])
# Create a cost matrix where cost_matrix[i, j] is the cost of
# misclassifying ground-truth label i as prediction j
# Shape: (num_classes, num_classes) = (3, 3)
cost_matrix = np.array([
[0, 5, 10],
[3, 0, 7],
[8, 2, 0]
], dtype=np.float32)
ARGS = {
"true": true,
"pred": pred,
"cost_matrix": cost_matrix
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float32, ndim 0, dtype family float, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| evaluate_DP_exact |
Ran ran; our contract draft was wrong, not the code |
amazon-research/fair-ordinal-regression/learn_fair_threshold_model/evaluation.py code served (permissive licence) · get_code("84b7d112ccd1070e") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float64, ndim 0, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Small example with 2 protected groups, 3 label classes, 8 samples
labels = np.array([0, 1, 2, 0, 1, 2, 0, 1], dtype=np.int64)
prot_attribute = np.array([0, 0, 0, 0, 1, 1, 1, 1], dtype=np.int64)
nr_groups = 2
ARGS = {
"labels": labels,
"prot_attribute": prot_attribute,
"nr_groups": nr_groups,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float64, ndim 0, dtype family float — no output shape is stated: observe_signature.py recorded float ndim 0, which disagrees with the fixture run
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| evaluate_EO_exact |
Ran ran; our contract draft was wrong, not the code |
amazon-research/fair-ordinal-regression/learn_fair_threshold_model/evaluation.py code served (permissive licence) · get_code("83a63654701028c4") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float64, ndim 0, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Create small test data for the function
np.random.seed(0)
# Number of samples
n = 10
# Number of classes (k)
k = 3
# Number of protected groups
nr_groups = 2
# Generate true labels in {0, 1, 2}
true_labels = np.random.randint(0, k, size=n)
# Generate predicted labels in {0, 1, 2}
predicted_labels = np.random.randint(0, k, size=n)
# Generate protected attributes in {0, 1}
prot_attribute = np.random.randint(0, nr_groups, size=n)
ARGS = {
"true_labels": true_labels,
"predicted_labels": predicted_labels,
"prot_attribute": prot_attribute,
"nr_groups": nr_groups
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type float64, ndim 0, dtype family float — no output shape is stated: observe_signature.py recorded float ndim 0, which disagrees with the fixture run
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| find_best_left_shift |
Ran ran; our contract draft was wrong, not the code |
amazon-research/fair-ordinal-regression/learn_fair_threshold_model/local_search_demographic_parity.py code served (permissive licence) · get_code("f6a02bddcd7a19b0") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
np.random.seed(0)
# Analyze what the function needs:
# - n: int (number of items)
# - n_thresh: int (number of thresholds)
# - nr_groups: int (number of protected groups)
# - labels: array of shape (n,) with values in range [0, nr_groups)
# - pred: array (appears unused in function body, but is a parameter)
# - prot_attribute: array of shape (n,) with values in range [0, nr_groups)
# - threshold_indices: array of shape (n_thresh,) with values in range [0, n]
# - helper_table_T: 2D array of shape (n_thresh+1, nr_groups)
# - helper_table_P: 2D array of shape (nr_groups, nr_groups)
# - el_per_group: array of shape (nr_groups,) with positive values
# - cost: scalar float
# - cost_matrix: 2D array of shape (nr_groups, n_thresh+1)
# - lambd: scalar float
# - scores: array of shape (n,) with float values
n = 10
n_thresh = 4
nr_groups = 3
# Construct labels: array of integers in [0, nr_groups)
labels = np.array([0, 1, 2, 0, 1, 2, 0, 1, 2, 0])
# Construct pred (unused but required)
pred = np.random.rand(n)
# Construct prot_attribute: array of integers in [0, nr_groups)
prot_attribute = np.array([0, 1, 2, 0, 1, 2, 0, 1, 2, 0])
# Construct threshold_indices: array of shape (n_thresh,) with values in range [1, n]
# Make sure not all are 0, and some are greater than previous
threshold_indices = np.array([2, 4, 6, 8])
# Construct helper_table_T: shape (n_thresh+1, nr_groups)
helper_table_T = np.random.rand(n_thresh + 1, nr_groups)
# Construct helper_table_P: shape (nr_groups, nr_groups)
helper_table_P = np.random.rand(nr_groups, nr_groups)
# Construct el_per_group: shape (nr_groups,) with positive values
el_per_group = np.array([3.0, 3.0, 4.0])
# Construct cost: scalar
cost = 5.0
# Construct cost_matrix: shape (nr_groups, n_thresh+1)
cost_matrix = np.random.rand(nr_groups, n_thresh + 1)
# Construct lambd: scalar
lambd = 0.5
# Construct scores: array of shape (n,)
scores = np.linspace(0.0, 1.0, n)
ARGS = {
'n': n,
'n_thresh': n_thresh,
'nr_groups': nr_groups,
'labels': labels,
'pred': pred,
'prot_attribute': prot_attribute,
'threshold_indices': threshold_indices,
'helper_table_T': helper_table_T,
'helper_table_P': helper_table_P,
'el_per_group': el_per_group,
'cost': cost,
'cost_matrix': cost_matrix,
'lambd': lambd,
'scores': scores,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| find_best_left_shift |
Ran ran; our contract draft was wrong, not the code |
amazon-research/fair-ordinal-regression/learn_fair_threshold_model/local_search_equal_opportunity.py code served (permissive licence) · get_code("78c7c4822d5efb84") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Set random seed for reproducibility
np.random.seed(0)
# Define parameters based on function signature and usage
n = 10 # number of samples
n_thresh = 5 # number of thresholds
nr_groups = 3 # number of protected attribute groups
# labels: array of class labels for each sample, used as indices into cost_matrix
# Range should be [0, num_classes)
labels = np.array([0, 1, 0, 2, 1, 0, 1, 2, 0, 1], dtype=int)
# pred: predictions (not directly used in function body, but required parameter)
pred = np.array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95], dtype=float)
# prot_attribute: protected attribute group for each sample, indices into nr_groups
prot_attribute = np.array([0, 1, 2, 0, 1, 2, 0, 1, 2, 0], dtype=int)
# threshold_indices: indices where thresholds are placed (must have length n_thresh)
# Start with some valid indices that aren't all 0
threshold_indices = np.array([2, 4, 6, 8, 10], dtype=int)
# scores: score for each sample, used for comparison with MY_EPS tolerance
scores = np.array([0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.95], dtype=float)
# helper_table_T: shape (num_classes, n_thresh, nr_groups)
# Used as: helper_table_T[:labels[ell], thresh, :]
num_classes = 3
helper_table_T = np.random.rand(num_classes, n_thresh + 1, nr_groups) * 5
# helper_table_P: shape (nr_groups, nr_groups)
# Used for computing fairness metrics via transpose
helper_table_P = np.random.rand(nr_groups, nr_groups) * 10
# el_per_group_that_are_bigger: shape (nr_groups,)
# Used for normalization: helper_table_P / el_per_group_that_are_bigger
el_per_group_that_are_bigger = np.array([2.0, 3.0, 2.5])
# cost: scalar cost value
cost = 5.0
# cost_matrix: shape (num_classes, n_thresh + 1)
# Accessed as cost_matrix[labels[...], thresh/thresh+1]
cost_matrix = np.random.rand(num_classes, n_thresh + 1) * 10
# lambd: regularization parameter for fairness
lambd = 0.5
ARGS = {
'n': n,
'n_thresh': n_thresh,
'nr_groups': nr_groups,
'labels': labels,
'pred': pred,
'prot_attribute': prot_attribute,
'threshold_indices': threshold_indices,
'helper_table_T': helper_table_T,
'helper_table_P': helper_table_P,
'el_per_group_that_are_bigger': el_per_group_that_are_bigger,
'cost': cost,
'cost_matrix': cost_matrix,
'lambd': lambd,
'scores': scores,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| find_best_right_shift |
Ran ran; our contract draft was wrong, not the code |
amazon-research/fair-ordinal-regression/learn_fair_threshold_model/local_search_demographic_parity.py code served (permissive licence) · get_code("527f5c42a18a3689") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Set random seed for reproducibility
np.random.seed(0)
# Parameters based on function signature and usage
n = 10 # number of elements
n_thresh = 3 # number of thresholds
nr_groups = 2 # number of protected attribute groups
num_classes = 4 # for cost_matrix rows
# labels: array of class labels for each element, indexed by threshold_indices
labels = np.array([0, 1, 2, 3, 1, 0, 2, 1, 3, 2])
# pred: predictions (used indirectly through indexing)
pred = np.array([0, 1, 1, 3, 1, 0, 2, 1, 3, 2])
# prot_attribute: protected attribute values (group membership for each element)
# 1-indexed access, so we need n elements
prot_attribute = np.array([0, 1, 0, 1, 0, 1, 0, 1, 0, 1])
# threshold_indices: indices of thresholds (must be sorted and <= n)
threshold_indices = np.array([2, 5, 10])
# helper_table_T: confusion matrix for thresholds
# shape: (n_thresh, nr_groups) - tracks transitions between thresholds
helper_table_T = np.array([[3, 2], [2, 3], [2, 3]], dtype=float)
# helper_table_P: parity matrix
# shape: (nr_groups, nr_groups) - tracks protected attribute distribution
helper_table_P = np.array([[5, 5], [5, 5]], dtype=float)
# el_per_group: number of elements per group
el_per_group = np.array([5.0, 5.0])
# cost: scalar cost value
cost = 2.5
# cost_matrix: cost for each (class, threshold) combination
# shape: (num_classes, n_thresh + 1)
cost_matrix = np.array([
[0.1, 0.2, 0.3, 0.4],
[0.15, 0.25, 0.35, 0.45],
[0.12, 0.22, 0.32, 0.42],
[0.18, 0.28, 0.38, 0.48]
], dtype=float)
# lambd: regularization parameter
lambd = 0.5
# scores: scores for ranking/ordering elements
scores = np.array([0.1, 0.2, 0.25, 0.3, 0.35, 0.4, 0.5, 0.6, 0.7, 0.8])
ARGS = {
'n': n,
'n_thresh': n_thresh,
'nr_groups': nr_groups,
'labels': labels,
'pred': pred,
'prot_attribute': prot_attribute,
'threshold_indices': threshold_indices,
'helper_table_T': helper_table_T,
'helper_table_P': helper_table_P,
'el_per_group': el_per_group,
'cost': cost,
'cost_matrix': cost_matrix,
'lambd': lambd,
'scores': scores
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| find_best_right_shift |
Ran ran; our contract draft was wrong, not the code |
amazon-research/fair-ordinal-regression/learn_fair_threshold_model/local_search_equal_opportunity.py code served (permissive licence) · get_code("3c276f90716a64a3") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Set random seed for reproducibility
np.random.seed(0)
# Define parameters based on function usage
n = 10 # total number of elements
n_thresh = 3 # number of thresholds
nr_groups = 2 # number of protected attribute groups
# labels: array of class labels for each element
labels = np.array([0, 1, 0, 1, 0, 1, 0, 1, 0, 1])
# pred: predictions (used indirectly through other parameters)
pred = np.array([0.1, 0.9, 0.2, 0.8, 0.3, 0.7, 0.4, 0.6, 0.5, 0.55])
# prot_attribute: protected attribute group for each element (0 or 1)
prot_attribute = np.array([0, 1, 0, 1, 0, 1, 0, 1, 0, 1])
# threshold_indices: indices where thresholds are applied
threshold_indices = np.array([2, 5, 8])
# helper_table_T: 3D table tracking label/threshold/protected_attribute combinations
# Shape: (n_classes=2, n_thresholds+1=4, nr_groups=2)
helper_table_T = np.random.randint(0, 5, size=(2, 4, 2))
# helper_table_P: 2D table for protected attribute comparisons
# Shape: (nr_groups=2, nr_groups=2)
helper_table_P = np.random.randint(0, 10, size=(2, 2))
# el_per_group_that_are_bigger: elements per group that are bigger than threshold
# Shape: (nr_groups=2,)
el_per_group_that_are_bigger = np.array([5, 5], dtype=float)
# cost: current misclassification cost
cost = 10.0
# cost_matrix: cost matrix for labels and thresholds
# Shape: (n_classes=2, n_thresholds+1=4)
cost_matrix = np.array([
[1.0, 2.0, 3.0, 4.0],
[2.0, 1.0, 2.0, 3.0]
])
# lambd: fairness penalty parameter
lambd = 0.5
# scores: prediction scores for each element
scores = np.array([0.1, 0.9, 0.2, 0.8, 0.3, 0.7, 0.4, 0.6, 0.5, 0.55])
ARGS = {
'n': n,
'n_thresh': n_thresh,
'nr_groups': nr_groups,
'labels': labels,
'pred': pred,
'prot_attribute': prot_attribute,
'threshold_indices': threshold_indices,
'helper_table_T': helper_table_T,
'helper_table_P': helper_table_P,
'el_per_group_that_are_bigger': el_per_group_that_are_bigger,
'cost': cost,
'cost_matrix': cost_matrix,
'lambd': lambd,
'scores': scores
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| get_best_constant_predictor_for_MAE |
Ran ran, honoured the contract we drafted |
amazon-research/fair-ordinal-regression/plot_utils.py code served (permissive licence) · get_code("bac89d70dd9a10fe") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int64, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Create sample training labels - the function computes median and rounds
train_labels = np.array([1.2, 2.5, 3.7, 4.1, 5.3, 6.8, 7.2, 8.9])
ARGS = {
"train_labels": train_labels
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type int64, ndim 0, dtype family int, shape () — shape () as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| make_predictions_with_model |
Ran ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT) |
amazon-research/fair-ordinal-regression/learn_fair_threshold_model/learn_fair_model.py code served (permissive licence) · get_code("5f8b1816e0edbfab") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family int, shape (2,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import numpy as np
# Set seed for reproducibility
np.random.seed(0)
# Create X_test: shape (2, 4) - 2 samples, 4 features
X_test = np.random.randn(2, 4).astype(np.float32)
# Create w: shape (4,) - weight vector matching feature dimension
w = np.random.randn(4).astype(np.float32)
# Create thresholds: sorted array of threshold values for searchsorted
# searchsorted requires sorted thresholds
thresholds = np.array([-1.0, 0.0, 1.0, 2.0], dtype=np.float32)
ARGS = {
"X_test": X_test,
"w": w,
"thresholds": thresholds,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type ndarray, ndim 1, dtype family int, shape (2,) — shape (2,) as a later run recorded it
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|