| build_GAP_output_format |
Ran ran on a synthesized input |
salesforce/UniversalFewShotNLP/Coreference/full.shot.train.on.target.data.py code served (permissive licence) · get_code("f6662f438f626dbd") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 2, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
torch.manual_seed(0)
# Analyzing the function:
# - example_id_list: list of example IDs (strings or ints)
# - gold_label_ids: list of label indices (0 or 1 based on usage with type indexing)
# - pred_prob_entail: list of probabilities (floats)
# - pred_label_ids_3way: list of labels (used but not directly in logic, seems unused)
# - threshold: float for comparison
# - dev_or_test: string, either 'dev' or 'test'
# The function:
# 1. Zips together 4 lists of same length
# 2. Uses example_id to index dictionaries
# 3. Uses 'type' (from gold_label_ids) to index into labellist[type] and scorelist[type]
# This suggests type should be 0 or 1
# 4. Compares prob with threshold
# 5. Creates output with prefix + str(ex_id)
# Building example data:
example_id_list = [1, 1, 2, 2]
gold_label_ids = [0, 1, 0, 1]
pred_prob_entail = [0.8, 0.3, 0.2, 0.9]
pred_label_ids_3way = [1, 0, 0, 1] # Not used in the function logic
threshold = 0.5
dev_or_test = 'dev'
ARGS = {
'example_id_list': example_id_list,
'gold_label_ids': gold_label_ids,
'pred_prob_entail': pred_prob_entail,
'pred_label_ids_3way': pred_label_ids_3way,
'threshold': threshold,
'dev_or_test': dev_or_test
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 2, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| calculate_scores |
Ran ran on a synthesized input |
salesforce/UniversalFewShotNLP/Coreference/gap_scorer.py code served (permissive licence) · get_code("84120ed8a77f6efa") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
class Annotation:
def __init__(self, name_a_coref, name_b_coref, gender):
self.name_a_coref = name_a_coref
self.name_b_coref = name_b_coref
self.gender = gender
gold_annotations = {
'ex1': Annotation(name_a_coref=True, name_b_coref=False, gender='M'),
'ex2': Annotation(name_a_coref=False, name_b_coref=True, gender='F'),
'ex3': Annotation(name_a_coref=True, name_b_coref=True, gender='M'),
}
system_annotations = {
'ex1': Annotation(name_a_coref=True, name_b_coref=False, gender='M'),
'ex2': Annotation(name_a_coref=True, name_b_coref=True, gender='F'),
'ex3': Annotation(name_a_coref=False, name_b_coref=True, gender='M'),
}
ARGS = {
'gold_annotations': gold_annotations,
'system_annotations': system_annotations,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| calculate_scores |
Ran ran on a synthesized input |
salesforce/UniversalFewShotNLP/Non-Entail-Approach/gap_scorer_modified.py code served (permissive licence) · get_code("b2f6b157b471666d") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
class Annotation:
"""Stand-in class for Annotation objects."""
def __init__(self, name_a_coref, name_b_coref, gender):
self.name_a_coref = name_a_coref
self.name_b_coref = name_b_coref
self.gender = gender
# Create gold annotations dict
gold_annotations = {
"ex_1": Annotation(name_a_coref=True, name_b_coref=False, gender="M"),
"ex_2": Annotation(name_a_coref=False, name_b_coref=True, gender="F"),
"ex_3": Annotation(name_a_coref=True, name_b_coref=True, gender="M"),
}
# Create system annotations dict (matching keys)
system_annotations = {
"ex_1": Annotation(name_a_coref=True, name_b_coref=False, gender="M"),
"ex_2": Annotation(name_a_coref=False, name_b_coref=False, gender="F"),
"ex_3": Annotation(name_a_coref=True, name_b_coref=None, gender="M"),
}
ARGS = {
"gold_annotations": gold_annotations,
"system_annotations": system_annotations,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| convert_examples_to_features |
Ran ran on a synthesized input |
salesforce/UniversalFewShotNLP/Coreference/0.shot.py code served (permissive licence) · get_code("65e3e28505f962b0") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import logging
logging.basicConfig(level=logging.INFO)
class MockTokenizer:
"""Minimal tokenizer for testing."""
def tokenize(self, text):
return text.split()
def convert_tokens_to_ids(self, tokens):
return [hash(token) % 1000 for token in tokens]
class MockExample:
"""Minimal example object."""
def __init__(self, guid, text_a, text_b, label):
self.guid = guid
self.text_a = text_a
self.text_b = text_b
self.label = label
examples = [
MockExample("ex0", "hello world", "foo bar", "positive"),
MockExample("ex1", "goodbye world", None, "negative"),
]
label_list = ["positive", "negative"]
tokenizer = MockTokenizer()
ARGS = {
"examples": examples,
"label_list": label_list,
"max_seq_length": 16,
"tokenizer": tokenizer,
"output_mode": "classification",
"cls_token_at_end": False,
"cls_token": "[CLS]",
"cls_token_segment_id": 0,
"sep_token": "[SEP]",
"sep_token_extra": False,
"pad_on_left": False,
"pad_token": 0,
"pad_token_segment_id": 0,
"sequence_a_segment_id": 0,
"sequence_b_segment_id": 1,
"mask_padding_with_zero": True,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| convert_examples_to_features |
Ran ran on a synthesized input |
salesforce/UniversalFewShotNLP/Coreference/full.shot.train.on.target.data.py code served (permissive licence) · get_code("30a9889ed4a6e986") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import logging
logger = logging.getLogger(__name__)
class Example:
"""A single training/test example for simple sequence classification."""
def __init__(self, guid, text_a, text_b, label, entity_label):
self.guid = guid
self.text_a = text_a
self.text_b = text_b
self.label = label
self.entity_label = entity_label
class SimpleTokenizer:
"""A minimal tokenizer that simulates basic tokenization."""
def __init__(self):
self.vocab = {
'[CLS]': 101,
'[SEP]': 102,
'[PAD]': 0,
'hello': 1,
'world': 2,
'this': 3,
'is': 4,
'test': 5,
'text': 6,
}
def tokenize(self, text):
"""Simple whitespace tokenizer."""
if not text:
return []
return text.lower().split()
def convert_tokens_to_ids(self, tokens):
"""Convert tokens to ids using vocab."""
return [self.vocab.get(token, 0) for token in tokens]
examples = [
Example(
guid='test_0',
text_a='hello world',
text_b='this is test',
label='positive',
entity_label='person'
),
Example(
guid='test_1',
text_a='test text',
text_b=None,
label='negative',
entity_label='location'
),
]
label_list = ['positive', 'negative', 'neutral']
entity_label_list = ['person', 'location', 'organization', 'other']
max_seq_length = 32
tokenizer = SimpleTokenizer()
output_mode = 'classification'
ARGS = {
'examples': examples,
'label_list': label_list,
'entity_label_list': entity_label_list,
'max_seq_length': max_seq_length,
'tokenizer': tokenizer,
'output_mode': output_mode,
'cls_token_at_end': False,
'cls_token': '[CLS]',
'cls_token_segment_id': 0,
'sep_token': '[SEP]',
'sep_token_extra': False,
'pad_on_left': False,
'pad_token': 0,
'pad_token_segment_id': 0,
'sequence_a_segment_id': 0,
'sequence_b_segment_id': 1,
'mask_padding_with_zero': True,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
- dtype family
- int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| examples_to_features |
Ran ran on a synthesized input |
salesforce/UniversalFewShotNLP/Coreference/full.shot.train.on.target.data.py code served (permissive licence) · get_code("6408f45358c253ca") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataLoader. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import torch.nn as nn
from transformers import AutoTokenizer
torch.manual_seed(0)
class Example:
def __init__(self, guid, text_a, text_b, label, entity_label):
self.guid = guid
self.text_a = text_a
self.text_b = text_b
self.label = label
self.entity_label = entity_label
class SimpleTokenizer:
def __init__(self):
self.cls_token = '[CLS]'
self.sep_token = '[SEP]'
self.pad_token = '[PAD]'
self.vocab = {
'[CLS]': 1,
'[SEP]': 2,
'[PAD]': 0,
'hello': 3,
'world': 4,
'test': 5,
'example': 6,
}
def tokenize(self, text):
return text.lower().split()
def convert_tokens_to_ids(self, tokens):
return [self.vocab.get(token, 0) for token in tokens]
class ArgsConfig:
def __init__(self):
self.max_seq_length = 16
source_examples = [
Example(guid=0, text_a='hello world', text_b='test example', label='positive', entity_label='entity_A'),
Example(guid=1, text_a='test', text_b=None, label='negative', entity_label='entity_B'),
]
label_list = ['positive', 'negative']
entity_label_list = ['entity_A', 'entity_B']
args = ArgsConfig()
tokenizer = SimpleTokenizer()
batch_size = 2
output_mode = 'classification'
ARGS = {
'source_examples': source_examples,
'label_list': label_list,
'entity_label_list': entity_label_list,
'args': args,
'tokenizer': tokenizer,
'batch_size': batch_size,
'output_mode': output_mode,
'dataloader_mode': 'sequential',
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- torch, transformers — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type DataLoader — a DataLoader output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataLoader output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| read_annotations_test |
Ran ran on a synthesized input |
salesforce/UniversalFewShotNLP/Coreference/gap_scorer_modified.py code served (permissive licence) · get_code("eb1bd7c8cdf42b2e") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned defaultdict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
from collections import defaultdict
class Annotation(object):
"""Container class for storing annotations of an example.
Attributes:
gender(None): The gender of the annotation. None indicates that gender was
not determined for the given example.
name_a_coref(None): bool reflecting whether Name A was recorded as
coreferential with the target pronoun for this example. None indicates
that no annotation was found for the given example.
name_b_coref(None): bool reflecting whether Name B was recorded as
coreferential with the target pronoun for this example. None indicates
that no annotation was found for the given example.
"""
def __init__(self):
self.gender = None
self.name_a_coref = None
self.name_b_coref = None
def read_annotations_test(input_list):
annotations = defaultdict(Annotation)
for tuple in input_list:
example_id = str(tuple[0])
A_coref = tuple[1]
B_coref = tuple[2]
annotations[example_id].name_a_coref = A_coref
annotations[example_id].name_b_coref = B_coref
return annotations
ARGS = {
"input_list": [
(1, True, False),
(2, False, True),
(3, True, True),
]
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type defaultdict — a defaultdict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a defaultdict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| wordpairID_2_tokenpairID |
Ran ran on a synthesized input |
salesforce/UniversalFewShotNLP/Non-Entail-Approach/train.coreference.py code served (permissive licence) · get_code("5df41f5886b2a575") How it ranRan in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.
import torch
import numpy as np
class MockTokenizer:
def tokenize(self, text):
# Simple tokenization by splitting on spaces and adding ## for continuations
tokens = text.split()
return tokens
def convert_tokens_to_ids(self, tokens):
# Simple mapping: each unique token gets an id
# Using a consistent mapping for testing
token_to_id = {
'hello': 100,
'world': 101,
'this': 102,
'is': 103,
'a': 104,
'test': 105,
'sentence': 106,
}
return [token_to_id.get(t, 107) for t in tokens]
# Create a mock tokenizer
tokenizer = MockTokenizer()
# Create test data
sentence = "hello world this is a test"
wordindex_left = 0
wordindex_right = 1
# Build full_token_id_list with [2, 2] marker and matching token ids
# full_token_id_list should contain the tokenized version of the sentence
# with [2, 2] as a separator marker
full_token_id_list = [100, 101, 2, 2, 102, 103, 104, 105]
# Alternative setup where we can find the span
# "hello world" should map to token ids [100, 101]
full_token_id_list = [100, 101, 2, 2, 102, 103, 104, 105]
ARGS = {
'sentence': "hello world this is a test",
'wordindex_left': 0,
'wordindex_right': 1,
'full_token_id_list': [100, 101, 2, 2, 102, 103, 104, 105],
'tokenizer': tokenizer,
'sent_1': True,
}
- Setup block drafted by
- a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
- Imports
- numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
- Returned
- type tuple — a tuple output has no array shape of its own; none is stated
- Environment
- image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
|
| load_GAP_coreference_data |
Not yet run unverified (harvested, no recorded run) |
salesforce/UniversalFewShotNLP/Coreference/preprocess.py code served (permissive licence) · get_code("f706083dc1ace8a0") How it ran |