SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2301.07597 · 2023

How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection

arXiv · PDF · Open in the Atlas

Code that ran

21 of 25 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 4 honoured the contract we drafted, 0 violated it and 17 have no contract verdict; 0 failures of our instrument; 4 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2301.07597.json · badge: /api/ran/2301.07597.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
lsc-1/pecola community (classifier label)
classifier basis not recorded on the link
21 ran · 4 no recorded run
FunctionStatusWhere it lives
auroc Ran
ran on a synthesized input
lsc-1/pecola/lib/metrics.py
code served (permissive licence) · get_code("a6359426276fb40b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float64, ndim 0, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Set random seed for reproducibility
np.random.seed(42)

# Create small sample scores for IND (in-distribution) and OOD (out-of-distribution) data
ind_scores = np.array([0.8, 0.9, 0.85, 0.95])  # 4 in-distribution scores
ood_scores = np.array([0.2, 0.3, 0.15, 0.25])  # 4 out-of-distribution scores

# Define ARGS dict with all parameters for auroc function
ARGS = {
    'ind_scores': ind_scores,
    'ood_scores': ood_scores,
    'order': 'largest2smallest'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float64, ndim 0, dtype family float — no output shape is stated: observe_signature.py recorded float ndim 0, which disagrees with the fixture run
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_base_score Ran
ran, honoured the contract we drafted
lsc-1/pecola/lib/inference/base.py
code served (permissive licence) · get_code("61f9329047b0ab17")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import numpy as np

torch.manual_seed(0)

class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 3)
    
    def forward(self, input_ids, attention_mask=None):
        # Return an object with logits attribute
        logits = self.linear(input_ids)
        
        class Output:
            pass
        
        output = Output()
        output.logits = logits
        return output

model = SimpleModel()

# Create a simple data loader as a list of tuples
# Each tuple is (input_ids, labels, attention_masks)
data_loader = [
    (torch.randn(2, 4), torch.randint(0, 3, (2,)), torch.ones(2, 4)),
    (torch.randn(2, 4), torch.randint(0, 3, (2,)), torch.ones(2, 4)),
]

ARGS = {
    'model': model,
    'data_loader': data_loader,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_cse_features Ran
ran on a synthesized input
lsc-1/pecola/lib/inference/draw.py
code served (permissive licence) · get_code("0256d90810fbc438")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import numpy as np

torch.manual_seed(0)

class MockModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 8)
    
    def forward(self, input_ids, attention_mask=None, return_dict=False, output_hidden_states=False):
        # Simulate a model that returns pooler_output
        batch_size = input_ids.shape[0]
        pooler_output = torch.randn(batch_size, 8)
        
        class Output:
            pass
        
        output = Output()
        output.pooler_output = pooler_output
        return output
    
    def eval(self):
        super().eval()
        return self

# Create mock data loader with 2 batches
# Each item: (input_ids, batch_labels, attention_masks)
data_loader = [
    (
        torch.randint(0, 100, (2, 4)),  # input_ids: batch_size=2, seq_len=4
        torch.tensor([0, 1]),            # batch_labels
        torch.ones((2, 4))               # attention_masks
    ),
    (
        torch.randint(0, 100, (2, 4)),  # input_ids: batch_size=2, seq_len=4
        torch.tensor([1, 0]),            # batch_labels
        torch.ones((2, 4))               # attention_masks
    )
]

model = MockModel()

ARGS = {
    'model': model,
    'data_loader': data_loader
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_d2u_score Ran
ran on a synthesized input
lsc-1/pecola/lib/inference/base.py
code served (permissive licence) · get_code("13081870f2ef7a7d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import numpy as np

torch.manual_seed(0)

# Create a minimal model that satisfies the function's usage
class DummyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 3)  # 4 input features, 3 output classes
    
    def forward(self, input_ids, attention_mask=None):
        # Simple forward pass that returns logits
        # input_ids: (batch, seq_len) -> flatten or pool to (batch, features)
        batch_size = input_ids.shape[0]
        # Return an object with .logits attribute
        logits = self.linear(input_ids.float())
        
        class Output:
            pass
        output = Output()
        output.logits = logits
        return output

# Create model
model = DummyModel()

# Create minimal data loader
# Each item should be (input_ids, labels, attention_masks)
data_loader = [
    (
        torch.randint(0, 10, (2, 4)),  # input_ids: batch_size=2, seq_len=4
        torch.randint(0, 3, (2,)),      # labels: batch_size=2, 3 classes
        torch.ones((2, 4))              # attention_masks: batch_size=2, seq_len=4
    ),
    (
        torch.randint(0, 10, (2, 4)),
        torch.randint(0, 3, (2,)),
        torch.ones((2, 4))
    )
]

ARGS = {
    "model": model,
    "data_loader": data_loader
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_dropout_score Ran
ran, honoured the contract we drafted
lsc-1/pecola/lib/inference/dropout.py
code served (permissive licence) · get_code("29588baf96ee353d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import numpy as np

torch.manual_seed(0)

# Build a minimal model that satisfies the function's usage
class MinimalModel(nn.Module):
    def __init__(self, num_classes=3):
        super().__init__()
        self.linear = nn.Linear(4, num_classes)
    
    def forward(self, input_ids, attention_mask=None):
        # The function calls model(input_ids, attention_mask=attention_masks)
        # and expects outputs with a .logits attribute or raw logits
        logits = self.linear(input_ids)
        # Return an object with .logits attribute
        class Output:
            def __init__(self, logits):
                self.logits = logits
        return Output(logits)

# Create a minimal data loader
# The function expects tuples of (input_ids, labels, attention_masks)
def create_data_loader(batch_size=2, num_batches=2, input_dim=4, num_classes=3):
    data_loader = []
    for _ in range(num_batches):
        input_ids = torch.randn(batch_size, input_dim)
        labels = torch.randint(0, num_classes, (batch_size,))
        attention_masks = torch.ones(batch_size, input_dim)
        data_loader.append((input_ids, labels, attention_masks))
    return data_loader

# Construct the model and data loader
model = MinimalModel(num_classes=3)
data_loader = create_data_loader(batch_size=2, num_batches=2, input_dim=4, num_classes=3)

ARGS = {
    'model': model,
    'data_loader': data_loader,
    'passes': 5
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_features Ran
ran on a synthesized input
lsc-1/pecola/lib/inference/draw.py
code served (permissive licence) · get_code("dd94c31be0cd6720")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import numpy as np

torch.manual_seed(0)

# Build a minimal model that mimics RoBERTa's interface
class MinimalRoBERTa(nn.Module):
    def __init__(self):
        super().__init__()
        # RoBERTa returns 13 hidden states (layer 0 through 12)
        self.hidden_dim = 4
        self.embeddings = nn.Embedding(100, self.hidden_dim)
        self.layers = nn.ModuleList([nn.Linear(self.hidden_dim, self.hidden_dim) for _ in range(12)])
    
    def forward(self, input_ids, attention_mask, return_dict=True, output_hidden_states=True):
        # Embed input
        hidden_state = self.embeddings(input_ids)  # (batch, seq_len, hidden_dim)
        
        # Generate hidden states through layers
        hidden_states = [hidden_state]
        for layer in self.layers:
            hidden_state = layer(hidden_state)
            hidden_states.append(hidden_state)
        
        # Return as dict with hidden_states
        class Output:
            pass
        
        output = Output()
        output.hidden_states = hidden_states  # List of 13 tensors
        return output

class MinimalRoBERTaModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.roberta = MinimalRoBERTa()

# Create tiny batches for data loaders
# Each batch: (input_ids, batch_labels, attention_masks)
batch_size = 2
seq_len = 3

# First data loader (idx=0) with real labels
data_loader_1 = [
    (torch.randint(0, 100, (batch_size, seq_len)),
     torch.tensor([0, 1]),
     torch.ones((batch_size, seq_len))),
    (torch.randint(0, 100, (batch_size, seq_len)),
     torch.tensor([1, 0]),
     torch.ones((batch_size, seq_len)))
]

# Second data loader (idx=1) with dummy labels (will be replaced with -1)
data_loader_2 = [
    (torch.randint(0, 100, (batch_size, seq_len)),
     torch.tensor([0, 0]),
     torch.ones((batch_size, seq_len))),
]

ARGS = {
    'model': MinimalRoBERTaModel(),
    'data_loaders': [data_loader_1, data_loader_2],
    'pooler_pos': 0
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_grad_norm_score Ran
ran on a synthesized input
lsc-1/pecola/lib/inference/base.py
code served (permissive licence) · get_code("e614043206c6e7fd")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (2,) as a later run recorded it. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import numpy as np

torch.manual_seed(0)

# Create a minimal model that satisfies the function's requirements
class MinimalModel(nn.Module):
    def __init__(self, num_classes=3):
        super().__init__()
        self.embedding = nn.Linear(4, 8)
        self.classifier = nn.Module()
        self.classifier.out_proj = nn.Linear(8, num_classes)
    
    def forward(self, input_ids, attention_mask=None):
        x = self.embedding(input_ids)
        logits = self.classifier.out_proj(x)
        return logits

# Create a minimal data loader (list of batches)
# Each batch is (input_ids, labels, attention_masks)
batch_size = 2
seq_len = 4
num_classes = 3

batch1 = (
    torch.randn(batch_size, seq_len),  # input_ids
    torch.randint(0, num_classes, (batch_size,)),  # labels
    torch.ones(batch_size, seq_len),  # attention_masks
)

batch2 = (
    torch.randn(batch_size, seq_len),  # input_ids
    torch.randint(0, num_classes, (batch_size,)),  # labels
    torch.ones(batch_size, seq_len),  # attention_masks
)

data_loader = [batch1, batch2]

# Create model instance
model = MinimalModel(num_classes=num_classes)

ARGS = {
    "model": model,
    "data_loader": data_loader,
    "num_classes": num_classes,
    "temperature": 1,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (2,) — shape (2,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_is_pos Ran
ran on a synthesized input
lsc-1/pecola/lib/metrics.py
code served (permissive licence) · get_code("7b9990d7d4520898")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (5,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Set random seed for reproducibility
np.random.seed(42)

# Create small sample arrays for ind_scores and ood_scores
ind_scores = np.array([0.9, 0.8, 0.7], dtype=np.float32)
ood_scores = np.array([0.3, 0.2], dtype=np.float32)

# Valid order parameter
order = 'largest2smallest'

ARGS = {
    'ind_scores': ind_scores,
    'ood_scores': ood_scores,
    'order': order
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (5,) — shape (5,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_key_positions Ran
ran on a synthesized input
lsc-1/pecola/lib/inference/generation.py
code served (permissive licence) · get_code("9aab08fc71951fbb")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 2, dtype family int, shape (2, 2) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import numpy as np

torch.manual_seed(0)

# Build a minimal BERT-like model that has the expected structure
class MinimalBertModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.bert = self.BertEncoder()
    
    class BertEncoder(nn.Module):
        def __init__(self):
            super().__init__()
            self.attention_layer = nn.Linear(4, 4)
        
        def forward(self, input_ids, attention_mask, return_dict=True, output_attentions=True):
            # Simulate BERT output with attention heads
            batch_size = input_ids.size(0)
            seq_len = input_ids.size(1)
            num_heads = 2
            
            # Create dummy attention tensor: (batch_size, num_heads, seq_len, seq_len)
            attentions = torch.randn(batch_size, num_heads, seq_len, seq_len)
            
            # Return a dict-like object with attentions
            class Output:
                pass
            
            output = Output()
            output.attentions = [attentions]  # List of attention tensors
            return output

class MinimalTokenizer:
    def __init__(self):
        self.pad_token_id = 0

# Create model and tokenizer
model = MinimalBertModel()
tokenizer = MinimalTokenizer()

# Create input tensors
batch_size = 2
seq_len = 5
input_ids = torch.tensor([
    [1, 2, 3, 0, 0],
    [4, 5, 0, 0, 0]
], dtype=torch.long)

attention_masks = torch.ones(batch_size, seq_len, dtype=torch.long)

topn = 2

ARGS = {
    "model": model,
    "tokenizer": tokenizer,
    "input_ids": input_ids,
    "attention_masks": attention_masks,
    "topn": topn
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 2, dtype family int, shape (2, 2) — shape (2, 2) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_model1 Ran
ran on a synthesized input
lsc-1/pecola/lib/networks.py
code served (permissive licence) · get_code("b78ad0b30fc6a1da")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned MockModel. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
from unittest.mock import MagicMock

torch.manual_seed(0)

# Create a minimal mock config class
class MockConfig:
    def __init__(self):
        self.hidden_size = 768
        self.num_hidden_layers = 2
        self.num_attention_heads = 8
        self.intermediate_size = 3072
        self.hidden_act = "gelu"
        self.hidden_dropout_prob = 0.1
        self.attention_probs_dropout_prob = 0.1
        self.max_position_embeddings = 512
        self.type_vocab_size = 2
        self.initializer_range = 0.02
        self.layer_norm_eps = 1e-12
        self.pad_token_id = 0
        self.vocab_size = 1000

# Create a minimal mock model class
class MockModel(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.embedding = nn.Embedding(config.vocab_size, config.hidden_size)
        self.encoder = nn.Linear(config.hidden_size, config.hidden_size)
    
    def forward(self, input_ids, attention_mask=None):
        embeddings = self.embedding(input_ids)
        return embeddings

# Create a minimal mock args class
class MockArgs:
    def __init__(self):
        self.model = "mock_model_name"

# Mock AutoConfig.from_pretrained
original_auto_config = __import__('transformers').AutoConfig.from_pretrained
def mock_config_from_pretrained(model_name, trust_remote_code=True):
    return MockConfig()

# Mock AutoModel.from_pretrained
original_auto_model = __import__('transformers').AutoModel.from_pretrained
def mock_model_from_pretrained(model_name, config=None, trust_remote_code=True):
    if config is None:
        config = MockConfig()
    return MockModel(config)

# Patch the functions
import transformers
transformers.AutoConfig.from_pretrained = mock_config_from_pretrained
transformers.AutoModel.from_pretrained = mock_model_from_pretrained

ARGS = {
    "args": MockArgs()
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch, transformers — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type MockModel — a MockModel output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a MockModel output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_msp_score Ran
ran, honoured the contract we drafted
lsc-1/pecola/lib/inference/dropout.py
code served (permissive licence) · get_code("a52d92262c816283")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import numpy as np

torch.manual_seed(0)

# Define a minimal model that returns logits
class MinimalModel(nn.Module):
    def __init__(self, num_classes=3):
        super().__init__()
        self.linear = nn.Linear(4, num_classes)
    
    def forward(self, input_ids, attention_mask=None):
        # Return an object with logits attribute
        logits = self.linear(input_ids)
        
        class Output:
            pass
        
        output = Output()
        output.logits = logits
        return output

# Create a tiny data loader (list of tuples)
# Each tuple: (input_ids, labels, attention_masks)
data_loader = [
    (torch.randn(2, 4), torch.tensor([0, 1]), torch.ones(2, 4)),
    (torch.randn(2, 4), torch.tensor([2, 0]), torch.ones(2, 4)),
]

model = MinimalModel(num_classes=3)

ARGS = {
    "model": model,
    "data_loader": data_loader,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_num_labels Ran
ran, honoured the contract we drafted
lsc-1/pecola/lib/exp.py
code served (permissive licence) · get_code("41058bada6b6f29d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn

torch.manual_seed(0)

# Analyzing the function:
# - get_num_labels takes one parameter: 'dataset' (a string)
# - It checks if 'clinc' is in the dataset string
# - It removes '_fewshot' from the dataset string
# - It looks up the resulting string in NUM_LABELS_MAP
# - The function only uses string operations on the dataset parameter

# Test with a dataset that exists in NUM_LABELS_MAP
ARGS = {
    'dataset': 'hc3'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_ppl_confidence Ran
ran on a synthesized input
lsc-1/pecola/lib/inference/generation.py
code served (permissive licence) · get_code("f70aa83668a8706c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import numpy as np
from math import log

torch.manual_seed(0)

# Create a minimal tokenizer with required attributes
class MinimalTokenizer:
    def __init__(self):
        self.mask_token_id = 103
        self.cls_token_id = 101

# Create a minimal model that implements the required interface
class MinimalModel(nn.Module):
    def __init__(self, vocab_size=1000, hidden_dim=4):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, hidden_dim)
        self.lm_head = nn.Linear(hidden_dim, vocab_size)
    
    def forward(self, input_ids, attention_mask, mode='lm'):
        # input_ids shape: (B, K)
        embedded = self.embedding(input_ids)  # (B, K, hidden_dim)
        logits = self.lm_head(embedded)  # (B, K, vocab_size)
        return logits

# Create model and tokenizer
model = MinimalModel(vocab_size=1000, hidden_dim=4)
tokenizer = MinimalTokenizer()

# Create a minimal data loader
# Each item is (input_ids, labels, attention_masks)
batch_size = 2
seq_length = 3
vocab_size = 1000

input_ids = torch.randint(104, vocab_size, (batch_size, seq_length))
labels = torch.randint(104, vocab_size, (batch_size, seq_length))
attention_masks = torch.ones((batch_size, seq_length))

# Create data loader as a list of batches
data_loader = [
    (input_ids, labels, attention_masks),
    (input_ids, labels, attention_masks)
]

ARGS = {
    'model': model,
    'tokenizer': tokenizer,
    'data_loader': data_loader
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
marginLoss Ran
ran on a synthesized input
lsc-1/pecola/lib/common.py
code served (permissive licence) · get_code("6c82794bd0139e76")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

# Create pooled embeddings: shape (batch_size, embedding_dim)
# Using batch_size=2, embedding_dim=4
pooled = torch.randn(2, 4)

# Create labels: shape (batch_size,)
# Using 2 samples with 2 different classes
labels = torch.tensor([0, 1])

ARGS = {
    "pooled": pooled,
    "labels": labels,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
roc Ran
ran on a synthesized input
lsc-1/pecola/lib/metrics.py
code served (permissive licence) · get_code("518d9f2cda25ff12")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 2, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# Set random seed for reproducibility
np.random.seed(42)

# Create small sample score arrays for IND (in-distribution) and OOD (out-of-distribution)
ind_scores = np.array([0.8, 0.9, 0.7, 0.85], dtype=np.float32)
ood_scores = np.array([0.3, 0.4, 0.2, 0.35], dtype=np.float32)

# Valid order parameter
order = 'largest2smallest'

ARGS = {
    'ind_scores': ind_scores,
    'ood_scores': ood_scores,
    'order': order
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 2, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
sclLoss Ran
ran on a synthesized input
lsc-1/pecola/lib/common.py
code served (permissive licence) · get_code("e97eacfdcbaf68a1")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Tensor, ndim 0, dtype family float, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn.functional as F

torch.manual_seed(0)

# pooled: a normalized embedding tensor, shape (batch_size, embedding_dim)
# Should be 2D tensor where first dim is batch size, second is embedding dimension
pooled = torch.randn(2, 4)

# labels: a tensor of class labels for each sample in the batch
# Shape (batch_size,) with integer class indices
labels = torch.tensor([0, 1])

ARGS = {
    "pooled": pooled,
    "labels": labels,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Tensor, ndim 0, dtype family float, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
set_device_and_seed Ran
ran on a synthesized input
lsc-1/pecola/select_strategy.py
code served (permissive licence) · get_code("96ab70351169ff04")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned device. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import random
import numpy as np
import os

torch.manual_seed(0)

ARGS = {
    'random_seed': 42
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type device — a device output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a device output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
set_device_and_seed Ran
ran on a synthesized input
lsc-1/pecola/augumentation_data/yake_mask.py
code served (permissive licence) · get_code("6a16ec67a0221341")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned device. The image it ran in was not recorded. The code mentions cuda, and the harness aliases cuda to the CPU for every sample; whether this run needed that is not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import random
import torch
import numpy as np
import os

torch.manual_seed(0)

ARGS = {
    'random_seed': 42
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type device — a device output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a device output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
small_fix Ran
ran on a synthesized input
lsc-1/pecola/augumentation_data/mask_filling.py
code served (permissive licence) · get_code("be4d25103faa09a4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

text = "Hello , world ! This is a test ? with various punctuation , marks ;"

ARGS = {
    "text": text
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
take_mean_and_reshape Ran
ran on a synthesized input
lsc-1/pecola/lib/exp.py
code served (permissive licence) · get_code("85474e5c0d2f6525")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned ndarray, ndim 1, dtype family float, shape (4,) as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np

# The function takes a single parameter 'x' which is passed to np.array()
# and then has .mean(axis=0) called on it, followed by .reshape(-1)
# So x should be array-like with at least 2 dimensions

# Create a small 2D array (batch_size=2, features=4)
x = np.array([
    [1.0, 2.0, 3.0, 4.0],
    [5.0, 6.0, 7.0, 8.0]
])

ARGS = {
    "x": x
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type ndarray, ndim 1, dtype family float, shape (4,) — shape (4,) as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
test_acc Ran
ran on a synthesized input
lsc-1/pecola/lib/common.py
code served (permissive licence) · get_code("50ec38fc4fd612e1")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import torch.nn as nn
import numpy as np

torch.manual_seed(0)

# Create a minimal model that satisfies the function's usage
class MinimalModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(4, 2)  # 4 input features, 2 classes
    
    def forward(self, input_ids, attention_mask=None):
        # Return object with .logits attribute
        logits = self.linear(input_ids)
        
        class Output:
            pass
        
        output = Output()
        output.logits = logits
        return output

# Create a minimal model instance
model = MinimalModel()

# Create a minimal data loader - list of tuples (input_ids, labels, attention_masks)
# Each batch should have: input_ids (batch_size, feature_dim), labels (batch_size,), attention_masks (batch_size, feature_dim)
data_loader = [
    (
        torch.randn(2, 4),  # input_ids: batch_size=2, feature_dim=4
        torch.tensor([0, 1], dtype=torch.long),  # labels: batch_size=2
        torch.ones(2, 4)  # attention_masks: batch_size=2, feature_dim=4
    ),
    (
        torch.randn(2, 4),  # second batch
        torch.tensor([1, 0], dtype=torch.long),
        torch.ones(2, 4)
    )
]

ARGS = {
    'model': model,
    'data_loader': data_loader,
    'metric': 'acc',
    'return_proba': False
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, sklearn, torch, tqdm — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple, ndim 1, dtype family float — a tuple output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_FFN_features Not yet run
unverified (harvested, no recorded run)
lsc-1/pecola/lib/inference/draw.py
code served (permissive licence) · get_code("d1c6ff9d27f3294b")
How it ran

no invocation recorded.

get_keyword_confidence Not yet run
unverified (harvested, no recorded run)
lsc-1/pecola/lib/inference/generation.py
code served (permissive licence) · get_code("24c1d85c8d245fbb")
How it ran

no invocation recorded.

get_model Not yet run
unverified (harvested, no recorded run)
lsc-1/pecola/lib/networks.py
code served (permissive licence) · get_code("c5c3b0d2f5c9931c")
How it ran

no invocation recorded.

get_tokenizer Not yet run
unverified (harvested, no recorded run)
lsc-1/pecola/lib/networks.py
code served (permissive licence) · get_code("73808e9a70c072ec")
How it ran

no invocation recorded.

Also present in other papers' code

Of 25 samples lifted from this paper's linked repositories, 25 carry a function identity and 0 do not. Of the 24 distinct functions with an identity, 15 are also present in other papers whose code we hold (15 with at least one paper dated earlier, 0 the same month, 0 later) and 9 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
auroc 2 other papers
2 dated earlier
2022.findings-emnlp.47 (2022 (year only))
Expose Backdoors on the Way: A Feature-Based Efficient Defense against Textual Backdoor Attacks
2022.findings-emnlp.497 (2022 (year only))
Holistic Sentence Embeddings for Better Out-of-Distribution Detection
get_base_score 2 other papers
2 dated earlier
1610.02136 (2016-10)
A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
2022.findings-emnlp.497 (2022 (year only))
Holistic Sentence Embeddings for Better Out-of-Distribution Detection
get_is_pos 2 other papers
2 dated earlier
2022.findings-emnlp.47 (2022 (year only))
Expose Backdoors on the Way: A Feature-Based Efficient Defense against Textual Backdoor Attacks
2022.findings-emnlp.497 (2022 (year only))
Holistic Sentence Embeddings for Better Out-of-Distribution Detection
roc 2 other papers
2 dated earlier
2022.findings-emnlp.47 (2022 (year only))
Expose Backdoors on the Way: A Feature-Based Efficient Defense against Textual Backdoor Attacks
2022.findings-emnlp.497 (2022 (year only))
Holistic Sentence Embeddings for Better Out-of-Distribution Detection
get_cse_features 1 other paper
1 dated earlier
2022.findings-emnlp.497 (2022 (year only))
Holistic Sentence Embeddings for Better Out-of-Distribution Detection
get_d2u_score 1 other paper
1 dated earlier
2022.findings-emnlp.497 (2022 (year only))
Holistic Sentence Embeddings for Better Out-of-Distribution Detection
get_dropout_score 1 other paper
1 dated earlier
2022.findings-emnlp.497 (2022 (year only))
Holistic Sentence Embeddings for Better Out-of-Distribution Detection
get_features 1 other paper
1 dated earlier
2022.findings-emnlp.497 (2022 (year only))
Holistic Sentence Embeddings for Better Out-of-Distribution Detection
get_grad_norm_score 1 other paper
1 dated earlier
2022.findings-emnlp.497 (2022 (year only))
Holistic Sentence Embeddings for Better Out-of-Distribution Detection
get_key_positions 1 other paper
1 dated earlier
2022.findings-emnlp.497 (2022 (year only))
Holistic Sentence Embeddings for Better Out-of-Distribution Detection
get_keyword_confidence 1 other paper
1 dated earlier
2022.findings-emnlp.497 (2022 (year only))
Holistic Sentence Embeddings for Better Out-of-Distribution Detection
get_msp_score 1 other paper
1 dated earlier
2022.findings-emnlp.497 (2022 (year only))
Holistic Sentence Embeddings for Better Out-of-Distribution Detection
get_num_labels 1 other paper
1 dated earlier
2022.findings-emnlp.497 (2022 (year only))
Holistic Sentence Embeddings for Better Out-of-Distribution Detection
get_ppl_confidence 1 other paper
1 dated earlier
2022.findings-emnlp.497 (2022 (year only))
Holistic Sentence Embeddings for Better Out-of-Distribution Detection
take_mean_and_reshape 1 other paper
1 dated earlier
2022.findings-emnlp.497 (2022 (year only))
Holistic Sentence Embeddings for Better Out-of-Distribution Detection

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

The introduction of ChatGPT has garnered widespread attention in both academic and industrial communities. ChatGPT is able to respond effectively to a wide range of human questions, providing fluent and comprehensive answers that significantly surpass previous public chatbots in terms of security and usefulness. On one hand, people are curious about how ChatGPT is able to achieve such strength and how far it is from human experts. On the other hand, people are starting to worry about the potential negative impacts that large language models (LLMs) like ChatGPT could have on society, such as fake news, plagiarism, and social security issues. In this work, we collected tens of thousands of comparison responses from both human experts and ChatGPT, with questions ranging from open-domain, financial, medical, legal, and psychological areas. We call the collected dataset the Human ChatGPT Comparison Corpus (HC3). Based on the HC3 dataset, we study the characteristics of ChatGPT's responses, the differences and gaps from human experts, and future directions for LLMs. We conducted comprehensive human evaluations and linguistic analyses of ChatGPT-generated content compared with that of humans, where many interesting results are revealed. After that, we conduct extensive experiments on how to effectively detect whether a certain text is generated by ChatGPT or humans. We build three different detection systems, explore several key factors that influence their effectiveness, and evaluate them in different scenarios. The dataset, code, and models are all publicly available at https://github.com/Hello-SimpleAI/chatgpt-comparison-detection.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2301.07597")
get_code_for_paper("2301.07597")
have("2301.07597")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2301.07597.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2301.07597.svg)](https://syntology.ai/paper/2301.07597)

Connect an agent — have() is free.