SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2401.06091 · NeurIPS · 2024

A Closer Look at AUROC and AUPRC under Class Imbalance

Haoran Zhang, Matthew Mcdermott, Lasse Hansen, Giovanni Angelotti, Jack Gallifant

arXiv · PDF · Open in the Atlas

Code that ran

25 of 34 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 1 honoured the contract we drafted, 1 violated it and 0 have no contract verdict; 23 failures of our instrument; 9 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2401.06091.json · badge: /api/ran/2401.06091.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
Lassehhansen/ArxivMLClaimSearch canonical (classifier label)
classifier basis: census
11 ran · 2 no recorded run
lassehhansen/arxiv-search canonical (classifier label)
classifier basis: census
8 ran · 4 no recorded run
hzhang0/auc_bias canonical (classifier label)
classifier basis: census
6 ran · 3 no recorded run
FunctionStatusWhere it lives
binary_metrics Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
hzhang0/auc_bias/auc_biases/metrics.py
pointer only (licence: NONE) · get_code("da3008f9c1d07696")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, hzhang0/auc_bias when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

block_until_running Ran
ran, violated the contract we drafted
hzhang0/auc_bias/command_launchers.py
pointer only (licence: NONE) · get_code("9d4328ab806dde75")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, hzhang0/auc_bias when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

combinations_base Ran
ran; our contract draft was wrong, not the code
hzhang0/auc_bias/experiments.py
pointer only (licence: NONE) · get_code("5a9f7480b832c15a")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, hzhang0/auc_bias when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

create_keyword_pattern Ran
ran; our contract draft was wrong, not the code
Lassehhansen/ArxivMLClaimSearch/src/arxiv_search.py
code served (permissive licence) · get_code("6f934cc1b439e9f8")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned Pattern. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def create_keyword_pattern(keywords):
    """
    Create a regex pattern for keyword matching.
    """
    pattern = '(?:(?<=\\W)|(?<=^))(' + '|'.join(map(re.escape, keywords)) + ')(?=\\W|$)'
    return re.compile(pattern, re.IGNORECASE)


# Setup code to construct valid arguments
keywords = ['python', 'regex', 'pattern', 'keyword']

ARGS = {
    'keywords': keywords
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type Pattern — a Pattern output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a Pattern output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
default_hparams Ran
ran; our contract draft was wrong, not the code
hzhang0/auc_bias/auc_biases/hparams_registry.py
pointer only (licence: NONE) · get_code("5a5be97e16d42730")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, hzhang0/auc_bias when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

extract_context_windows_df Ran
ran; our contract draft was wrong, not the code
Lassehhansen/ArxivMLClaimSearch/src/claim_search_v3.py
code served (permissive licence) · get_code("79c40b375614cc13")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re
import pandas as pd

# Create sample data
text1 = "The quick brown fox jumps over the lazy dog and the fox runs fast"
text2 = "Machine learning is fascinating and machine learning requires practice"

# Create compiled regexes
regex1 = re.compile(r'fox')
regex2 = re.compile(r'machine')

compiled_regexes = [regex1, regex2]

# Create a DataFrame with metadata
df = pd.DataFrame({
    'id': [1, 2],
    'content': [text1, text2],
    'source': ['story', 'tutorial'],
    'priority': [10, 20]
})

# Define window size
window_size = 2

# Create the ARGS dictionary
ARGS = {
    'df': df,
    'text_column': 'content',
    'compiled_regexes': compiled_regexes,
    'window_size': window_size
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_context_windows Ran
ran; our contract draft was wrong, not the code
Lassehhansen/ArxivMLClaimSearch/src/claim_search_openai.py
code served (permissive licence) · get_code("3f8be895e9cf2daa")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def create_keyword_pattern(keywords):
    """
    Create a regex pattern for keyword matching.
    """
    pattern = '(?:(?<=\\W)|(?<=^))(' + '|'.join(map(re.escape, keywords)) + ')(?=\\W|$)'
    return re.compile(pattern, re.IGNORECASE)

def get_context_windows(text, keywords, window_size):
    """
    Extract context windows around keywords in the text, ensuring that overlapping
    windows are merged into one.
    """
    context_windows = []
    keyword_pattern = create_keyword_pattern(keywords)
    matches = list(keyword_pattern.finditer(text))
    merged_matches = []
    i = 0
    while i < len(matches):
        current_match = matches[i]
        start_pos = current_match.start()
        end_pos = current_match.end()
        while i + 1 < len(matches) and matches[i + 1].start() - end_pos <= window_size * 2:
            i += 1
            end_pos = matches[i].end()
        merged_matches.append((start_pos, end_pos))
        i += 1
    for start_pos, end_pos in merged_matches:
        words = text.split()
        start_word_pos = len(text[:start_pos].split()) - 1
        end_word_pos = len(text[:end_pos].split())
        context_words = words[max(0, start_word_pos - window_size):min(len(words), end_word_pos + window_size)]
        context_str = ' '.join(context_words)
        context_windows.append(context_str)
    return context_windows


ARGS = {
    "text": "The quick brown fox jumps over the lazy dog. The fox is clever and smart. A lazy dog sleeps all day.",
    "keywords": ["fox", "lazy", "dog"],
    "window_size": 2
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_context_windows Ran
ran; our contract draft was wrong, not the code
Lassehhansen/ArxivMLClaimSearch/src/claim_search_v3.py
code served (permissive licence) · get_code("2dbdcd8611f7c89e")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

# Create sample text with some content to search through
text = """
The quick brown fox jumps over the lazy dog.
The fox is very clever and smart.
Another fox was seen running through the forest.
The dog barked at the fox yesterday.
"""

# Create compiled regex patterns to find matches
# Pattern 1: matches the word "fox"
regex1 = re.compile(r'\bfox\b')
# Pattern 2: matches the word "dog"
regex2 = re.compile(r'\bdog\b')

compiled_regexes = [regex1, regex2]

# Window size: number of words around the match to include
window_size = 3

ARGS = {
    'text': text,
    'compiled_regexes': compiled_regexes,
    'window_size': window_size,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family float — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_grouped_venue_papers Ran
ran; our contract draft was wrong, not the code
lassehhansen/arxiv-search/openreview_papers/paper.py
pointer only (licence: NONE) · get_code("019cb7a55d32fcdd")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, lassehhansen/arxiv-search when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

get_papers Ran
ran; our contract draft was wrong, not the code
lassehhansen/arxiv-search/openreview_papers/paper.py
pointer only (licence: NONE) · get_code("449e5f6e8c9560ed")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, lassehhansen/arxiv-search when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

get_slurm_jobs Ran
ran, honoured the contract we drafted
hzhang0/auc_bias/command_launchers.py
pointer only (licence: NONE) · get_code("f2059a864ab0dc16")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, hzhang0/auc_bias when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

load_papers Ran
ran; our contract draft was wrong, not the code
lassehhansen/arxiv-search/openreview_papers/utils.py
pointer only (licence: NONE) · get_code("a65cf511fdf4c398")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, lassehhansen/arxiv-search when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

papers_to_list Ran
ran; our contract draft was wrong, not the code
lassehhansen/arxiv-search/openreview_papers/utils.py
pointer only (licence: NONE) · get_code("f193f2c1639e1788")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, lassehhansen/arxiv-search when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

process_all_context_windows Ran
ran; our contract draft was wrong, not the code
Lassehhansen/ArxivMLClaimSearch/src/claim_search_v4.py
code served (permissive licence) · get_code("e06e4c3c4735ffbb")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned DataFrame. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd
import numpy as np

# Create a small DataFrame with context windows
data = {
    'context_window': [
        'This is the first context window about machine learning.',
        'This is the second context window about deep learning.',
    ]
}
new_df = pd.DataFrame(data)

# Define all parameters for process_all_context_windows
ARGS = {
    'new_df': new_df,
    'model': 'gpt-3.5-turbo',
    'system_prompt': 'You are a helpful assistant that summarizes text.',
    'introduction_statement_prompt': 'Please analyze the following text:',
    'end_statement_prompt': 'Provide a concise summary.',
    'openai_api_key': 'sk-test-key-1234567890',
    'texts_before_pause': 1000,
    'pause_duration': 1,
    'max_workers': 1,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, openai, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type DataFrame — a DataFrame output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a DataFrame output the recorded ndim 2 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
process_file Ran
ran; our contract draft was wrong, not the code
Lassehhansen/ArxivMLClaimSearch/src/arxiv_search.py
code served (permissive licence) · get_code("c4d0403439e31706")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import json
import re
import tempfile
import os

# Create a temporary JSONL file with test data
temp_dir = tempfile.mkdtemp()
temp_file = os.path.join(temp_dir, "test.jsonl")

test_data = [
    {"text": "We computed the AUROC metric", "meta": {"author": "Alice", "year": 2020}},
    {"text": "The AUPRC score was calculated", "meta": {"author": "Bob", "year": 2021}},
    {"text": "Both AUROC and AUPRC were evaluated", "meta": {"author": "Charlie", "year": 2022}},
    {"text": "No metrics mentioned here", "meta": {"author": "David", "year": 2023}},
    {"text": "AUROC\\textbf{test} with LaTeX commands", "meta": {"author": "Eve", "year": 2024}},
]

with open(temp_file, 'w', encoding='utf-8') as f:
    for item in test_data:
        f.write(json.dumps(item) + "\n")

# Define the regex patterns
auroc_pattern = re.compile(r'AUROC', re.IGNORECASE)
auprc_pattern = re.compile(r'AUPRC', re.IGNORECASE)

# Define metadata keys
metadata_keys = ['author', 'year']

# Define arguments
ARGS = {
    'file_path': temp_file,
    'auroc_pattern': auroc_pattern,
    'auprc_pattern': auprc_pattern,
    'metadata_keys': metadata_keys,
    'remove_latex': True,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
process_file Ran
ran; our contract draft was wrong, not the code
Lassehhansen/ArxivMLClaimSearch/src/arxiv_search_regex.py
code served (permissive licence) · get_code("1ba401f99899baa4")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re
import json
import tempfile
import os

# Create a temporary file with test data
temp_file = tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.jsonl', encoding='utf-8')
temp_file_path = temp_file.name

# Write test data with LaTeX commands and metric mentions
test_data = [
    {'text': r'The model achieved an AUROC of 0.95 using \textbf{bold} formatting.', 'meta': {'author': 'Alice', 'year': 2023}},
    {'text': r'Results show AUPRC = 0.88 with \emph{emphasis} and $x^2$ math.', 'meta': {'author': 'Bob', 'year': 2022}},
    {'text': r'No metrics here just plain text with \newline escape', 'meta': {'author': 'Charlie', 'year': 2021}},
]

for entry in test_data:
    temp_file.write(json.dumps(entry) + '\n')
temp_file.close()

# Create regex patterns for AUROC and AUPRC
auroc_regex = re.compile(r'AUROC|auroc', re.IGNORECASE)
auprc_regex = re.compile(r'AUPRC|auprc', re.IGNORECASE)

# Define metadata keys to extract
metadata_keys = ['author', 'year']

# Remove LaTeX flag
remove_latex = True

ARGS = {
    'file_path': temp_file_path,
    'auroc_regex': auroc_regex,
    'auprc_regex': auprc_regex,
    'metadata_keys': metadata_keys,
    'remove_latex': remove_latex,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
process_file Ran
ran; our contract draft was wrong, not the code
lassehhansen/arxiv-search/src/arxiv_search.py
pointer only (licence: NONE) · get_code("66ad8fb02e5834bf")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, lassehhansen/arxiv-search when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

process_file Ran
ran; our contract draft was wrong, not the code
lassehhansen/arxiv-search/src/arxiv_search_regex.py
pointer only (licence: NONE) · get_code("5e0a79f969743a5f")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, lassehhansen/arxiv-search when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

process_with_gpt_with_retries Ran
ran; our contract draft was wrong, not the code
Lassehhansen/ArxivMLClaimSearch/src/claim_search_v3.py
code served (permissive licence) · get_code("66d9d0c1c6c5bb7d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import openai
import time
from openai import OpenAI

# Define ARGS dictionary with all required parameters
ARGS = {
    "context_window": "What is the capital of France?",
    "model": "gpt-3.5-turbo",
    "system_prompt": "You are a helpful assistant that provides accurate geographical information.",
    "openai_api_key": "sk-test-1234567890abcdefghijklmnopqrstuvwxyz",
    "max_retries": 5
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
openai — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
process_with_gpt_with_retries Ran
ran; our contract draft was wrong, not the code
Lassehhansen/ArxivMLClaimSearch/src/claim_search_v4.py
code served (permissive licence) · get_code("85c492fe533d7a74")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import openai
from openai import OpenAI

ARGS = {
    'context_window': 'This is a sample context window with information to be processed by the model.',
    'model': 'gpt-3.5-turbo',
    'system_prompt': 'You are a helpful assistant that processes and summarizes information.',
    'introduction_statement_prompt': 'Please analyze the following:',
    'end_statement_prompt': 'Provide a clear summary.',
    'openai_api_key': 'sk-test-key-placeholder-1234567890',
    'max_retries': 3,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
openai — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
random_hparams Ran
ran; our contract draft was wrong, not the code
hzhang0/auc_bias/auc_biases/hparams_registry.py
pointer only (licence: NONE) · get_code("ab9845f95d99f073")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, hzhang0/auc_bias when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

remove_latex_commands Ran
ran; our contract draft was wrong, not the code
lassehhansen/arxiv-search/src/arxiv_search.py
pointer only (licence: NONE) · get_code("a56f744d089c3d45")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, lassehhansen/arxiv-search when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

remove_latex_commands Ran
ran; our contract draft was wrong, not the code
Lassehhansen/ArxivMLClaimSearch/src/arxiv_search.py
code served (permissive licence) · get_code("5c99e79304386be5")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def remove_latex_commands(s):
    """
    Remove LaTeX commands from a string.
    
    Parameters:
        s (str): The input string.
        
    Returns:
        str: The string with LaTeX commands removed.
    """
    if s is None:
        return ''
    s = re.sub('\\\\[nrt]|[\\n\\r\\t]', ' ', s)
    s = re.sub('\\\\[nrt]|[\\n\\r\\t]', ' ', s)
    s = re.sub('\\\\[a-zA-Z]+', '', s)
    s = re.sub('\\\\.', '', s)
    s = re.sub('\\\\begin\\{.*?\\}.*?\\\\end\\{.*?\\}', '', s, flags=re.DOTALL)
    s = re.sub('\\$.*?\\$', '', s)
    s = re.sub('\\\\[.*?\\\\]', '', s)
    s = re.sub('\\\\\\(.*?\\\\\\)', '', s)
    s = re.sub('\\\\\\[.*?\\\\\\]', '', s)
    s = re.sub('(?<=\\W)\\\\|\\\\(?=\\W)', '', s)
    return s.strip()

# Test string with various LaTeX commands
test_string = r"This is \textbf{bold} text with \emph{emphasis}. Here's math: $x^2 + y^2 = z^2$ and \begin{equation}a + b = c\end{equation}. Also \[inline\] and \(more\) stuff."

ARGS = {
    "s": test_string
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
remove_latex_commands Ran
ran; our contract draft was wrong, not the code
Lassehhansen/ArxivMLClaimSearch/src/arxiv_search_regex.py
code served (permissive licence) · get_code("476852e5a210420b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import re

def remove_latex_commands(s):
    """
    Removes LaTeX commands from a given string.

    Parameters:
    - s (str): A string potentially containing LaTeX commands.

    Returns:
    - str: The input string with LaTeX commands removed.

    Behavior:
    - This function cleans the input string by removing common LaTeX formatting commands,
      such as command keywords preceded by backslashes, and content within LaTeX math environments.
    - It aims to leave plain text content unaffected, making it more readable or suitable for further processing.
    """
    if s is None:
        return ''
    s = re.sub('\\\\[nrt]|[\\n\\r\\t]', ' ', s)
    s = re.sub('\\\\[a-zA-Z]+', '', s)
    s = re.sub('\\\\.', '', s)
    s = re.sub('\\\\begin\\{.*?\\}.*?\\\\end\\{.*?\\}', '', s, flags=re.DOTALL)
    s = re.sub('\\$.*?\\$', '', s)
    s = re.sub('\\\\[.*?\\\\]', '', s)
    s = re.sub('\\\\\\(.*?\\\\\\)', '', s)
    s = re.sub('\\\\\\[.*?\\\\\\]', '', s)
    s = re.sub('(?<=\\W)\\\\|\\\\(?=\\W)', '', s)
    return s.strip()


ARGS = {
    's': r'This is \textbf{bold} text with $x^2$ math and \begin{equation}a+b\end{equation} environments.'
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
remove_latex_commands Ran
ran; our contract draft was wrong, not the code
lassehhansen/arxiv-search/src/arxiv_search_regex.py
pointer only (licence: NONE) · get_code("b93ed8c955f1e9b1")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, lassehhansen/arxiv-search when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. A run is recorded.

add_scaler Not yet run
unverified (harvested, no recorded run)
hzhang0/auc_bias/auc_biases/models.py
pointer only (licence: NONE) · get_code("0c0d237b93f855a4")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, hzhang0/auc_bias when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

combinations Not yet run
unverified (harvested, no recorded run)
hzhang0/auc_bias/experiments.py
pointer only (licence: NONE) · get_code("f64641d62dc45aa9")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, hzhang0/auc_bias when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

create_keyword_pattern Not yet run
unverified (harvested, no recorded run)
lassehhansen/arxiv-search/src/claim_search_openai.py
pointer only (licence: NONE) · get_code("af7bf864cd646a1a")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, lassehhansen/arxiv-search when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

get_context_windows Not yet run
unverified (harvested, no recorded run)
lassehhansen/arxiv-search/src/claim_search_openai.py
pointer only (licence: NONE) · get_code("d21c640c6f69c578")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, lassehhansen/arxiv-search when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

get_hparams Not yet run
unverified (harvested, no recorded run)
hzhang0/auc_bias/experiments.py
pointer only (licence: NONE) · get_code("73d0cb5bbc5c4f0c")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, hzhang0/auc_bias when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

jsonl_folder_filtering Not yet run
unverified (harvested, no recorded run)
Lassehhansen/ArxivMLClaimSearch/src/arxiv_search_regex.py
code served (permissive licence) · get_code("469bb31279dee5bc")
How it ran

no invocation recorded.

jsonl_folder_filtering Not yet run
unverified (harvested, no recorded run)
lassehhansen/arxiv-search/src/arxiv_search_regex.py
pointer only (licence: NONE) · get_code("20187fbba2f056ae")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, lassehhansen/arxiv-search when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

process_with_gpt Not yet run
unverified (harvested, no recorded run)
Lassehhansen/ArxivMLClaimSearch/src/claim_search_openai.py
code served (permissive licence) · get_code("1dfd354e2f7c0776")
How it ran

no invocation recorded.

process_with_gpt_with_retries Not yet run
unverified (harvested, no recorded run)
lassehhansen/arxiv-search/src/claim_search_v3.py
pointer only (licence: NONE) · get_code("56f3d7119ff6c9c6")
How it ran

The body of this sample is withheld: GitHub detected no licence in this paper's repository, lassehhansen/arxiv-search when it was checked. The setup block that called it is withheld with it, because it is written against that body. source_path and origin say where the code lives, to read under the repository's own terms. No invocation recorded.

Also present in other papers' code

Of 34 samples lifted from this paper's linked repositories, 34 carry a function identity and 0 do not. Of the 27 distinct functions with an identity, 9 are also present in other papers whose code we hold (5 with at least one paper dated earlier, 0 the same month, 4 later) and 18 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

FunctionAlso present inEarliest-datedOthers named
block_until_running 2 other papers
2 dated earlier
2210.10769 (2022-10)
"Why did the Model Fail?": Attributing Model Performance Changes to Distribution Shifts
2302.12254 (2023-02)
Change is Hard: A Closer Look at Subpopulation Shift
get_slurm_jobs 2 other papers
2 dated earlier
2210.10769 (2022-10)
"Why did the Model Fail?": Attributing Model Performance Changes to Distribution Shifts
2302.12254 (2023-02)
Change is Hard: A Closer Look at Subpopulation Shift
combinations 1 other paper
1 dated earlier
2210.10769 (2022-10)
"Why did the Model Fail?": Attributing Model Performance Changes to Distribution Shifts
combinations_base 1 other paper
1 dated earlier
2210.10769 (2022-10)
"Why did the Model Fail?": Attributing Model Performance Changes to Distribution Shifts
create_keyword_pattern 1 other paper
1 later
2405.05506 (2024-05)
Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias
default_hparams 1 other paper
1 later
2605.16372 (2026-05)
SWORDBENCH: Evaluating Orthogonality of Steering Image Representations
get_hparams 1 other paper
1 dated earlier
2210.10769 (2022-10)
"Why did the Model Fail?": Attributing Model Performance Changes to Distribution Shifts
random_hparams 1 other paper
1 later
2605.16372 (2026-05)
SWORDBENCH: Evaluating Orthogonality of Steering Image Representations
remove_latex_commands 1 other paper
1 later
2405.05506 (2024-05)
Cross-Care: Assessing the Healthcare Implications of Pre-training Data on Language Model Bias

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

In machine learning (ML), a widespread claim is that the area under the precisionrecall curve (AUPRC) is a superior metric for model comparison to the area under the receiver operating characteristic (AUROC) for tasks with class imbalance. This paper refutes this notion on two fronts. First, we theoretically characterize the behavior of AUROC and AUPRC in the presence of model mistakes, establishing clearly that AUPRC is not generally superior in cases of class imbalance. We further show that AUPRC can be a harmful metric as it can unduly favor model improvements in subpopulations with more frequent positive labels, heightening algorithmic disparities. Next, we empirically support our theory using experiments on both semi-synthetic and real-world fairness datasets. Prompted by these insights, we reviewed over 1.5 million scientific papers to understand the origin of this invalid claim-finding it is often made without citation, misattributed to papers that do not argue this point, and aggressively overgeneralized from source arguments. Our findings represent a dual contribution: a significant technical advancement in understanding the relationship between AUROC and AUPRC and a stark warning about unchecked assumptions in the ML community.

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2401.06091")
get_code_for_paper("2401.06091")
have("2401.06091")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2401.06091.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2401.06091.svg)](https://syntology.ai/paper/2401.06091)

Connect an agent — have() is free.