We have not lifted any functions out of this paper's repositories yet, so there is nothing we have run. The repositories linked to it are listed below.
Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.
State-of-the-art (SOTA) performance refers to the highest performance achieved by some model on a test sample, preferably under controlled conditions such as public data (reproducibility) or public challenges (independent sample). Thousands of classifiers are applied, and the highest performance becomes the new reference point for a particular problem. In effect, this set-up is an estimate of the expected best performance among all classifiers applied to a random sample; a sample maximum estimate. In this paper, we argue that SOTA should instead be estimated by the expected performance of the best classifier, which can be done without knowing which classifier it is. Our contribution is the formal distinction between the two, and an investigation into the practical consequences of using the former to estimate the latter. This is done by presenting sample maximum estimator distributions for non-identical and dependent classifiers. We illustrate the impact on real world examples from public challenges.
The same record, over MCP at https://syntology.ai/mcp:
get_harvested_code_for_paper("2303.07272")
get_code_for_paper("2303.07272")
have("2303.07272")
The run record, dated, one paper per request, free:
curl https://syntology.ai/api/ran/2303.07272.json
A badge for a README (the split and the date, never a ratio):
[](https://syntology.ai/paper/2303.07272)
Connect an agent — have() is free.