SYNTOLOGY HomeExplorerAtlasCodeMethodologyAboutDevelopersFeedPricing
Paper · 2307.02028 · 2023

EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models

arXiv · PDF · Open in the Atlas

Code that ran

11 of 16 samples (functions and classes) lifted from this paper's linked repositories executed on a synthesized input in Syntology's jail (0 of them constructed an object rather than computing a result; 2 honoured the contract we drafted, 0 violated it and 6 have no contract verdict; 3 failures of our instrument; 5 no recorded run), measured 2026-11-22; repository links from the Papers with Code archive attributed under CC BY-SA, no affiliation; bodies inline only under a permissive licence as GitHub detects it. Not a reproduction of the paper's results.

"Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results. This record, dated, as JSON: /api/ran/2307.02028.json · badge: /api/ran/2307.02028.svg

Each repository's role is a label: a classifier's output recorded on the link, or, where no role is recorded, what the Papers with Code archive marked. The line under it names its basis. Neither says who wrote the code.

RepositoryRoleRan
som-shahlab/ehrshot-benchmark canonical (classifier label)
classifier basis: census
5 ran · 4 no recorded run
stefanhgm/ehrshot-benchmark community (classifier label)
classifier basis not recorded on the link
6 ran · 1 no recorded run
FunctionStatusWhere it lives
calc_metrics Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
som-shahlab/ehrshot-benchmark/ehrshot/7b_eval_zero_shot.py
code served (permissive licence) · get_code("e387d3515af3e462")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np
from typing import List, Dict

# Set random seed for reproducibility
np.random.seed(0)

# Create synthetic test data
N = 20  # number of samples

# y_test: binary labels (0 or 1)
y_test = np.array([0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1], dtype=np.float64)

# y_test_proba: predicted probabilities for positive class (values between 0 and 1)
y_test_proba = np.array([0.1, 0.9, 0.8, 0.2, 0.85, 0.15, 0.05, 0.95, 0.75, 0.25, 
                          0.88, 0.12, 0.92, 0.87, 0.10, 0.08, 0.91, 0.20, 0.89, 0.86], dtype=np.float64)

# test_patient_ids: list of patient IDs corresponding to each test sample
# Using values from a small set of patients so resampling creates duplicates
test_patient_ids = [1, 2, 3, 4, 5, 1, 2, 3, 4, 5, 1, 2, 3, 4, 5, 1, 2, 3, 4, 5]

ARGS = {
    'y_test': y_test,
    'y_test_proba': y_test_proba,
    'test_patient_ids': test_patient_ids,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
jaxtyping, loguru, numpy, sklearn — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
compute_random_num Ran
ran, honoured the contract we drafted
som-shahlab/ehrshot-benchmark/ehrshot/labelers/core.py
code served (permissive licence) · get_code("26d24fff8949b09f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import hashlib
import struct

def compute_random_num(seed: int, num_1: int, num_2: int):
    network_num_1 = struct.pack('!q', num_1)
    network_num_2 = struct.pack('!q', num_2)
    network_seed = struct.pack('!q', seed)
    to_hash = network_seed + network_num_1 + network_num_2
    hash_object = hashlib.sha256()
    hash_object.update(to_hash)
    hash_value = hash_object.digest()
    result = 0
    for i in range(len(hash_value)):
        result = (result * 256 + hash_value[i]) % 100
    return result

ARGS = {
    "seed": 42,
    "num_1": 12345,
    "num_2": 67890
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
format_icl_examples Ran
ran on a synthesized input
stefanhgm/ehrshot-benchmark/ehrshot/10b_fit_and_eval_decoder.py
code served (permissive licence) · get_code("b570adb57127d95c")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
from typing import Sequence
from dataclasses import dataclass

@dataclass(frozen=True)
class ICLExample:
    index: int
    label: int
    text: str

# Create sample ICLExample instances
example1 = ICLExample(
    index=0,
    label=1,
    text="Patient presents with fever and cough for 3 days."
)

example2 = ICLExample(
    index=1,
    label=0,
    text="Patient reports feeling well with no symptoms."
)

example3 = ICLExample(
    index=2,
    label=1,
    text="Patient has chest pain and shortness of breath."
)

# Create a sequence of ICLExample instances
icl_examples_sequence = [example1, example2, example3]

# Define ARGS dict with the parameter
ARGS = {
    "icl_examples": icl_examples_sequence
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
get_splits Ran
ran; our contract draft was wrong, not the code
som-shahlab/ehrshot-benchmark/ehrshot/9_make_cohort_plots.py
code served (permissive licence) · get_code("ff1a98172ef1054f")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned dict. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pandas as pd
import tempfile
import os

# Create a temporary directory with the required CSV file
temp_dir = tempfile.mkdtemp()

# Create a sample person_id_map.csv with split information
data = {
    'split': ['train', 'train', 'val', 'test', 'train', 'val'],
    'omop_person_id': [1001, 1002, 1003, 1004, 1005, 1006]
}
splits_df = pd.DataFrame(data)
splits_df.to_csv(os.path.join(temp_dir, 'person_id_map.csv'), index=False)

ARGS = {
    'splits_dir': temp_dir
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type dict — a dict output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a dict output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
import_run_evaluation Ran
ran on a synthesized input
stefanhgm/ehrshot-benchmark/ehrshot/20_run_llm_variants_experiments.py
code served (permissive licence) · get_code("da830e412673ca56")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned function. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import os
import tempfile

# Create a temporary eval file with a run_evaluation function
eval_code = '''
def run_evaluation():
    """Dummy evaluation function for testing"""
    return {"result": "success"}
'''

# Write the eval file to a temporary location
with tempfile.NamedTemporaryFile(mode='w', suffix='.py', delete=False) as f:
    f.write(eval_code)
    temp_eval_path = f.name

ARGS = {
    'eval_py': temp_eval_path
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type function — a function output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a function output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
load_data Ran
ran on a synthesized input
stefanhgm/ehrshot-benchmark/ehrshot/10a_fit_and_eval_encoder.py
code served (permissive licence) · get_code("2612d55d41fef50d")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned tuple. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import pickle
import pandas as pd
import tempfile
import os
from io import BytesIO

# Create temporary directory for test files
temp_dir = tempfile.mkdtemp()

# Create a minimal splits CSV file
splits_csv_path = os.path.join(temp_dir, 'splits.csv')
splits_data = {
    'task': ['task1', 'task1', 'task2'],
    'split_name': ['train', 'val', 'train'],
    'shot_size': [5, 5, 10],
    'fold': [0, 0, 1],
    'patient_id': [101, 102, 103],
    'prediction_time': ['2023-01-01 10:00:00', '2023-01-02 11:00:00', '2023-01-03 12:00:00'],
    'label_type': ['binary', 'binary', 'binary'],
    'label_value': ['True', 'False', 'True'],
    'serialization_idx': [0, 1, 2]
}
splits_df = pd.DataFrame(splits_data)
splits_df.to_csv(splits_csv_path, index=False)

# Create a minimal serializations pickle file
serializations_path = os.path.join(temp_dir, 'serializations.pkl')
tasks_serializations = {
    'task1': {
        0: {'data': [1, 2, 3], 'metadata': 'sample1'},
        1: {'data': [4, 5, 6], 'metadata': 'sample2'}
    },
    'task2': {
        2: {'data': [7, 8, 9], 'metadata': 'sample3'}
    }
}
with open(serializations_path, 'wb') as f:
    pickle.dump(tasks_serializations, f)

# Create args object
class Args:
    def __init__(self, splits_path, serializations_path):
        self.splits_path = splits_path
        self.serializations_path = serializations_path

ARGS = {
    'args': Args(splits_csv_path, serializations_path)
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
loguru, pandas — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type tuple — a tuple output has no array shape of its own; none is stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
lr_lambda Ran
ran, honoured the contract we drafted
som-shahlab/ehrshot-benchmark/ehrshot/7_eval_finetune.py
code served (permissive licence) · get_code("3111cc4393ec453b")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned float, ndim 0, dtype family float; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

torch.manual_seed(0)

ARGS = {
    "current_step": 5,
    "total_steps": 10
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type float, ndim 0, dtype family float — no output shape is stated: observe_signature.py could not observe it (read:AttributeError)
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
resolve_instruction_from_serializations Ran
ran on a synthesized input
stefanhgm/ehrshot-benchmark/ehrshot/10a_fit_and_eval_encoder.py
code served (permissive licence) · get_code("61478e91a0817d84")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned str. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch
import numpy as np
from typing import Sequence, Iterable

torch.manual_seed(0)

# Create a sequence of serialization tuples
# Each entry is a tuple/list where:
# - entry[0] is the instruction (string)
# - entry[1] is some other data
tasks_serializations = [
    ("instruction_one", "data_1"),
    ("instruction_two", "data_2"),
    ("instruction_one", "data_3"),
    ("instruction_three", "data_4"),
]

# Create serialization indices that point to entries with the same instruction
serialization_indices = [0, 2]  # Both point to entries with "instruction_one"

# Create sub_task string
sub_task = "task_classification"

ARGS = {
    "tasks_serializations": tasks_serializations,
    "serialization_indices": serialization_indices,
    "sub_task": sub_task,
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type str — a str output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a str output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
select_icl_example_indices Ran
ran on a synthesized input
stefanhgm/ehrshot-benchmark/ehrshot/10b_fit_and_eval_decoder.py
code served (permissive licence) · get_code("29373a416d078cca")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned list, ndim 1, dtype family int; no output shape is stated. The image it ran in was not recorded. If your copy returns the same type and ndim on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import random
from typing import Sequence, Set, Optional, List

# Set random seed for reproducibility
random.seed(42)

# Construct arguments for select_icl_example_indices
labels = [0, 1, 0, 1, 1, 0, 1, 0]  # 8 examples with mixed labels
shot_count = 4  # Request 4 shots (in SUPPORTED_ICL_SHOTS)
seed = 123  # Random seed for deterministic selection
forbidden_indices = {0, 1}  # Exclude indices 0 and 1

ARGS = {
    'labels': labels,
    'shot_count': shot_count,
    'seed': seed,
    'forbidden_indices': forbidden_indices
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
none outside the standard library — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type list, ndim 1, dtype family int — a list output has no array shape of its own; none is stated
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
tune_hyperparams Ran
ran; the contract check could not run (RAISES, DEP_MISSING or TIMEOUT)
som-shahlab/ehrshot-benchmark/ehrshot/7_eval_finetune.py
code served (permissive licence) · get_code("1ccac98a15f1c7bc")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned LogisticRegression. The image it ran in was not recorded. If your copy returns the same type on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import numpy as np
from sklearn.linear_model import LogisticRegression

# Create small training data
np.random.seed(0)
n_train = 10
n_val = 5
n_features = 4

X_train = np.random.randn(n_train, n_features).astype(np.float32)
X_val = np.random.randn(n_val, n_features).astype(np.float32)
y_train = np.random.randint(0, 2, n_train)
y_val = np.random.randint(0, 2, n_val)

# Create a model that has the required interface
model = LogisticRegression(random_state=0, max_iter=100)

# Define parameter grid for tuning
param_grid = {
    'C': [0.1, 1.0],
    'penalty': ['l2']
}

ARGS = {
    'X_train': X_train,
    'X_val': X_val,
    'y_train': y_train,
    'y_val': y_val,
    'model': model,
    'param_grid': param_grid,
    'n_jobs': 1
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
numpy, scipy, sklearn — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type LogisticRegression — a LogisticRegression output has no array shape of its own; none is stated — the instrument coerces any output to an array; for a LogisticRegression output the recorded ndim 0 and dtype family float describe that coercion, not the output, and are not stated
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
validate_icl_shots Ran
ran on a synthesized input
stefanhgm/ehrshot-benchmark/ehrshot/10b_fit_and_eval_decoder.py
code served (permissive licence) · get_code("b0a8dc17cfae1347")
How it ran

Ran in Syntology's sandbox on this synthesized input (drafted by a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API) and returned int, ndim 0, dtype family int, shape () as a later run recorded it. The image it ran in was not recorded. If your copy returns the same type, ndim and shape on these inputs, it matched what we observed; nothing here checks that. It does NOT mean the output is correct: nothing has compared it to the paper.

import torch

SUPPORTED_ICL_SHOTS = (0, 2, 4, 6)

def validate_icl_shots(shot_count: int) -> int:
    if shot_count not in SUPPORTED_ICL_SHOTS:
        raise ValueError(f'Unsupported --icl_shots={shot_count}. Supported values: {SUPPORTED_ICL_SHOTS}')
    return shot_count

# Setup valid arguments
ARGS = {
    'shot_count': 2
}
Setup block drafted by
a language model, Claude Haiku 4.5, prompt python_fixture_v1, through the batch API
Imports
torch — import names as written, not package names to install; a name the repository itself provides is not a package
Returned
type int, ndim 0, dtype family int, shape () — shape () as a later run recorded it
dtype family
int means an integer dtype; float means any other dtype the instrument saw, bool and complex included (fixture_check.py records only these two families)
Environment
image not recorded; re-verify against syntology-harvest-sandbox:v5-deps; not recorded for this sample: the harness installs a cuda-to-cpu shim (cuda_shim.py) before every sample and no run records whether a sample needed it; seed: none by the harness; any seed is in the setup block
export_tuning_summary_table Not yet run
unverified (harvested, no recorded run)
stefanhgm/ehrshot-benchmark/ehrshot/12_merge_and_plot_revision_results.py
code served (permissive licence) · get_code("3eeebfadd62f674c")
How it ran

no invocation recorded.

fit_logreg_lbfgs Not yet run
unverified (harvested, no recorded run)
som-shahlab/ehrshot-benchmark/ehrshot/7_eval_finetune.py
code served (permissive licence) · get_code("937f2be12f9f3025")
How it ran

no invocation recorded.

generate_shots Not yet run
unverified (harvested, no recorded run)
som-shahlab/ehrshot-benchmark/ehrshot/6_generate_shots.py
code served (permissive licence) · get_code("f2dd7454f7178ecb")
How it ran

no invocation recorded.

get_k_samples Not yet run
unverified (harvested, no recorded run)
som-shahlab/ehrshot-benchmark/ehrshot/6_generate_shots.py
code served (permissive licence) · get_code("f591a2afaa8691c1")
How it ran

no invocation recorded.

guo_los_zeroshot_rollouts Not yet run
unverified (harvested, no recorded run)
som-shahlab/ehrshot-benchmark/ehrshot/7b_eval_zero_shot.py
code served (permissive licence) · get_code("0dec51f61a914fe7")
How it ran

no invocation recorded.

Also present in other papers' code

Of 16 samples lifted from this paper's linked repositories, 16 carry a function identity and 0 do not. Of the 16 distinct functions with an identity, 0 are also present in other papers whose code we hold (0 with at least one paper dated earlier, 0 the same month, 0 later) and 16 are only here among the 52,977 papers whose code Syntology holds (as of 2026-09-26).

Only here means: among the 52,977 papers whose code Syntology holds (51,887 with lifted samples, 1,090 through their assemblies only), as of 2026-09-26, no other paper's lifted samples or current assembly bricks contain an identical function body after normalisation.

Presence, not direction: an identical function body in two papers' repositories does not say who wrote it, which came from which, or that either paper's method rests on it. Matching is exact after normalisation, so a renamed or edited function is not matched. Dates are when papers appeared, not when code was written. Only .py files the harvester lifted are covered; notebooks are not read. A new-style arXiv id YYMM.NNNN dates a paper to the month 20YY-MM; otherwise Paper.year dates it to the year; otherwise it is undated. Dates are when the paper appeared, not when the code was written. Read 2026-11-22.

Repositories linked to this paper

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

While the general machine learning (ML) community has benefited from public datasets, tasks, and models, the progress of ML in healthcare has been hampered by a lack of such shared assets. The success of foundation models creates new challenges for healthcare ML by requiring access to shared pretrained models to validate performance benefits. We help address these challenges through three contributions. First, we publish a new dataset, EHRSHOT, which contains deidentified structured data from the electronic health records (EHRs) of 6,739 patients from Stanford Medicine. Unlike MIMIC-III/IV and other popular EHR datasets, EHRSHOT is longitudinal and not restricted to ICU/ED patients. Second, we publish the weights of CLMBR-T-base, a 141M parameter clinical foundation model pretrained on the structured EHR data of 2.57M patients. We are one of the first to fully release such a model for coded EHR data; in contrast, most prior models released for clinical data (e.g. GatorTron, ClinicalBERT) only work with unstructured text and cannot process the rich, structured data within an EHR. We provide an end-to-end pipeline for the community to validate and build upon its performance. Third, we define 15 few-shot clinical prediction tasks, enabling evaluation of foundation models on benefits such as sample efficiency and task adaptation. Our model and dataset are available via a research data use agreement from our website: https://ehrshot.stanford.edu. Code to reproduce our results are available at our Github repo: https://github.com/som-shahlab/ehrshot-benchmark

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2307.02028")
get_code_for_paper("2307.02028")
have("2307.02028")

The run record, dated, one paper per request, free:

curl https://syntology.ai/api/ran/2307.02028.json

A badge for a README (the split and the date, never a ratio):

[![Syntology run record](https://syntology.ai/api/ran/2307.02028.svg)](https://syntology.ai/paper/2307.02028)

Connect an agent — have() is free.