Prompts Without Evidence: How Neuroimaging Mentions Shift Clinical Vision-Language Model Predictions

Nam Doan, Simone Vu, Balloccu

arXiv · PDF · Open in the Atlas

Code that ran

We lifted 18 functions out of this paper's own repositories and ran 15 of them in a sandbox. "Ran" means the function executed on a synthesized input and returned a value. It is not a reproduction of the paper's results.

Repositories linked to this paper

• long21wt/scaffold-effect

Some links come from the archived Papers with Code dataset (CC BY-SA 4.0): attribution and licence.

Abstract

Trustworthy clinical AI must ground its performance in genuine evidence rather than surfacelevel artifacts. We evaluate 12 open-weight vision-language models (VLMs) on two clinical neuroimaging cohorts for binary classification of affective disorders and cognitive decline. Both include structural MRI collected incidentally for unrelated studies, whose neuroimaging markers carry no reliable individual-level diagnostic signal. Nevertheless, smaller VLMs gain up to 0.58 in F1 when neuroimaging context is introduced, becoming competitive with models an order of magnitude larger. Confidence estimation indicates the calibration improvement on those smaller models is driven mainly by the prompt-level MRI reference, even without images. Our expert preliminary clinical case study finds faithfulness remains low in every condition examined, with models introducing unverified clinical details. Finally, preference alignment suppresses MRI-referencing behavior but collapses performance toward random baseline, leaving the underlying failure mode unresolved. These results caution against reading surface metric gains as evidence of genuine multimodal integration, with direct implications for clinical VLM deployment 1 .

For agents

The same record, over MCP at https://syntology.ai/mcp:

get_harvested_code_for_paper("2603.28387")
get_code_for_paper("2603.28387")
have("2603.28387")

Connect an agent — have() is free.