PSF-Med Leaderboard

How often does a medical vision-language model flip its yes/no answer when the same question about the same chest X-ray is rephrased? That's what this board tracks, and lower is better. Every paraphrase pair passed an equivalence filter judged by GPT-5-mini, which agrees with a Claude Haiku cross-check on 91.6 to 94.4% of pairs. A 1,200-pair clinician audit anchors the rubric.

Dataset: saillab/psf-med Code: GitHub Flip gallery: examples

Open medical VLMs on the full benchmark

The binary yes/no subset covers three populations: MIMIC-CXR from the US (1,539 questions, 5,076 pairs), PadChest from Spain (8,445 questions, 36,244 pairs), and VinDr-CXR from Vietnam (2,807 questions, 8,612 pairs). Cells show the pairwise flip rate in percent; the denominator is the paraphrase pair rather than the question. Click a column header to sort.

#	Model	MIMIC-CXR	PadChest	VinDr-CXR	Mean ▾
1	MedGemma-27B	6.4	13.9	8.1	9.5
2	CheXone	8.2	12.0	11.1	10.4
3	MedGemma-4B	8.3	13.4	15.3	12.3
4	MedGemma-1.5-4B	7.4	17.8	0.8†	12.6
5	LLaVA-Rad	15.6	0.8†	11.6	13.6
6	RadFM	13.7	32.8	54.7	33.7

† marks a cell where the model said yes to more than 98% of the original questions. A flip rate that low tells you the model has collapsed onto a single answer, so we leave those cells out of the mean.

Frontier API models on the hardest 500

Same three populations, but trimmed to the 500 hardest questions in each dataset and run in August 2026. So this is a stress test of the upper tail: don't compare these rates with the full-benchmark table above. The pooled column carries 95% Wilson intervals.

#	Model	MIMIC-CXR	PadChest	VinDr-CXR	Pooled ▾	Pairs
1	Kimi K3 reasoning: low	6.8	3.1	8.0	5.71 [5.1-6.3]	5,725
2	GPT-5-mini reasoning: low	11.2	2.4	5.1	5.93 [5.4-6.5]	6,192
3	Claude Opus 5 thinking off	9.3	5.8	10.6	8.36 [7.7-9.1]	6,194
4	Claude Opus 5 adaptive thinking	7.9	7.6	13.5	9.53 [8.8-10.3]	6,209
5	GPT-5.6 Sol reasoning: low	8.9	3.8	17.7	9.71 [9.0-10.5]	6,208
6	Claude Haiku 4.5 default	5.2	11.8	16.2	11.13 [10.4-11.9]	6,199

Each row shows the reasoning or thinking setting that arm used. Even the current flagships typically flip on about one hardest-set pair in ten. No family is immune.

Methodology & caveats

Every API arm used the same prompt, byte for byte: “You are a radiologist answering yes/no questions about chest X-ray images. Look at the image carefully and answer the question with exactly 'yes' or 'no'.” Images were capped at 512px on the long edge and saved as JPEG at quality 60, with 16-bit files rescaled to their min and max before conversion.
A pairwise flip is a paraphrase whose yes/no answer differs from the original question's answer on the same image. Query-level rates tend to run about double the pairwise ones.
The full-benchmark set keeps only questions with an unambiguous yes/no reference. And VinDr holds positive presence cases only, so it measures phrasing stability more than balanced accuracy.
Kimi K3 drops 5 to 15.6% of responses to always-on-thinking truncation at max_tokens=256. We drop unreadable output rather than coerce it, which leaves those cells slightly favorable.
Raw cells, intervals, accuracy, and the image-reliance controls for the frontier probe live in frontier_binary500_summary.json. The full-benchmark table comes from psf_med_leaderboard_data.json.

Submit a model

Run your model on saillab/psf-med with the prompt above, using greedy decoding or your vendor's defaults. If those differ, report both. Then open a discussion on this Space with per-dataset pairwise flip rates and raw outputs. We'll check them against the release manifest and add the row.

Citation

@dataset{psf_med_2026,
  title={{PSF-Med}: A Clinician-Audited Benchmark for Paraphrase Sensitivity in Medical Vision-Language Models},
  author={Sadanandan, Binesh and Behzadan, Vahid and Jayan, Lekshmy and Gopinatha Kurup, Arun},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/saillab/psf-med}
}

Maintained by SAIL Lab, University of New Haven · Last updated 2026-08-16 · Companion artifacts: mechanistically-guided LoRA adapters (CHIL 2026)