A new prompt-search method improved how two multimodal AI models ranked chest-X-ray cases across three test tasks, researchers report in a Sept. 30 arXiv preprint. It optimizes AUROC, which measures how often a positive case receives a higher score than a negative one, instead of the share of correct labels.
Accuracy can mislead when one label is far more common: a model may score well by repeatedly choosing that label, even if its scores do not separate cases with and without a finding. Ranking matters because screening teams may choose different score cutoffs for different uses. AUROC evaluates the ordering across cutoffs; it does not show that a model is safe at any specific clinical threshold.
Pair-level Pareto Prompt Evolution, or Ranking-PE, changes what the search rewards. It compares the model’s existing scores for positive and negative cases in pairs, then uses those results to filter candidate prompts, guide the language model that proposes revisions, and select the final prompt. The average pair score equals empirical AUROC. The authors report no additional model calls.
The researchers evaluated atelectasis, a small area of collapsed lung; cardiomegaly, an enlarged heart; and consolidation, a cloudy or whiter area in the lung on an X-ray. They treated each as a separate binary task, using chest X-rays and structured health-record context from MIMIC. Within each model pipeline, Ranking-PE and the accuracy-based Accuracy-PE search used the same starting weights and search budget. On Qwen3-VL-8B after supervised fine-tuning with its vision encoder trainable, mean test AUROC rose from 68.0% with Accuracy-PE to 73.8% with Ranking-PE. On medically pretrained MedGemma-4B, which skipped that tuning stage, it rose from 53.5% with Accuracy-PE to 69.7% with Ranking-PE. Both are unweighted averages across the three tasks.
These are retrospective benchmark results from one MIMIC cohort, not a clinical deployment or prospective test. The authors describe small patient-disjoint test sets with limited representation of negative cases. The study does not establish how well the ranking gains transfer to other hospitals or affect patient care.
Sources
- Study preprint (arXiv:2609.40361)
- Chest X-ray terms explained by the Radiological Society of North America and American College of Radiology
- Image: City of Minneapolis Archives, archival X-ray review photo; CC BY 2.0
