Bewertung von Basismodellen an deutschen medizinischen Lizenzprüfungen: Implikationen für KI-assistierte Ausbildung

npj | digital medicine Article Published in partnership with Seoul National University Bundang Hospital https://doi.org/10.1038/s41746-026-03082-7 Evaluating foundation models on official German medical licensing examinations: Implications for high-stakes assessment and AI-assisted medical education Check for updates 1,2 1 3 3,4 3 Lasse Cirkel , Johannes Knitza , Volker Schillings , Alexander Oksche , Jan Carl Becker & Sebastian Kuhn1 1234567890():,; 1234567890():,; We evaluated proprietary and open-weight foundation models on 24 German medical licensing examinations (2019–2024), including 7485 items and response data from 119,878 sittings. For fair comparison, the eight vision-capable models were evaluated on the full benchmark and all thirteen on a shared text-only subset. On the full benchmark, Gemini 3.1 Pro achieved the highest overall accuracy, reaching 99.31% on the first (M1) and 98.37% on the second (M2) examination. On the shared text-only subset, proprietary frontier models performed at near-ceiling levels, several open- weight models (including GLM-5 and DeepSeek V3.2-Thinking) were highly competitive, and even compact ones exceeded mean student performance. Image-present items were more difficult for both students and models, but the associated decline was disproportionately larger for models than for students. Human- and model-defined difficulty subsets showed limited overlap, and model-hard subsets revealed residual differences among top systems. These findings underscore the rapid progress of these models, particularly open-weight systems, and the value of official German medical licensing examinations as a restricted-access benchmark with reduced public exposure. They carry implications for high-stakes assessment and AI-assisted medical education, notably multimodal assessment, human-aligned educational tools, and privacy-preserving local deployment. Large language models (LLMs) are rapidly advancing from experimental This problem is not unique to medical AI, but reflects a broader pattern systems to increasingly capable tools for healthcare-related tasks. Early in AI evaluation as leading systems converge toward ceiling performance on milestones were marked by general-purpose models passing the United widely used benchmarks6. The same shift is now evident in medicine, where States Medical Licensing Examination (USMLE) without domain- reasoning-enabled models and recent frontier systems achieve very high specific fine-tuning1,2. Since then, performance on medical examina- scores on several established public benchmarks5,7–9. Although such results tions has improved substantially, with contemporary systems achieving document substantial capability gains, they also compress performance passing or near-ceiling scores across multiple national licensing and differences at the top end and make aggregate benchmark accuracy less specialist examinations3. In parallel, these models are increasingly being informative for comparing leading systems or tracking further progress. explored for clinical applications such as documentation support and This has increased the need for more discriminative medical evaluations medical question answering4,5. However, rising scores on established that remain informative in the near-ceiling regime. benchmarks also create an evaluation problem: once performance However, increasing benchmark difficulty alone does not resolve approaches saturation, benchmark results become less informative about another central evaluation problem: public exposure. Modern foundation true medical generalization and less useful for distinguishing among models are trained on large-scale web-crawled corpora and other broad data frontier systems. sources, meaning that publicly accessible medical test sets may be 1 Marburg University, Institute for Digital Medicine, Marburg, Germany. 2Marburg University, Institute of Artificial Intelligence in Medicine, Marburg, Germany. 3 German Institute for State Examinations in Medicine, Pharmacy, Dentistry and Psychotherapy (IMPP), Mainz, Germany. 4Justus Liebig University Giessen, Rudolf- Buchheim-Institute of Pharmacology, Giessen, Germany. e-mail: lasse.cirkel@gmail.com npj Digital Medicine | (2026)9:670 1 https://doi.org/10.1038/s41746-026-03082-7 Article represented directly or indirectly in training data. For example, Gallifant alternative, yet large-scale comparative evidence on their medical reasoning et al. reported that 99.21% of the MedQA test split was detectable in Dolma performance relative to proprietary frontier systems remains limited. 1.6, which they used as an open pretraining corpus proxy, based on 8-gram Here, we evaluate proprietary and locally deployable open-weight overlap screening10. More broadly, benchmark data contamination is foundation models on the first (M1, taken after the second year) and increasingly recognized as a substantial source of evaluation bias, and its second (M2, taken after the fifth year) German medical licensing exam- reliable detection remains methodologically challenging11. Although public- inations within the six-year medical curriculum, in direct cooperation benchmark hardening strategies such as question rephrasing and option with IMPP. Both examinations consist of written single-best-answer augmentation can increase difficulty and may reduce exact overlap12, they multiple-choice items with five response options (A–E) and are admi- do not eliminate the fact that such resources remain rooted in publicly nistered twice yearly (spring and fall), with approximately 10,000 annual accessible source material; moreover, rephrased samples can make con- examination sittings in each of M1 and M2. The study is based on a tamination harder to detect rather than irrelevant13. As a result, strong restricted-access dataset of 7485 official examination items from 2019 to performance on public medical benchmarks can overestimate true gen- 2024, reducing the risk of direct benchmark contamination relative to eralization to previously unseen exam-style or clinical questions. Restricted- publicly accessible medical question-answering resources. To enable fair access evaluation sets are therefore particularly valuable because they reduce comparison across heterogeneous model classes, we distinguish a shared the risk of direct benchmark contamination and provide a more stringent text-only subset from image-present items. Because a substantial pro- test of medical reasoning on items that are less likely to have appeared in pre- portion of M2 items is embedded in sequentially linked clinical case training data. vignettes, in which several consecutive items share a common clinical A further limitation of current evidence is its narrow coverage of scenario, we apply a conservative vignette-aware exclusion rule to prevent language, region, and medical training context. Most established medical within-vignette leakage in text-only analyses. We further compare pro- benchmarks are English-language and aligned with US-centric educational prietary frontier systems with deployment-relevant open-weight alter- and guideline environments, leaving open how well frontier models gen- natives and relate model difficulty to official student difficulty indices at eralize to region-specific standards in other languages. The German medical the item level. Finally, to retain discriminatory power in the near-ceiling licensing examinations therefore provide a particularly stringent test case regime, we define difficulty-enriched text-only subsets based on psy- because they are administered in German and reflect national curricular chometric and model-derived difficulty. Together, these analyses provide priorities and guideline-concordant practice. Earlier studies evaluating a rigorous evaluation framework for assessing contemporary foundation LLMs on these examinations or on substantial subsets of their questions14–19 models on official German medical licensing examinations across lan- provided useful proof-of-concept observations. However, they relied on guage, modality, and deployment context, with broader implications for examination items accessed through secondary educational platforms high-stakes assessment and AI-assisted medical education. rather than on the authoritative primary dataset provided directly by the German Institute for State Examinations in Medicine, Pharmacy, Dentistry Results and Psychotherapy (IMPP), and they were conducted without official Overall performance on the full benchmark in vision- cooperation with IMPP. Consequently, they could not benchmark models capable models on the authoritative source material itself or consistently leverage official To assess end-to-end multimodal performance, we first evaluated vision- item-level metadata within a controlled evaluation pipeline. Methodological capable models on the full benchmark, defined as all retained items for the handling also varied across studies, including heterogeneous treatment of respective examination set (Fig. 1A,B). On M1 (n = 3747), Gemini 3.1 Pro media-containing questions, the absence of an explicit vignette-aware achieved the highest overall accuracy at 99.31% (95% CI 98.99–99.53), strategy for fair text-only subset construction, interface-based rather than followed by GPT-5.4 at 98.48% (95% CI 98.03–98.82) and Claude Opus 4.6 fully standardized programmatic evaluation workflows in some cases, and at 96.93% (95% CI 96.33–97.44). Among open-weight vision-capable comparatively narrow model coverage focused mainly on early ChatGPT models, Kimi K2.5 reached 96.72% (95% CI 96.10–97.24), followed by versions and a small number of other commercial systems. Qwen3-VL at 94.45% (95% CI 93.67–95.14) and Mistral Large 3 at 87.75% Current evidence also incompletely captures multimodal medical (95% CI 86.66–88.76). reasoning. Many medically relevant tasks require the joint interpretation of On M2 (n = 3738), Gemini 3.1 Pro again achieved the highest overall textual and visual information, including clinical photographs, radiological accuracy at 98.37% (95% CI 97.91–98.73). Claude Opus 4.6 and GPT-5.4 images, histopathology, electrocardiograms, and scientific figures20. This is followed at 97.30% each (95% CI 96.73–97.77). Among open-weight vision- no longer a peripheral issue confined to specialized imaging systems, capable models, Kimi K2.5 achieved 94.52% (95% CI 93.74–95.20), followed because leading general-purpose foundation models are now natively by Qwen3-VL at 93.18% (95% CI 92.32–93.94) and Mistral Large 3 at multimodal and are increasingly evaluated for healthcare use. Yet most 91.33% (95% CI 90.39–92.19). Smaller vision-capable open-weight models, established medical benchmarks remain predominantly text-based, and including Gemma 3 and Ministral 3, showed lower overall performance and although recent multilingual multimodal medical-examination bench- are included in Fig. 1; full numerical results for all evaluated models are marks have begun to address this gap21,22, evidence from official examination provided in Suppl. Table S2. settings with corresponding human performance data remains limited. Moreover, multimodal evaluation requires separating text-only from Performance on the shared text-only subset across all models image-present items rather than relying on aggregate examination accuracy Because not all evaluated models supported image input, we next compared alone. As a result, it remains unclear how current models perform on official all models on the shared text-only subset (M1 n = 3227; M2 n = 3244), which medical examination items that include visual material alongside text, and provides the fairest common basis for cross-model comparison (Fig. 2A, B). whether image-present items reveal limitations that are obscured in pre- On M1 text-only items, Gemini 3.1 Pro achieved the highest accuracy dominantly text-based evaluation. at 99.63%, followed by GPT-5.4 at 99.50%, Claude Opus 4.6 at 99.26%, A further translational consideration concerns deployment under real- GLM-5 at 99.10%, Kimi K2.5 at 98.76%, DeepSeek V3.2-Thinking at world governance and data-protection constraints. Although the strongest 98.73%, and Qwen3-VL at 98.42%. Additional strong performance was multimodal performance is currently led by proprietary, API-hosted sys- observed for MiniMax M2.5 at 98.17% and gpt:oss 120B at 97.27%, whereas tems, their use may be restricted in healthcare environments where sensitive Mistral Large 3 and gpt:oss 20B reached 94.45% and 94.24%, respectively. medical information cannot readily be processed through external cloud On M2 text-only items, Gemini 3.1 Pro again ranked first at 98.86%, services. This is particularly relevant in Europe, where data-protection, followed by Claude Opus 4.6 at 98.15% and GPT-5.4 at 98.03%. The governance, and data-sovereignty requirements are often stringent. Locally strongest open-weight models on this subset were GLM-5 at 96.42%, deployable open-weight models, therefore, represent an important DeepSeek V3.2-Thinking at 95.84%, Kimi K2.5 at 95.75%, MiniMax M2.5 at npj Digital Medicine | (2026)9:670 2 https://doi.org/10.1038/s41746-026-03082-7 Article Fig. 1 | Overall performance on the full benchmark in vision-capable models. by item counts. The gray bar in each panel indicates mean student accuracy across Ranked accuracy for vision-capable models on the full benchmark, defined as all the corresponding item set. Full numerical results (k/n, accuracy, and 95% Wilson retained items, on (A) M1 and (B) M2, aggregated across examinations and weighted confidence intervals) are provided in Suppl. Table S2. Fig. 2 | Performance on the shared text-only subset across all models. Ranked text- size tier. The gray bar in each panel indicates mean student accuracy across the only accuracy for all evaluated models on (A) M1 and (B) M2, aggregated across corresponding text-only item set. Full numerical results (k/n, accuracy, and 95% examinations and weighted by item counts. Models are grouped by access type and Wilson confidence intervals) are provided in Suppl. Table S2. 95.01%, and Qwen3-VL at 94.73%. gpt:oss 120B and Mistral Large 3 reached 69.81% on image-present items (gap 28.61 points). Mistral Large 3 reached 93.74% and 93.16%, respectively, whereas gpt:oss 20B achieved 89.61% (Fig. 94.45% on text-only items but only 46.15% on image-present items (gap 2A, B; Suppl. Table S2). 48.30 points). Several large open-weight models also achieved very high accuracy on On M2, Gemini 3.1 Pro achieved 98.86% on text-only items and the shared text-only subset, particularly in M1. 94.44% on image-present items (gap 4.41 points). GPT-5.4 achieved 98.03% versus 92.59% (gap 5.43 points), and Claude Opus 4.6 achieved Modality-stratified performance and comparison with the human 98.15% versus 90.74% (gap 7.41 points). Among open-weight models, reference Kimi K2.5 reached 95.75% on text-only items and 85.65% on image- We next quantified performance separately on the shared text-only subset present items (gap 10.10 points), Qwen3-VL reached 94.73% versus and on image-present items for vision-capable models (Fig. 3A, B). Across 81.71% (gap 13.02 points), and Mistral Large 3 reached 93.16% versus the evaluated model inventory, accuracy on image-present items was con- 77.78% (gap 15.38 points) (Fig. 3A, B; Suppl. Tables S3 and S4). sistently lower than on text-only items. To contextualize these model-level gaps against the human On M1, Gemini 3.1 Pro achieved 99.63% on text-only items versus reference, we computed the corresponding mean student accuracies on 97.31% on image-present items, corresponding to a text–image gap of 2.32 the same item sets from the official item-level student correctness percentage points. GPT-5.4 achieved 99.50% versus 92.12% (gap 7.39 proportions. Mean student accuracy was 71.21% on M1 text-only items points), and Claude Opus 4.6 achieved 99.26% versus 82.50% (gap 16.76 versus 64.27% on M1 image-present items (gap 6.94 percentage points). Among open-weight models, Kimi K2.5 reached 98.76% on text- points), and 74.66% on M2 text-only items versus 70.74% on M2 only items but 84.04% on image-present items (gap 14.72 points), whereas image-present items (gap 3.92 points). Image-present items were Qwen3-VL showed a larger gap, with 98.42% on text-only items versus therefore also harder for human candidates than text-only items, npj Digital Medicine | (2026)9:670 3 https://doi.org/10.1038/s41746-026-03082-7 Article Fig. 3 | Modality-stratified performance reveals a persistent text–image gap. and grouped by access type and size tier. C, D Error-rate ratio between image- A, B Accuracy on the shared text-only subset versus the image-present subset for present and text-only items, defined as (1 − accimage)/(1 − acctext), for the same vision-capable models on (A) M1 and (B) M2, with the corresponding mean student models on (C) M1 and (D) M2. Values are shown on a logarithmic scale; the dashed accuracy shown as a reference. Each model is shown with paired text-only and reference line at 1 × corresponds to equal error rates. Full numerical results for image-present accuracies connected by a line; numerical labels indicate the absolute absolute accuracies are reported in Suppl. Tables S3 and S4. text–image gap in percentage points. Models are ordered consistently across panels although direct comparison of absolute percentage-point gaps is lim- Item-level correlation between human and model difficulty ited by floor and ceiling effects. We next assessed whether item-level model difficulty aligned with official We additionally quantified the modality-specific performance drop student difficulty on eligible items from the shared text-only subset. For M1, using the error-rate ratio, i.e. the ratio of the error rate on image-present item-level human difficulty and model difficulty showed a positive corre- items to the error rate on text-only items, which is invariant to the lation (ρ = 0.318, 95% CI 0.286–0.346; n = 3227 items). For M2, the asso- baseline accuracy level. For human candidates, the error-rate ratio was ciation was slightly stronger (ρ = 0.333, 95% CI 0.302–0.364; n = 3244 items). 1.24 × on M1 and 1.15 × on M2, indicating only a modest increase in error These findings indicate moderate, but incomplete, alignment between rate from text-only to image-present items. Across all evaluated vision- human and model difficulty at the item level (Suppl. Fig. S1). capable models, the error-rate ratio was substantially larger: on M1, it ranged from 7.2 × (Gemini 3.1 Pro) to 23.5 × (Claude Opus 4.6) for Difficulty-enriched text-only subsets reveal residual model dif- proprietary frontier models, from 9.7 × to 19.1 × for open-weight models ferences and human–model discordance ≥ 100B parameters, and from 2.9 × to 3.3 × for open-weight models < Given only moderate alignment between human and model difficulty, we 100B parameters. On M2, error-rate ratios ranged from 3.8 × to 5.0 × for next evaluated difficulty-enriched subsets of the shared text-only bench- proprietary frontier models, from 3.2 × to 3.5 × for open-weight models mark separately for M1 and M2 (Fig. 4). Human-hard-text was defined by ≥ 100B parameters, and from 1.8 × to 1.9 × for open-weight models < an official student p-value of ≤ 0.30, whereas model-hard-text comprised 100B parameters (Fig. 3C, D). items missed by at least two models from the top-tier comparison pool. Taken together, these results show persistent text–image performance In M1, the human-hard-text subset comprised 128 items, with a mean gaps across models. Image-present items were also more difficult for human student p-value of 0.221. On this subset, GPT-5.4 achieved 99.22%, Gemini candidates, but the modality penalty was disproportionately larger for 3.1 Pro 98.44%, and Claude Opus 4.6 and GLM-5 each 96.88%. In M2, the models: in absolute terms, all evaluated models except Gemini 3.1 Pro on human-hard-text subset comprised 197 items, with a mean student p-value M1 showed a larger drop than students, and in relative terms the multi- of 0.194. On this subset, Gemini 3.1 Pro achieved 97.97%, GPT-5.4 and plicative increase in error rate was far larger for models (≈ 1.2 × for students Claude Opus 4.6 each 95.94%, and GLM-5 92.89%. versus several-fold for most models). Because this ratio normalizes each By contrast, performance was more dispersed across models on the solver against its own text-only performance, this contrast indicates that model-hard-text subset. In M1, this subset comprised 93 items, with a mean image-present items pose an additional, model-specific difficulty beyond student p-value of 0.558. Gemini 3.1 Pro achieved 89.25%, GPT-5.4 84.95%, the general difficulty they also present to human candidates. The error-rate Claude Opus 4.6 80.65%, and GLM-5 75.27%. In M2, the model-hard-text ratio is nonetheless sensitive to baseline accuracy and should not be used to subset comprised 323 items, with a mean student p-value of 0.633. Gemini rank individual models against one another. 3.1 Pro reached 89.16%, followed by Claude Opus 4.6 at 82.97% and GPT- npj Digital Medicine | (2026)9:670 4 https://doi.org/10.1038/s41746-026-03082-7 Article Fig. 4 | Difficulty-enriched text-only subsets reveal residual performance dif- accuracy, error bars denote 95% Wilson confidence intervals, and the gray bar in ferences beyond aggregate benchmark accuracy. Performance of evaluated models each panel shows mean student accuracy for the corresponding subset. The model- on difficulty-enriched text-only subsets in M1 and M2. (A) M1 human-hard-text, hard-text subsets showed stronger descriptive separation among top-performing defined as items with student p ≤ 0.30; (B) M1 model-hard-text, defined as items on systems than the human-hard-text subsets. Full numerical results are reported in which at least two models from the top-tier comparison pool answered incorrectly; Suppl. Tables S5 and S6. (C) M2 human-hard-text; and (D) M2 model-hard-text. Bars indicate model 5.4 at 79.88%; among open-weight models, Kimi K2.5 achieved 60.06%, noting a recurring legal, administrative, or ethical component. By contrast, followed by Mistral Large 3 and Qwen3-VL at 53.87%, and GLM-5 at Set B was characterized as probing recognition of specific clinical or 52.01% (Fig. 4; Suppl. Tables S5 and S6). laboratory patterns and recall of narrowly defined entities, tests, or standard Overlap between the two subset definitions was limited, with 12 shared therapies. Because item content is protected, these characterizations are items in M1 and 40 in M2 (Suppl. Table S7). Cross-classification further reported as a convergent qualitative impression across models rather than as showed that most model-hard-text items were not human-hard-text (81/93 a quantified agreement metric (Suppl. Note 1). in M1 and 283/323 in M2), whereas most human-hard-text items were not model-hard-text (116/128 in M1 and 157/197 in M2). Exam-level robustness and empirical probes for direct To characterize what distinguishes these two off-diagonal cells, we contamination asked five of the evaluated language models to describe, in qualitative terms To assess robustness across individual examination administrations and to and without knowledge of the selection criteria, the recurring properties and empirically probe for direct benchmark contamination, we examined exam- the competence probed by a sample of items from each cell (Set A, items that wise accuracy across the 12 M1 and 12 M2 examinations on the shared text- were model-hard-text but not human-hard-text; Set B, items that were only subset. Reported pre-training cutoffs of the evaluated models span June human-hard-text but not model-hard-text; Suppl. Note 1). Across these five 2024 (gpt:oss 20B and gpt:oss 120B) to August 2025 (GPT-5.4), allowing the models, the two cells were distinguished primarily by the cognitive task type same exam-wise data to support both a robustness assessment and a tem- they probe. In M1, the models converged on contrasting the classification of porally structured contamination probe. described scenarios into concepts and terminology, with a marked psy- Across administrations, the strongest models showed consistently chosocial component (Set A), against the explanation of mechanisms and high performance with relatively narrow exam-wise ranges, whereas the application of scientific principles in the basic natural sciences (Set B); some lower-performing systems exhibited broader dispersion. On M1, here, the task-type contrast was additionally aligned with a behavioral- exam-wise text-only accuracy ranged from 98.92% to 100.00% for versus natural-science disciplinary tilt. In M2, an equally consistent task- Gemini 3.1 Pro, from 98.90% to 100.00% for GPT-5.4, and from 98.06% type axis emerged that cut across clinical specialties rather than tracking a to 100.00% for Claude Opus 4.6. Strong open-weight models also single subject area: the models broadly converged on characterizing Set A as remained comparatively stable, including Kimi K2.5 (97.70% to 99.61%), probing situation-appropriate, procedural, and norm-governed clinical Qwen3-VL (96.93% to 99.29%), GLM-5 (98.47% to 100.00%), and decision-making, namely selecting the appropriate next action within rea- DeepSeek V3.2-Thinking (97.32% to 99.64%). On M2, Gemini 3.1 Pro listic, management-oriented scenarios, with several models additionally ranged from 98.10% to 100.00%, GPT-5.4 from 96.54% to 100.00%, and npj Digital Medicine | (2026)9:670 5 https://doi.org/10.1038/s41746-026-03082-7 Article Claude Opus 4.6 from 96.93% to 100.00%. Among open-weight models, Discussion Kimi K2.5 ranged from 93.10% to 98.55%, Qwen3-VL from 93.54% to In this study, we benchmarked frontier proprietary and open-weight large 96.74%, GLM-5 from 94.64% to 98.85%, and DeepSeek V3.2-Thinking language models on official German medical licensing examination items from 93.92% to 99.64%. Full exam-wise heatmaps for all models are provided through direct cooperation with IMPP. Four main findings provided in Suppl. Fig. S2 (M1) and Suppl. Fig. S3 (M2). emerge. First, performance on the shared text-only subset approached To probe for direct contamination effects, we first examined whether ceiling levels in the strongest models, indicating that aggregate exam models showed an accuracy drop across the pre-training cutoff boundary. accuracy is becoming increasingly compressed as a discriminator of frontier Three evaluated open-weight models have reported cutoffs that fall capability. Second, this near-ceiling behavior did not extend to image- between the Spring 2024 (administered March–April 2024) and Fall 2024 present items, where the relative increase in error rate was several-fold larger (administered October 2024) examinations: gpt:oss 20B and gpt:oss 120B for models than for human candidates. Third, model difficulty aligned with (both June 2024) and Gemma 3 (August 2024). If direct memorization of human difficulty only partially: although item-level human and model pre-training-resident items contributed substantially to performance, difficulty were positively correlated, the association was only moderate, and these models should perform measurably worse on Fall 2024 than on human-hard-text and model-hard-text subsets showed limited overlap. Spring 2024 items. The observed Spring-to-Fall 2024 differences on the Fourth, several large open-weight models were already highly competitive shared text-only subset were − 0.22 percentage points (pp) (95% CI − on text-only items, and even compact local models exceeded average student 4.32, + 3.79) on M1 and − 1.03 pp (95% CI − 6.14, + 4.04) on M2 for performance, underscoring the practical relevance of privacy-preserving gpt:oss 20B; − 0.14 pp (95% CI − 3.57, + 3.21) on M1 and − 3.35 pp (95% deployment candidates. CI − 7.24, + 0.14) on M2 for gpt:oss 120B; and − 0.86 pp (95% CI − 7.84, A central implication of these results is that medical LLM bench- + 6.08) on M1 and − 2.80 pp (95% CI − 9.66, + 4.01) on M2 for Gemma marking is entering a regime in which aggregate text-only accuracy alone is 3. All six confidence intervals comfortably include zero. Differences of no longer sufficient. On this restricted-access benchmark, several top comparable magnitude were observed for proprietary frontier models models now operate at or near the upper end of the human performance whose cutoffs fall well after both 2024 administrations (e.g., GPT-5.4 range, while our within-family temporal analysis illustrates how quickly this M2 − 3.76 pp; Claude Opus 4.6 M2 − 2.62 pp; Gemini 3.1 Pro M2 − 2.26 compression can arise over only a few model generations. This broader pp), indicating that the modest Spring-to-Fall drops observed on M2 pattern is increasingly visible across established medical benchmarks, where reflect an examination-specific difficulty difference affecting all models recent reasoning-enabled systems have already reached near-saturation uniformly rather than a contamination-driven effect on cutoff-critical performance on several widely used evaluations9. Together, these findings models. On M1, Spring-to-Fall differences were centered near zero across suggest that once benchmark formats become highly familiar and increas- the inventory. This boundary-crossing test is necessarily restricted to the ingly solvable by frontier architectures, average performance metrics lose three open-weight models whose cutoff lies within our examination much of their ability to distinguish among top-tier systems. This is parti- range; for the remaining models, direct boundary testing is not feasible. cularly relevant beyond English-language settings, because recent multi- Second, we examined whether examinations from earlier years lingual and cross-national analyses suggest that medical LLM performance showed systematically higher accuracy than recent examinations, as can vary substantially across languages and examination contexts, with would be expected if older items had propagated through web-indexed lower accuracy often observed outside English benchmarks3,23. These materials over time. Per-model Spearman correlations between observations further underline the value of region-specific benchmark chronological examination index (1–12) and exam-wise accuracy on resources beyond English-language exam settings. the shared text-only subset were small in magnitude and not con- The difficulty-enriched subset analyses further illustrate this limitation sistently signed across the model inventory (Suppl. Table S8). For M1, of aggregate accuracy. The human-hard-text subset captured items on correlations ranged from − 0.586 (GPT-5.4) to + 0.357 (gpt:oss 20B) which student performance approached guessing level, yet the strongest with no systematic pattern by model class or reported cutoff. For M2, LLMs still answered these questions with very high accuracy. By contrast, correlations ranged from − 0.336 (Mistral Large 3) to + 0.322 (gpt:oss the model-hard-text subset showed greater descriptive separation among 120B), again with mixed signs across models with both early and late frontier models; because this subset was defined from the errors of the top- cutoffs. This second probe is necessarily a coarser signal than the tier model pool, however, this separation is partly circular by construction boundary-crossing test, but applies to the full model inventory and and should be read as descriptive rather than as an unbiased ranking of the tests an alternative contamination mechanism (gradual web propa- same models. Student performance on these items was only moderately gation of older material) rather than direct ingestion of specific reduced relative to the overall benchmark. These two subset types, therefore, item text. capture different notions of difficulty, and their limited overlap reinforces Taken together, exam-wise performance was robust across the the conclusion from the correlation analysis: models do not simply repro- 2019–2024 administrations, and the two contamination probes are con- duce the human hierarchy of difficult items. sistent with low direct benchmark contamination across the evaluated The exploratory LLM-assisted characterization of sampled off- model inventory. Neither approach rules out indirect exposure through diagonal items provides one possible qualitative explanation for this mis- paraphrased or restructured material in secondary educational platforms, match. Because this characterization was generated by judge models under and reliable corpus-level contamination detection would require access to neutral set labels, rather than by formal author-coded content analysis, it provider pre-training data, which is not publicly disclosed for most eval- should be interpreted cautiously. Still, the responses consistently pointed to uated models. differences in the type of task elicited. In M1, this task-type difference partly overlapped with domain differences between more classificatory or Secondary analyses terminology-oriented items and more mechanistic basic-science items. In Among the three evaluated base models without native thinking support, M2, where both off-diagonal item sets spanned clinically heterogeneous two-stage chain-of-thought prompting improved performance relative to subjects, the contrast was more clearly one of competence type: context- direct constrained answering in both M1 and M2, with the largest gains sensitive procedural and norm-governed clinical judgment on one side, and observed in Ministral 3 (Suppl. Fig. S4; Suppl. Tables S9 and S10). In a disease-specific recognition or retrieval of established biomedical knowledge separate within-family sub-analysis on a fixed recent text-only subset, on the other. A further contributing factor to the models’ relatively weaker performance across successive OpenAI model generations increased from performance on these procedural, norm-governed items (several of which 66.16% for GPT-3.5 Turbo to 98.30% for GPT-5.4, illustrating the rapid carry a legal, administrative, or ethical component) may be our deliberately compression of benchmark performance toward ceiling levels (Suppl. neutral primary prompt, which did not state that the items originated from Fig. S5). the German medical licensing examinations. Although the German npj Digital Medicine | (2026)9:670 6 https://doi.org/10.1038/s41746-026-03082-7 Article language and item content provided contextual cues, this neutral framing and some compact local models already exceeded average student perfor- may have left models without the specific jurisdictional context required for mance. This is particularly relevant for healthcare environments and guideline- or regulation-dependent decisions, and could be examined in medical universities in which external API use is constrained by privacy, future work by explicitly specifying the examination context in the prompt. governance, or data-sovereignty requirements regarding patient and stu- This task-type characterization supports the interpretation that model dif- dent data. Under such conditions, the practical question is not only which ficulty and student difficulty are related but not interchangeable signals. For model ranks first overall, but which classes of models are already strong medical educators developing AI-assisted tutoring systems, this means that enough to support local experimentation, educational use, or privacy- model-derived difficulty estimates should be calibrated against human sensitive clinical prototyping. The strong performance of compact models learner performance rather than treated as direct proxies for student further supports the feasibility of privacy-preserving, locally deployed AI difficulty. tutoring systems within university infrastructures. In educational settings, The most clinically relevant weakness revealed by this benchmark was these deployment choices are also governance choices. Institutions must the persistent gap between text-only and image-present performance. balance capability with privacy, data sovereignty, and academic integrity, Although the strongest proprietary multimodal models remained highly particularly when students use AI systems for preparation on clinically accurate overall, all vision-capable systems performed worse on image- framed materials. The growing strength of locally deployable open-weight present items than on text-only items, and the performance loss was models makes this trade-off increasingly actionable for universities and especially pronounced in several open-weight models. To assess whether examination-adjacent training environments. this reflected a model-specific limitation or a general property of image- At the same time, benchmark performance should not be conflated present items, we additionally compared the model-level error rates with the with translational readiness: recent commentary has emphasized that corresponding student-level reference. Image-present items were more medical LLM evaluations can create an overly optimistic picture when challenging for human candidates than text-only items, but this penalty was conclusions are drawn too directly from benchmark scores alone27. Con- more pronounced for models. In absolute terms, every evaluated vision- versely, more workflow-near studies are beginning to use more realistic capable model except Gemini 3.1 Pro on M1 showed a greater drop in endpoints, including prospective randomized evaluations of physician accuracy on image-present items than human candidates did, despite the performance on clinically framed patient-care tasks and clinician–AI col- models’ near-ceiling performance on text-only items. In relative terms, the laborative diagnostic workflows28,29. In this context, our benchmark is best difference was even more marked, with the increase in error rate being understood as a capability signal and comparative screening instrument approximately 1.2-fold for human candidates versus several-fold for most rather than a substitute for workflow-level clinical validation. The multi- models. Because this ratio normalizes each solver against its own text-only modal results further suggest that the remaining gap between open-weight baseline, it suggests that image-present items are not only more challenging and proprietary systems is currently more consequential in image- in general but also present an additional difficulty specific to the models. dependent settings than in text-only ones. However, the ratio is sensitive to baseline accuracy and should not be used to An additional strength of this study is that these results were obtained rank individual models’ vision capability. The strongest models (e.g., under a deliberately constrained evaluation design. Performance reflects Gemini 3.1 Pro) remained highly accurate on image-present items, whereas relatively raw model capability under standardized conditions: zero-shot several open-weight vision models declined substantially. This matters prompting, single-pass inference, no retrieval augmentation, no external because clinically relevant reasoning often requires integrating textual tools, no interactive decomposition, and no majority-vote or ensemble context with visual and other non-text modalities rather than relying on text strategy. This makes the benchmark conservative in one sense, because alone24,25. Our findings, therefore, suggest that text-only medical benchmark some models might achieve even higher scores in optimized agentic pipe- saturation should not be mistaken for broadly solved medical reasoning. As lines, but it also isolates intrinsic model capability more cleanly than heavily medical curricula increasingly emphasize integrative, multimodal clinical scaffolded evaluation setups. In this respect, the study does not primarily reasoning, the use of LLMs in educational settings should account for these show what can be achieved with complex engineering around a model; persistent visual deficits to avoid giving learners a misleading impression of rather, it shows how strong current systems already are on a restricted- model competence across modalities. This interpretation is consistent with access medical benchmark without such support. prior work showing that high multimodal multiple-choice accuracy can still A further strength is that the benchmark was constructed directly from conceal substantial weaknesses in image-grounded reasoning and rationale the authoritative primary examination dataset in cooperation with IMPP. quality26. A model may perform near ceiling on text-based exam questions The underlying examination items arise from an extensive expert-driven while still showing meaningful deficits when visual interpretation becomes development and quality-assurance process involving appointed subject- necessary for the correct answer. One plausible hypothesis for the generally matter experts, iterative revision, formal review before exam deployment, smaller text–image gap observed in most models in M2 than in M1 is that and post-exam review of items challenged by students or flagged by sta- M2 image-present items may remain partially answerable from the tistical irregularities30,31. This strengthens the benchmark’s value as an accompanying clinical vignette and textual context, whereas image-present evaluation resource derived from a highly curated national examination items in M1 may more often depend directly on the visual content itself, for process. example, in questions involving structural, schematic, or diagrammatic Several limitations should be considered. First, performance on a interpretation. Because we did not perform formal image-type or image- licensing-style multiple-choice examination is not equivalent to safe or necessity annotation, this explanation remains a hypothesis rather than an effective clinical deployment. Our benchmark does not evaluate open-ended established mechanism; testing it would require such annotation, which is clinical generation, longitudinal reasoning over evolving patient data, cali- beyond the scope of the present study. bration, uncertainty communication, or downstream effects on decision These findings also have translational implications. For medical edu- quality and patient outcomes. Similarly, the multimodal analysis remains cation and assessment, near-ceiling performance on text-only licensing- exam-based rather than workflow-based and should not be interpreted as a style items suggests that conventional multiple-choice formats may become comprehensive assessment of real-world clinical imaging use. Second, less informative as stand-alone indicators of competence in the AI era, although the restricted-access nature of the dataset reduces the risk of direct whereas multimodal and difficulty-enriched items may retain greater dis- benchmark contamination relative to public web-exposed resources, criminatory value. Accordingly, the educational challenge is not simply indirect exposure through licensed reproductions or secondary educational whether models can pass exams, but how assessment formats, AI literacy, materials cannot be fully excluded. Two empirical probes based on the and governance should evolve in response. Proprietary frontier systems temporal structure of the examination series and the reported pre-training retained the strongest overall and multimodal performance, but several large cutoffs of evaluated models did not reveal evidence of a substantial direct open-weight models came close to these systems on shared text-only tasks, contamination effect across the evaluated model inventory, but cannot rule npj Digital Medicine | (2026)9:670 7 https://doi.org/10.1038/s41746-026-03082-7 Article out indirect exposure through paraphrased or restructured secondary invalidated items were excluded before benchmarking and scoring, such materials; reliable corpus-level contamination detection would require that model performance was evaluated only against items retained in the access to pre-training data, which is not publicly disclosed for most eval- official IMPP answer key. Across the included examinations, the number of uated models. Third, the benchmark is intentionally national and language- officially invalidated items varied by exam; exact per-examination counts specific. This is a major strength for assessing regional generalization are reported in Suppl. Table S1. beyond English, US-centric benchmarks, but it also means that the findings After removal of officially invalidated items, the final benchmark should not be assumed to transfer unchanged to other healthcare systems or comprised 7485 evaluated items, including 3747 M1 items and 3738 M2 examination frameworks. Fourth, the difficulty-enriched subsets were items. Throughout this study, text-only denotes the shared analysis subset operationally defined to recover discrimination under near-ceiling condi- used for fair cross-model benchmarking, whereas image-present denotes tions, and alternative subset definitions could also be justified. The model- items that themselves included at least one image. In M1, all retained items hard-text subsets should therefore be interpreted as exploratory and without an image were included in the shared text-only subset, yielding 3227 descriptive difficulty-enriched subsets rather than externally validated text-only and 520 image-present items. In M2, text-only classification was benchmark strata. In addition, model rankings should be interpreted stricter because many items are embedded in sequentially linked case descriptively; exhaustive paired model-versus-model superiority testing was vignettes. Once an image occurred in any item within a vignette, that item not performed, and each model–item pair was evaluated once without and all subsequent items from the same vignette were excluded from the repeated sampling or majority voting. Small accuracy differences among shared text-only subset, even if those later items did not contain an image frontier systems in the near-ceiling regime may therefore be sensitive to themselves. This conservative rule was applied to avoid within-vignette stochastic response variability and should not be overinterpreted. Fifth, we leakage from earlier visual information. Accordingly, the shared M2 text- did not perform a formal subject-level, image-type, image-necessity, or only subset comprised 3244 items, while 432 items were image-present. A human-coded qualitative item-content analysis. The exploratory LLM- small additional group of 62 M2 items had no image of their own but was assisted characterization of human–model discrepancy items provides excluded from text-only analyses under this vignette-aware rule; these items qualitative context for the observed mismatch, but it should not be inter- remained part of full-benchmark analyses in vision-capable models. A per- preted as a validated qualitative coding study. It was based on sampled text- examination accounting of retained and excluded items is provided in only items, used judge models from the evaluated model pool, did not Suppl. Table S1. include human rater coding or inter-rater agreement, and the protected item M1 predominantly assesses foundational preclinical biomedical sci- texts cannot be reproduced for external item-level auditing. Accordingly, the ences, whereas M2 predominantly assesses clinically applied knowledge fine-grained item-level sources of the observed mismatch between human across medical specialties. In M2, approximately 55–65% of items occur in and model difficulty, as well as modality-specific subgroup effects among sequentially linked case vignettes, whereas approximately 35–45% are single image-present items, remain to be characterized in future work. Finally, the questions linked to an individual case context. All examination items fol- vignette-aware exclusion rule prevented within-vignette visual leakage but lowed a single-best-answer multiple-choice format with five response also restricted the prompt context provided to models: for items embedded options (A–E) and exactly one correct answer according to the official IMPP in case vignettes, models received the vignette text and preceding items but answer key. All content was in German and reflected German medical not subsequent items from the same vignette. Human candidates, by con- education standards and guideline-concordant practice. trast, see the entire vignette including all linked items, and may use infor- Items may include one or more figures, such as clinical photographs, mation from later items as cues for earlier ones, which may slightly radiological images, histopathology, electrocardiograms or scientific dia- underestimate model performance relative to the human reference for grams. For the present study, an item was classified as image-present if the these items. question as evaluated included at least one image. Overall, official German medical licensing examinations provide a stringent, restricted-access benchmark, with reduced direct public exposure Data access and benchmark leakage considerations and explicit contamination-probe analyses, for evaluating medical foun- This study was conducted in formal cooperation between the Institute for dation models in a non-English, deployment-relevant setting. On this Digital Medicine at Philipps University Marburg and IMPP, the rights- benchmark, frontier systems approached near-ceiling performance on the holding examination authority. Beyond access to the examination items shared text-only subset, whereas performance on image-present items themselves, this cooperation provided three advantages that were not remained consistently lower, indicating that multimodal medical reasoning available to earlier studies. First, we obtained the official item-level student is still substantially less mature than text-based exam performance. These difficulty indices (p-values), which are not available through the secondary findings suggest that aggregate benchmark accuracy alone is no longer providers and are required for the item-level human–model difficulty sufficient for frontier model evaluation. Future medical AI benchmarks alignment reported here. Second, we obtained the complete official item set should therefore be modality-stratified, difficulty-enriched, and explicitly together with the official post-examination invalidation decisions, so that designed for fair comparison across heterogeneous model classes, while each item could be scored strictly against the official answer key; we eval- translational assessment should increasingly consider the growing relevance uated all items and then excluded the officially invalidated ones from scoring of strong open-weight models for privacy-sensitive local deployment. (Suppl. Table S1). Third, direct provision enabled standardized, fully automated preprocessing across the entire 2019–2024 item set. By contrast, Methods earlier studies without official IMPP cooperation obtained items indirectly, Dataset and examination structure extracting them individually from secondary platforms, which is more We benchmarked LLM performance on official German medical licensing error-prone and typically limited to smaller, manually assembled subsets. examination items provided by IMPP, covering all spring and fall exam- The evaluated dataset is not publicly accessible in its benchmarked ination sessions from 2019 through 2024. The dataset comprised the first form. In the German medical education context, the principal secondary (M1) and second (M2) of the three German medical state examinations, distribution channel for IMPP items is commercial subscription-based yielding 12 M1 and 12 M2 examinations across the six-year study period. exam-preparation services that reproduce official examination items under Each examination contains 320 multiple-choice items, corresponding to licensing arrangements with IMPP, and access to these services requires a 7680 nominal item positions before post-examination exclusions. paid individual account. Because such items are not openly published on the Following each examination, candidates may formally challenge items web, the dataset is less exposed to direct inclusion in web-crawled pre- perceived as ambiguous, flawed or factually incorrect. Challenged items are training corpora than fully public medical question-answering benchmarks, reviewed by the responsible expert committees, and a small number of items which reduces the risk of direct benchmark contamination and provides a may subsequently be invalidated and removed from official scoring. All such more stringent, reduced-exposure evaluation setting. Partial overlap npj Digital Medicine | (2026)9:670 8 https://doi.org/10.1038/s41746-026-03082-7 Article Table 1 | Model inventory and capability annotation. Final Open-weight models were evaluated locally from publicly released evaluated model configurations included in this study weights using a standardized local inference pipeline. To capture a broad deployment-relevant range, open-weight models were stratified by publicly Model family Evaluated Vision Thinking Cutoff configuration reported nominal parameter scale into models with fewer than 100B parameters and models with at least 100B parameters. For mixture-of- Open-weight models ( < 100B parameters) experts systems, these values refer to the total nominal parameter count Gemma 3 27B Y N Aug 2024 reported by the model provider and do not imply that all parameters are Ministral 3 14B Y N n.d. active for any given token or inference step. This grouping was used as a gpt:oss 20B N Y Jun 2024 practical distinction between smaller locally deployable systems and Open-weight models (≥ 100B parameters) frontier-scale open-weight models rather than as a direct estimate of effective per-token compute. For local models that exposed an explicit native gpt:oss 120B N Y Jun 2024 thinking or reasoning mode, this mode was enabled whenever available and Qwen3-VL 235B Y Y n.d. configured to the highest supported setting. Open-weight base models MiniMax M2.5 229B N Y n.d. without native thinking support were additionally evaluated under a two- Mistral Large 3 675B Y N n.d. stage chain-of-thought (CoT) prompting condition, as described below. DeepSeek 685B N Y n.d. V3.2-Thinking Prompting strategies All evaluations were conducted using zero-shot prompting in German to GLM-5 744B N Y n.d. match the original examination language. Full prompt templates are pro- Kimi K2.5 1000B Y Y n.d. vided in the Supplementary Material. No external tools, retrieval augmen- Proprietary models (API-hosted) tation or task-specific prompt tuning were used. For multi-item case GPT-5.4 gpt-5.4-2026- Y Y Aug 2025 vignettes in M2, each item was presented together with its vignette text and 03-05 the question texts and answer options of any preceding items within the Gemini 3.1 Pro gemini-3.1-pro- Y Y Jan 2025 same vignette; official solutions, model-selected answers from preceding preview items, and subsequent items within the same vignette were not provided. Claude claude-opus-4-6 Y Y May 2025 We defined a primary prompting condition applied across all models and an Opus 4.6 additional chain-of-thought condition for base models without native Vision indicates native multimodal image input. Thinking indicates an explicit native reasoning thinking support. mode, enabled at the highest available setting when supported. Cutoff indicates the approximate Under the primary constrained zero-shot condition, all models were pre-training data cutoff as reported by the respective model provider in their official documentation; “n.d.” indicates that the provider did not publicly disclose a cutoff date in a verifiable form. instructed to return exclusively the single letter corresponding to the correct option (A–E), without additional explanation. This served as the main benchmarking condition throughout the study. through user-generated study materials cannot be fully excluded. To For base models without an explicit native thinking or reasoning mode, empirically assess the contribution of direct contamination effects, we we additionally evaluated a chain-of-thought (CoT) condition by appending additionally performed two complementary post-hoc analyses based on the the German instruction "Lass uns Schritt fuer Schritt vorgehen.” ("Let’s think temporal structure of the examination series and the reported pre-training step by step.”). To standardize final answer extraction, this condition was data cutoffs of the evaluated models. implemented as a two-stage inference procedure: in the first stage, the model generated an unconstrained reasoning response; in the second, the same Ethics model was prompted to map this response to a single option letter (A–E). This study used official examination materials and aggregated, fully anon- This condition was applied only to base models without native thinking ymized examination statistics provided by IMPP. No identifiable personal support. Models with native thinking support were not prompted in this data were accessed, and no participants were recruited or contacted. way, in order to avoid conflating built-in reasoning mechanisms with According to institutional requirements, this work did not require ethics externally elicited reasoning formats. committee approval or informed consent. Inference setup Model selection Benchmark evaluations were conducted between February and March 2026, We evaluated a broad set of LLMs spanning both proprietary, API-hosted using the model versions and configurations listed in Table 1. Unless systems and locally deployed open-weight models. The model inventory otherwise specified, models were evaluated using provider-default inference was designed to cover the major frontier systems available at the time of settings or the corresponding default local pipeline settings. No additional evaluation, together with widely used open-weight architectures across a decoding parameters were manually tuned or harmonized across models. broad range of parameter scales. This allowed comparison between pro- The only systematic configuration change applied across models was the prietary reference models and locally deployable alternatives under a shared activation of native thinking or reasoning modes at the highest available benchmarking framework. The full model inventory, grouped by access setting where supported. These provider-specific reasoning or thinking modality and size tier and annotated for vision capability and native settings should not be interpreted as equivalent inference-time compute thinking support, is provided in Table 1. budgets across systems. The resulting comparison is therefore a pragmatic Proprietary models were accessed via the respective vendor APIs. All evaluation of the evaluated model systems under standardized task prompts proprietary models included in this study supported multimodal image and commonly available default or maximum-supported reasoning con- input and an explicit native thinking or reasoning mode. Where exposed by figurations, rather than a compute-matched comparison of underlying the provider, this mode was enabled at the highest available setting. Exact model architectures. model identifiers, including preview or date-stamped snapshot versions For vision-capable models, images were provided in the native input where applicable, are reported in Table 1 for transparency and version format supported by the respective API or local inference pipeline. For traceability. Directly comparable parameter counts were not used for pro- OpenAI and Anthropic models, images were passed as base64-encoded prietary models because these are generally not publicly disclosed in a inputs; for Google Gemini, as PIL image objects; and for locally deployed standardized and verifiable form by providers. open-weight vision models, as the original image files. Images were provided npj Digital Medicine | (2026)9:670 9 https://doi.org/10.1038/s41746-026-03082-7 Article in their original PNG or JPEG format. No auxiliary OCR, image captioning, was defined as one minus the mean item-level correctness across the or manual image-to-text conversion was introduced. finalized text-only model pool used for the alignment analysis. Associa- Text-only models did not receive image inputs under any condition. tions between human difficulty and model difficulty were assessed Accordingly, direct comparisons across the full model inventory were separately for M1 and M2 using Spearman correlation on eligible items restricted to the shared text-only subset defined in Section Dataset and from the shared text-only subset with valid official p-values, with 95% examination structure. Analyses involving all retained items, including confidence intervals estimated by non-parametric bootstrap resampling image-present items, were restricted to vision-capable models. of items (10,000 resamples). Each model–item pair was evaluated once, without repeated sampling, To increase discriminatory power beyond near-ceiling overall accuracy, ensembling, or majority voting. Requests were retried up to three times only we additionally defined difficulty-enriched text-only subsets and analyzed in the event of transient API or transport errors or empty responses; retries them separately for M1 and M2. The human-hard-text subset comprised were not triggered by answer content, formatting deviations, or incorrect text-only items with an official student p-value of ≤ 0.30. The model-hard- predictions. Across the final benchmark runs, every model–item pair pro- text subset comprised text-only items on which at least two models from a duced a non-empty generation that was not an API error, transport error, or top-tier comparison pool answered incorrectly. This pool included all pro- content-policy refusal. No manual post-processing was performed beyond prietary frontier models and all open-weight models in the ≥100B size tier. deterministic answer parsing. These subsets were designed to capture complementary forms of difficulty: items that were exceptionally difficult for students and items that remained Answer extraction and scoring discriminative at the model frontier. Subset performance was summarized Model outputs were normalized to uppercase and parsed using a determi- using accuracy and Wilson 95% confidence intervals. Overlap between nistic regular-expression-based procedure to extract a single response difficulty-enriched subsets was assessed descriptively. option (A–E). Specifically, responses were scanned for the first occurrence of As an exploratory, non-reproductive probe of the human–model diffi- a valid option letter from A to E, while surrounding characters such as culty mismatch, we additionally performed an LLM-assisted qualitative parentheses or other formatting symbols were ignored. This procedure characterization of the two off-diagonal cells of the cross-classification of therefore accommodated minor deviations from the constrained output human-hard-text and model-hard-text items (Suppl. Table S7). Within each format, including outputs such as A), (A) or *A*. Parsing was applied cell, items were ranked continuously: items that were model-hard-text but not identically across all models and prompting conditions, and no manual human-hard-text ("Set A”) by p × (1 − a), and items that were human-hard- correction or adjudication of model outputs was performed. text but not model-hard-text ("Set B”) by (1 − p) × a, where p is the official As a robustness fallback, responses in which no valid option letter from student correctness proportion and a is the mean accuracy of the top-tier A to E could be detected anywhere in the generation were scored as incorrect. comparison pool. The 10 highest-ranked items from each cell were retained. In practice, this fallback did not apply to any response in the final benchmark The two anonymized item sets were presented under neutral labels (Set A and runs. Accuracy was defined as the proportion of evaluated items for which Set B) to five of the evaluated models—GPT-5.4, Gemini 3.1 Pro, Claude Opus the extracted model response matched the official IMPP answer key. 4.6, GLM-5, and DeepSeek V3.2-Thinking—with no information about the underlying selection pattern and no predefined category scheme, so that Statistical analysis categories had to emerge from the items themselves. Each model character- The primary endpoint was accuracy, defined as the proportion of correctly ized the recurring properties and the competence probed by the questions in answered items among all evaluated items. Accuracy was reported for the each set, and then contrasted the two sets. This procedure is exploratory and full benchmark, for the shared text-only subset, and for the image-present descriptive; it does not constitute a validated benchmark stratification or a subset, as applicable. Because image inputs were available only for vision- formal multi-rater content annotation with inter-rater agreement. The capable models, direct comparisons across all models were restricted to the prompt and the per-model summaries are reported in Suppl. Note 1. shared text-only subset. This subset comprised 3227 items in M1 and 3244 As a secondary within-family analysis, we assessed temporal perfor- items in M2, with the stricter M2 definition reflecting the vignette-aware mance trends across successive OpenAI model generations under the same exclusion rule described in Section Dataset and examination structure. zero-shot evaluation protocol. For comparability across generations, Analyses involving image-present items were restricted to vision-capable reasoning-enabled OpenAI models in this secondary within-family analysis models and used all retained image-present items (M1 n = 520; M2 n = 432). were evaluated using medium reasoning settings where applicable, rather In M2, the shared text-only subset and the image-present subset therefore than the highest available reasoning setting used in the main benchmark do not exhaust the full benchmark, because a small number of no-image analyses. Unless otherwise stated, analyses were performed in Python items were excluded from text-only analyses under the vignette-aware rule (version 3.13.7) using standard libraries, including pandas and numpy. while remaining part of the full-benchmark evaluation. Accuracy estimates were aggregated across examinations and weighted Data availability by item counts, corresponding to the total number of correct responses divided The datasets analyzed in this study were provided by IMPP and are not by the total number of evaluated items. We report two-sided 95% confidence publicly available. They contain official examination materials, including intervals for accuracy using the Wilson score method. Mean student accuracy item texts, answer options, answer keys, images, item identifiers, and psy- shown as a reference in figures was computed from the official item-level chometric metadata, which are subject to legal, contractual, copyright, and student correctness proportions over the corresponding item set. These official examination-security restrictions. Public release, including release in item- student performance statistics were derived from 119,878 examination sittings level or pseudonymized form, could enable reconstruction or redistribution across the 24 included examination administrations. of protected examination content. Access to the underlying examination To quantify modality-specific performance differences in vision- materials is therefore controlled by the rights holder and cannot be granted capable models, we computed two complementary metrics. The text–image by the authors. To support transparency, the manuscript and Supplemen- gap was defined as the absolute difference in accuracy between text-only and tary Information report aggregate dataset accounting, exclusion counts, image-present items, in percentage points. The error-rate ratio was defined modality counts, model-level results, confidence intervals, and the defini- as the ratio of the error rate on image-present items to the error rate on text- tions of all analysis subsets used in the study. only items, (1 − accimage)/(1 − acctext). To compare model difficulty with human difficulty at the item level, Code availability we used the official IMPP item difficulty index (p-value), defined as the The analysis code is not publicly available because it is tightly coupled to proportion of students answering an item correctly. Human difficulty was restricted-access examination files, protected item identifiers, answer keys, operationalized as 1 − p. For this correlation analysis, model difficulty image paths, and preprocessing steps specific to the IMPP source material. npj Digital Medicine | (2026)9:670 10 https://doi.org/10.1038/s41746-026-03082-7 Article Releasing the code in executable form would not permit independent 18. Geissler, M. E. et al. The Performance of Artificial Intelligence on a reproduction without access to the restricted datasets and could disclose National Medical Licensing Examination. Dtsch Arztebl Int. 121, details of the protected examination-material processing workflow. The 888–889 (2024). computational procedures are described in the Methods and Supplementary 19. Madrid, J. et al. Performance of Plug-In Augmented ChatGPT and Its Information, including item exclusion rules, text-only and image-present Ability to Quantify Uncertainty: Simulation Study on the German subset definitions, the M2 vignette-aware rule, prompt construction, answer Medical Board Examination. JMIR Med. Educ. 11, e58375 (2025). extraction, scoring, confidence-interval calculation, contamination-probe 20. Moor, M. et al. Foundation models for generalist medical artificial analyses, and model grouping. Additional clarification of these procedures intelligence. Nature 616, 259–265 (2023). can be provided by the corresponding author upon reasonable request, 21. Matos, J. et al. WorldMedQA-V: A Multilingual, Multimodal Medical subject to legal, contractual, copyright, and examination-security restrictions. Examination Dataset for Multimodal Language Models Evaluation. Findings of the Association for Computational Linguistics: NAACL Received: 23 April 2026; Accepted: 26 July 2026; 2025 7218–7231 (2025). 22. Choi, B., Bae, S., Kweon, S. & Choi, E. KorMedMCQA-V: A Multimodal Benchmark for Evaluating Vision-Language Models on the Korean References Medical Licensing Examination. https://doi.org/10.48550/arXiv.2602. 1. Kung, T. H. et al. Performance of ChatGPT on USMLE: Potential for 13650 (2026). AI-assisted medical education using large language models. PLOS 23. Alonso, I., Oronoz, M. & Agerri, R. MedExpQA: Multilingual Digit Health 2, e0000198 (2023). benchmarking of Large Language Models for Medical Question 2. Nori, H., King, N., McKinney, S. M., Carignan, D. & Horvitz, E. Answering. Artif. Intell. Med. 155, 102938 (2024). Capabilities of GPT-4 on Medical Challenge Problems. https://doi. 24. AlSaad, R. et al. Multimodal Large Language Models in Health Care: org/10.48550/arXiv.2303.13375 (2023). Applications, Challenges, and Future Outlook. J. Med. Internet Res 3. Zong, H. et al. Large Language Models in Worldwide Medical Exams: 26, e59505 (2024). Platform Development and Comprehensive Analysis. J. Med. Internet 25. Rao, V. M. et al. Multimodal generative AI for medical image Res. 26, e66114 (2024). interpretation. Nature 639, 888–896 (2025). 4. Van Veen, D. et al. Adapted large language models can outperform 26. Jin, Q. et al. Hidden flaws behind expert-level accuracy of multimodal medical experts in clinical text summarization. Nat. Med. 30, GPT-4 vision in medicine. npj Digital Med. 7, 190 (2024). 1134–1142 (2024). 27. Agrawal, M., Chen, I. Y., Gulamali, F. & Joshi, S. The evaluation illusion 5. Singhal, K. et al. Toward expert-level medical question answering with of large language models in medicine. npj Digital Med. 8, 600 (2025). large language models. Nat. Med. 31, 943–950 (2025). 28. Goh, E. et al. GPT-4 assistance for improvement of physician 6. Phan, L. et al. Humanity’s Last Exam https://doi.org/10.48550/arXiv. performance on patient care tasks: a randomized controlled trial. Nat. 2501.14249 (2025). Med. 31, 1233–1238 (2025). 7. Snell, C., Lee, J., Xu, K. & Kumar, A. Scaling LLM Test-Time Compute 29. Everett, S. S. et al. From tool to teammate in a randomized controlled Optimally can be More Effective than Scaling Model Parameters trial of clinician-AI collaborative workflows for diagnosis. npj Digit. https://doi.org/10.48550/arXiv.2408.03314 (2024). Med. 9, 409 (2026). 8. Open Life Science AI. Open Medical LLM Leaderboard. https:// 30. Institut für medizinische und pharmazeutische Prüfungsfragen huggingface.co/spaces/openlifescienceai/open_medical_llm_ (IMPP). Fachbereich Medizin. https://www.impp.de/ueber-uns/ leaderboard. Accessed March 23, 2026. organisation/fachbereiche/medizin.html. Accessed March 23, 2026. 9. Nori, H. et al. From Medprompt to o1: Exploration of Run-Time Strategies 31. Institut für medizinische und pharmazeutische Prüfungsfragen for Medical Challenge Problems and Beyond https://doi.org/10.48550/ (IMPP). Sachverständige. https://www.impp.de/informationen/ arXiv.2411.03590 (2024). partner/sachverstaendige.html. Accessed March 23, 2026. 10. Gallifant, J. et al. Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks 2406.12066 (2024). Acknowledgements 11. Ravaut, M. et al. A Comprehensive Survey of Contamination No funding was received for this research. Detection Methods in Large Language Models https://doi.org/10. 48550/arXiv.2404.00699 (2024). Author contributions 12. Zuo, Y. et al. MedXpertQA: Benchmarking Expert-Level Medical L.C. conceived the study, performed the experiments and analyses, and Reasoning and Understanding https://doi.org/10.48550/arXiv.2501. drafted the manuscript. J.K. contributed to study design, interpretation of the 18362 (2025) results, and critical revision of the manuscript. V.S., A.O., and J.C.B. 13. Yang, S., Chiang, W.-L., Zheng, L., Gonzalez, J. E. & Stoica, I. Rethinking contributed to data provision, interpretation of the findings, and critical Benchmark and Contamination for Language Models with Rephrased revision of the manuscript. S.K. supervised the study and critically revised Samples https://doi.org/10.48550/arXiv.2311.04850 (2023). the manuscript. All authors approved the final manuscript. 14. Jung, L. B. et al. ChatGPT Passes German State Examination in Medicine With Picture Questions Omitted. Dtsch Arztebl Int. 120, Funding 373–374 (2023). Open Access funding enabled and organized by Projekt DEAL. 15. Roos, J., Kasapovic, A., Jansen, T. & Kaczmarczyk, R. Artificial Intelligence in Medical Education: Comparative Analysis of ChatGPT, Bing, and Competing interests Medical Students in Germany. JMIR Med. Educ. 9, e46482 (2023). The authors declare no competing interests. 16. Meyer, A., Riese, J. & Streichert, T. Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written Additional information German Medical Licensing Examination: Observational Study. JMIR Supplementary information The online version contains Med. Educ. 10, e50965 (2024). supplementary material available at 17. Humbsch, P., Horn, E., Bohm, K. & Gintrowicz, R. ChatGPT im Einsatz https://doi.org/10.1038/s41746-026-03082-7. für technology-enhanced learning in Anästhesiologie und Notfallmedizin und potenzielle klinische Anwendung von KI- Correspondence and requests for materials should be addressed to Sprachmodellen. Die Anaesthesiologie 73, 324–335 (2024). Lasse Cirkel. npj Digital Medicine | (2026)9:670 11 https://doi.org/10.1038/s41746-026-03082-7 Article Reprints and permissions information is available at http://www.nature.com/reprints Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. © The Author(s) 2026 npj Digital Medicine | (2026)9:670 12