From simulation to pedagogy: multi-model and randomized evaluation of structured AI standardized patients

npj | digital medicine Article Published in partnership with Seoul National University Bundang Hospital Article in Press https://doi.org/10.1038/s41746-026-03372-0 From simulation to pedagogy: multi-model and randomized evaluation of structured AI standardized patients Check for updates 1 2 1 1 1 1 1,3 1,4 Ping Wu , Yu Han , Jing Zhang , Yunqi Li , Mengna Jiang , Xinyu Lu , Haibin Zhang , Danyang Xu , Hao Ming1,5, Lihong Wang1,6 & Qingping Wen1 Standardized patients (SPs) are central to clinical communication training but remain constrained by cost, scalability, and reliance on trained actors. We present AI standardized patients (AI-SPs), large 1234567890():,; 1234567890():,; language model–driven patient simulators built on a three-layer information architecture that adaptively modulates information disclosure according to learner skill. In Phase 1, blinded expert assessment of 350 simulated consultations generated by five frontier LLMs showed that, under a common architecture, learner skill level accounted for more of the observed performance variation than model choice (η2 = 0.31 vs. 0.06); whether this reflects the architecture itself cannot be determined without comparing models with and without it. In Phase 1b, analysis of 155 live student consultations identified systematic failures to elicit safety-critical information, enabling automated curriculum diagnostics without expert observation. In a Phase 2 three-arm pilot randomized study (n = 58), OSCE performance gains with AI-SP training did not differ significantly from those with human SP training; the trial was designed to estimate effect sizes for a future confirmatory study, not to demonstrate equivalence. Self-efficacy gains were numerically largest with AI-SP training, but this exploratory difference did not survive correction for multiple comparisons. These findings suggest that architecture-driven AI-SPs could provide a scalable, model-portable approach to clinical communication training. Clinical communication skills are fundamental to patient safety, yet their have advanced AI-based patient simulation: Luo et al.16 demonstrated LLM- development requires deliberate practice with realistic patient encounters1,2. based history-taking training in ophthalmology; Lavigne et al.17 reported a Standardized patients (SPs) remain the gold standard3,4, but their depen- two-arm RCT showing AI-simulated clinical encounters improved OSCE dence on trained actors, scheduling coordination, and faculty supervision performance over standard training; and Yu et al.18 developed an LLM- creates resource barriers that constrain training frequency, particularly in based simulated patient agent validated with medical students. However, specialties such as anaesthesiology, where preoperative consultations these studies share important limitations: all used a single LLM platform demand integration of history-taking, risk communication, and emotional (raising generalisability concerns), none compared AI-SP directly against support within high-stakes, time-pressured encounters5,6. human SP practice (the clinical gold standard), and none employed a Large language models (LLMs) offer a potential solution. Current structured pedagogical framework governing information disclosure. models demonstrate clinical knowledge7,8, conversational fluency9, and the Creating educationally effective AI-SPs requires more than conversational capacity to adopt complex personas10, capabilities that surpass prior virtual fluency; it demands pedagogically controlled information disclosure that patient systems built on branching scripts or constrained natural language rewards skilled interviewing, emotional responses calibrated to commu- processing11,12. Early work has shown promise for generative AI in perso- nication quality, and clinical consistency across conversational turns19. nalised medical education13 and clinical communication14,15. Recent studies 1 Department of Anesthesiology, The First Affiliated Hospital of Dalian Medical University, Dalian, 116000, China. 2Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxfordshire, UK. 3Department of Anesthesiology, First Hospital of Shanxi Medical University, Taiyuan, 030001, China. 4 Department of Anesthesiology, Affiliated Hospital of Chifeng University, Chifeng, 024000, China. 5Department of Anesthesiology, Shandong Second Provincial General Hospital, Jinan, 250000, China. 6Department of Anesthesiology, Chifeng Obstetrics and Gynecology Hospital, Chifeng, 024000, China. e-mail: wenqp@dmu.edu.cn npj Digital Medicine Article in Press https://doi.org/10.1038/s41746-026-03372-0 Article Here, we introduce a three-layer information architecture for AI-SP pre-specified educational viability threshold, while DeepSeek-R1 and GPT- design that addresses this gap. Surface information (Layer 1) is volunteered 4o fell below it. Post-hoc comparisons confirmed that the top-tier models spontaneously; prompted information (Layer 2) is disclosed only upon significantly outperformed the lower tier (Bonferroni-corrected p < 0.025), direct questioning; and hidden information (Layer 3) is withheld, not while within-tier differences were non-significant. through deliberate concealment but because the simulated patient lacks the The key finding was that student skill level, rather than model choice, health literacy to recognise its clinical relevance, requiring targeted, empa- drove fidelity variation. Two-way ANOVA confirmed that student level thetic questioning to elicit (Fig. 1a). This design mirrors documented patient accounted for five-fold more variance than model identity (η2 = 0.31 vs behaviour in which critical clinical details emerge only through skilled 0.06), with no significant interaction (p < 0.001; Table 2; Fig. 3b). The largest communication20, and operationalises cognitive apprenticeship theory21 by differentiation occurred in educational value and communication natural- generating a calibrated gradient between what novice and competent stu- ness, and the competent–novice gap was consistent across all six dimensions dents can discover. Crucially, the hidden-information discovery rate pro- and all five models (Supplementary Fig. 2). vides an objective, automated metric of clinical communication skill without This gradient extended to hidden information disclosure (Table 3): requiring real-time expert observation, enabling scalable assessment competent students discovered all 26 hidden Layer 3 items, while novice alongside scalable practice. The iceberg metaphor (Supplementary Fig. 1) students discovered only three, none of which were safety-critical. Missed further visualises this graduated disclosure: Layer 1 sits above the waterline items included self-discontinued clopidogrel creating stent thrombosis risk accessible to all students, while Layer 3 resides in deep water, discoverable (S3), unreported ephedrine-containing supplements (S4), and saffron only through empathetic probing that builds sufficient patient trust. capsules posing serotonin syndrome risk with concurrent sertraline (S5). We evaluate this approach through a three-phase progressive valida- Fidelity was robust across scenario complexity (Fig. 3c). Top- tion pipeline targeting preoperative anaesthesia consultations (Fig. 1d–j). performing models maintained scores near the viability threshold even This pipeline follows a deliberate escalation logic: deploying an AI training for high-complexity scenarios, with decrements of < 1 point from low to system in a clinical education setting without first establishing its fidelity and high complexity. GPT-4o showed the largest complexity-related decline ecological validity would be methodologically premature and ethically (change score = 1.2). questionable. Each phase, therefore, addresses a prerequisite question that must be answered before the next phase can be justified. Phase 1 addresses Phase 1b: ecological validation through live student interactions construct validity: can AI-SPs produce educationally adequate simulations? This phase asked whether the Phase 1 findings hold when real students Seven anaesthesiology educators blindly evaluated dialogues from five interact freely with AI-SPs, and whether the system can generate actionable frontier LLMs (GPT-4o, Claude 4.5 Sonnet, Gemini 2.5 Flash, Qwen-2.5 curriculum diagnostic data. To assess whether scripted-dialogue findings Plus, DeepSeek-R1) across five clinical scenarios at two student skill levels generalise to authentic interactions, 31 medical students each completed all (Competent, Novice), yielding 350 assessments. Phase 1b addresses ecolo- five scenarios, generating 155 live consultations. Sessions exhibited features gical validity: do scripted-dialogue findings generalise to authentic student absent from scripted dialogues, including spontaneous follow-up questions, interactions? Thirty-one students completed 155 real-time consultations to unexpected tangents, premature topic transitions, and natural commu- characterise interaction quality and per-item discovery patterns under nication errors. uncontrolled conditions. Phase 2 addresses training efficacy: does AI-SP The overall hidden-information discovery rate was 65.6%, with sub- practice improve clinical skills comparably to human SP practice? A three- stantial inter-scenario variation (Table 4): S2 (thyroidectomy) achieved the arm pilot RCT (n = 58) compared AI-SP, human SP, and standard curri- highest rate and S4 (lumbar disc) the lowest, consistent with the difficulty of culum controls on OSCE performance and self-efficacy. eliciting information that patients minimise. Translating the three-layer architecture into a system that functions Per-item analysis revealed wide variation in discovery rates (Fig. 4). reliably across multiple LLMs required a structured prompt engineering Items related to common clinical concerns (S2) were near-universally dis- process (Fig. 2). Over four iterative development rounds with expert review, covered, while the hardest items clustered in S4 (where the patient mini- a clinician–engineer team identified and resolved recurring failure modes, mised substance use), with concussion history proving the most elusive. including role boundary violations, premature information disclosure, Student-level variation was similarly wide, with a clear separation between affective flattening, and stylistic artefacts, producing a final six-section top- and bottom-quartile performers. prompt structure that is identical across all five LLMs tested. A post-training questionnaire indicated high overall satisfaction (Table 5; Cronbach’s α = 0.85). Students rated learning effectiveness and Results communication skill improvement most highly, while their understanding Results are presented following the progressive validation sequence: first of the three-layer architecture received the lowest score, suggesting that the establishing that AI-SPs can produce educationally adequate simulations pedagogical mechanism operated effectively without being transparent to (Phase 1), then confirming these findings generalise to authentic student learners. interactions (Phase 1b), and finally testing whether AI-SP practice translates to measurable training outcomes (Phase 2). Because this journal presents Phase 2: randomized controlled trial Methods after Results, we summarise the design briefly here for orientation: This phase asked whether AI-SP training produces skill gains comparable to Phase 1 was a blinded expert evaluation of scripted dialogues from five human SP practice, and whether it confers any additional benefits. Sixty-one hosted LLMs (seven evaluators; 350 assessments); Phase 1b was a live single- students were randomized via sex-stratified block randomization; three model study of 31 students (155 consultations); and Phase 2 was a three-arm withdrew before training, yielding 58 analysed participants across three randomized educational study (n = 58: AI-SP, human SP, standard curri- arms (Fig. 5). Groups were comparable at baseline on all measures (Table 6). culum) with pre– and post-training OSCE and self-efficacy assessment. Full Inter-rater reliability differed markedly by scoring instrument: the participant, instrument, and analysis details are given in Methods. binary checklist achieved good reliability (ICC = 0.72) while the global rating scale showed poor reliability (ICC = 0.15; Fig. 6). Accordingly, the checklist Phase 1: expert validation of AI-SP fidelity score was designated as the primary outcome, with the total score reported This phase asked whether AI-SPs can produce educationally viable simu- as secondary (full ICC values with 95% CIs in Supplementary Tables 3–6). lations, and whether fidelity depends on the LLM or on the pedagogical For the primary outcome, the OSCE checklist score, all three groups architecture. Blinded expert evaluation revealed significant between-model improved significantly from baseline, with no significant between-group differences (p < 0.001), with a two-tier structure (Fig. 3a; Table 1; per- difference (Table 6; Fig. 6a; per-scenario score changes in Supplementary scenario and per-evaluator scores in Supplementary Tables 1 and 2): three Table 7 and Supplementary Fig. 3). The AI-SP group showed the numeri- models (Qwen-2.5 Plus, Claude 4.5 Sonnet, Gemini 2.5 Flash) exceeded the cally largest checklist improvement, but the effect size versus human SP was npj Digital Medicine Article in Press https://doi.org/10.1038/s41746-026-03372-0 Article Fig. 1 | AI standardized patient system design and three-phase validation over- f Phase 1b (student interaction study): students systematically fail to elicit clinically view. a Three-layer information architecture: Layer 1 (surface) is volunteered critical hidden information. g Phase 1b: information discovery declines with spontaneously, Layer 2 (prompted) requires direct questioning, and Layer 3 (hid- increasing hiddenness. h Phase 2 (randomized controlled trial): three-arm design den) requires empathetic probing to unlock safety-critical information. b Five comparing AI-SP, human SP, and control arms. i Phase 2: blinded scoring across five preoperative scenarios spanning low to high complexity, each containing safety- standardized scenarios. j Phase 2: no significant difference in objective skill gain critical hidden items. c Unified structured prompt enabling consistent deployment (p = 0.483) while AI-SP training significantly improves self-efficacy (p = 0.034, across five large language models. d Phase 1 (expert evaluation of AI-SP dialogues): d = 0.62). AI-SP, AI standardized patient; SP, standardized patient; OSCE, objective blinded evaluation of 350 dialogues by seven educators. e Phase 1: pedagogical structured clinical examination. architecture, rather than model choice, drives performance (η2 = 0.31 vs 0.06). npj Digital Medicine Article in Press https://doi.org/10.1038/s41746-026-03372-0 Article Fig. 2 | AI standardized patient prompt engineering framework. Starting from a where the LLM offered unsolicited medical advice (behavioural constraints base LLM, four iterative development rounds addressed distinct failure modes: restricting the model to the patient role). Expert review after each round refined the verbose and formal patient responses (persona calibration), premature disclosure of prompt. The final system prompt comprises six structured sections (identity, Layers safety-critical Layer 3 information (gated release with explicit trigger conditions), 1–3 with trigger conditions, emotional model, and safety rules), averaging 78 lines flat emotional affect regardless of interviewer quality (conditional emotional model per scenario, deployed identically across all five LLMs. linking empathy to trust and information access), and role boundary violations small. The control group’s comparable improvement suggests a substantial where voice-based interaction has an inherent advantage. That the two testing effect from repeated OSCE exposure. ANCOVA likewise showed no modalities achieved near-identical satisfaction despite these complementary significant between-group difference after adjusting for baseline scores, strengths suggests students valued different aspects of each format. Both which alone explained nearly half of post-training variance, leaving limited arms scored above 3.8/5.0 on all dimensions, with high willingness to residual variance for training-specific effects. continue using AI-SP and recommend it to peers, indicating acceptability The secondary outcome, the total OSCE score (checklist + global for sustained deployment. rating), showed a consistent pattern: all groups improved substantially with no significant between-group difference (Fig. 6c; Table 6), paralleling the Discussion checklist finding with a medium AI-SP versus human SP effect size. This study shows that structured AI standardized patients, governed by a Self-efficacy improved within all three groups (Wilcoxon signed-rank three-layer information architecture, can achieve expert-validated fidelity p < 0.01 in each arm). The between-arm difference in self-efficacy change and produce meaningful diagnostic data about student clinical commu- was not significant on the omnibus test (Kruskal–Wallis p = 0.064; Table 6). nication skills. When used for training in a pilot trial, they yielded OSCE In exploratory pairwise comparisons, the AI-SP arm’s gain exceeded that of gains that did not differ significantly from those after human SP practice or controls (+10.2 vs +6.1; unadjusted p = 0.034, d = 0.62) and, less clearly, that test-retest learning alone, together with an exploratory signal of larger self- of the human SP arm (+10.2 vs +6.8; unadjusted p = 0.069, d = 0.51); neither efficacy gains in the AI-SP arm. Three findings merit particular discussion. difference remained significant after Holm correction (adjusted p = 0.103 The most consequential finding is that the variance attributable to and 0.138, respectively). The AI-SP arm also started from the lowest baseline student skill level (η2 = 0.31) was five-fold larger than that attributable to (Table 6). Individual trajectories (Fig. 7a) show nearly uniform upward model choice (η2 = 0.06). This is consistent with the three-layer information shifts among AI-SP students, whereas the human SP and control groups architecture, rather than the underlying LLM, being a primary driver of were more variable, including several students whose self-efficacy declined, pedagogical differentiation; we note, however, that because the same partly reflecting differences in baseline. Per-item analysis (Supplementary architecture was applied to every model, our design cannot fully separate the Table 8; Supplementary Fig. 4) showed the largest numerical AI-SP architecture’s contribution from that of prompt quality, persona coherence, advantages in managing time pressure and handling special cases, but no or language register (see the limitations discussed below). The practical item-level difference between arms was significant (all Kruskal-Wallis implication is that institutions need not invest in the most expensive p≥0.060, uncorrected). commercial API to achieve educationally adequate AI-SP performance: Training satisfaction did not differ significantly between arms under the same architecture, several models met the viability threshold, so in (Table 6). Per-item comparison (Fig. 7b; all eleven satisfaction dimensions this setting the LLM behaved as a largely replaceable engine. Indeed, three in Supplementary Fig. 5) shows a mixed pattern beneath this overall simi- models spanning distinct accessibility tiers (a Chinese-developed model, larity: AI-SP scored numerically higher on time flexibility and emotional Qwen-2.5 Plus; a premium Western model, Claude 4.5 Sonnet; and a free- support (affordances of asynchronous, judgement-free practice), while tier model, Gemini 2.5 Flash) all exceeded the viability threshold, while the human SP scored higher on interaction naturalness, the one dimension open-weight DeepSeek-R1—accessed via API in this study but deployable npj Digital Medicine Article in Press https://doi.org/10.1038/s41746-026-03372-0 Article Fig. 3 | Phase 1 expert fidelity evaluation (n = 7 evaluators, 350 assessments). between competent and novice student interactions across six dimensions. Error a Overall fidelity by model. Box plots show median and IQR; diamonds indicate bars: SEM. c Fidelity heatmap across models and scenario complexity levels (Low: means; dashed line marks the educational viability threshold (≥20/30). Significance S1; Moderate: S2, S5; High: S3, S4). Bold cell borders indicate cells ≥20/30 (educa- brackets: Bonferroni-corrected (*p < 0.05; **p < 0.01). b Fidelity comparison tionally viable). Table 1 | AI-SP fidelity scores across six evaluation dimensions (mean ± SD, scale 1–5). Total scores sum to 30 Model Clin. Emot. Info. Comm. Behav. Pedag. Total real. auth. ctrl. nat. cons. val. (/30) Qwen-2.5 Plus 3.51 ± 0.70 3.60 ± 0.75 3.34 ± 0.59 3.59 ± 0.71 3.60 ± 0.62 3.39 ± 0.71 21.0 ± 2.9 Claude 4.5 Sonnet 3.53 ± 0.65 3.41 ± 0.73 3.26 ± 0.56 3.46 ± 0.77 3.63 ± 0.57 3.53 ± 0.61 20.8 ± 2.9 Gemini 2.5 Flash 3.53 ± 0.76 3.33 ± 0.77 3.20 ± 0.67 3.41 ± 0.79 3.46 ± 0.72 3.19 ± 0.77 20.1 ± 3.5 DeepSeek-R1 3.23 ± 0.75 3.21 ± 0.66 3.23 ± 0.59 3.20 ± 0.73 3.30 ± 0.64 3.07 ± 0.79 19.2 ± 3.1 GPT-4o 3.10 ± 0.85 3.19 ± 0.75 3.21 ± 0.61 3.13 ± 0.70 3.33 ± 0.61 3.13 ± 0.80 19.1 ± 3.3 Bold indicates highest score per dimension. Educational viability threshold: ≥20/30 (≥3.33 mean per dimension). Three models exceeded this threshold; DeepSeek-R1 and GPT-4o fell slightly below. Table 2 | AI-SP fidelity scores by student interaction level (total communication naturalness and emotional authenticity, dimensions score /30) requiring culturally calibrated patient behaviour including colloquial expressions and age-appropriate speech patterns. GPT-4o’s lower fidelity Model Competent Novice Change despite strong clinical knowledge23 highlights that benchmark superiority GPT-4o 21.23 16.94 4.29 does not guarantee culturally authentic patient simulation, a finding rele- Gemini 2.5 Flash 22.17 18.06 4.11 vant for non-English medical education contexts worldwide. Claude 4.5 Sonnet 22.66 18.97 3.69 The three-layer architecture’s 88.5 percentage-point discovery differ- Qwen-2.5 Plus 22.57 19.49 3.09 ential between competent and novice students is consistent with its func- tioning as a pedagogical mechanism that operationalises cognitive DeepSeek-R1 20.66 17.83 2.83 apprenticeship principles21: just as real patients share sensitive information Change = competent − novice. Higher change indicates greater sensitivity to student skill level. selectively based on interaction quality20, AI-SPs withhold Layer 3 items Sorted by differentiation magnitude. until students demonstrate the requisite communication skills. This mechanism offers three advantages over traditional SP training. First, the hidden-information discovery rate provides an objective, automated self-hosted—offers a pathway to institution-controlled deployment for assessment metric without requiring expert observation. Second, AI-SPs settings with data-residency constraints22. This landscape means resource- apply disclosure rules deterministically, creating a consistent assessment constrained programmes can select from multiple viable options and focus standard that human SPs (whose disclosure behaviour may vary despite investment on instructional design rather than model selection. training3) cannot match. Third, per-item difficulty profiling generates That said, model choice is not irrelevant. In our Chinese-language granular curriculum diagnostic data that would be difficult to obtain at scale context, Qwen’s native training data conferred advantages in with human SPs. npj Digital Medicine Article in Press https://doi.org/10.1038/s41746-026-03372-0 Article Table 3 | Hidden information (Layer 3) discovery rates by scenario and student level Scenario Hidden items n Competent Novice S1 (Low) Aspirin, alcohol, snoring, loose molar 4 4/4 (100%) 0/4 (0%) S2 (Mod.) Herbal thyroid tea, dental prosthesis, missed metformin, unresolved cold 4 4/4 (100%) 0/4 (0%) S3 (High) Self-stopped clopidogrel, palpitations, husband death, OSA, memory decline, dentures, penicillin allergy 7 7/7 (100%) 1/7 (14.3%) S4 (High) Daily ibuprofen, fitness supplements (ephedrine), concussion, heavy drinking, benzodiazepine use, e-cigarettes 6 6/6 (100%) 0/6 (0%) S5 (Mod.) Saffron capsules (5-HT risk), double-dose lorazepam, extreme diet, advance directive, smoking 5 5/5 (100%) 2/5 (40.0%) Total 26 26/26 (100%) 3/26 (11.5%) Novice students failed to discover any safety-critical hidden items across all five scenarios. n = number of hidden Layer 3 items per scenario. Table 4 | Phase 1b live student–AI-SP interaction results by scenario (n = 31 students, 155 sessions) Scenario Hidden Discovery rate Items Messages Duration items found S1 Cholecystectomy 4 65.3% ± 31.4% 2.6/4 45 ± 12 15.4 min S2 Thyroidectomy 4 85.5% ± 34.0% 3.4/4 34 ± 9 14.9 min S3 Knee replacement 7 67.3% ± 37.8% 4.7/7 38 ± 13 14.5 min S4 Lumbar disc 6 51.1% ± 36.2% 3.1/6 37 ± 17 10.7 min S5 Mastectomy 5 58.7% ± 42.2% 2.9/5 35 ± 13 13.3 min Overall 26 65.6% ± 37.8% — 38 ± 14 13.8 min Discovery rate = proportion of hidden Layer 3 items disclosed during the interaction. Values are mean ± SD across 31 students. Fig. 4 | Per-item hidden-information discovery rates across five scenarios (n = 31 students, 26 items). Items ranked by discovery rate and colour- coded by scenario. S2 items were near-universally discovered ( > 80%) while S4 items, particularly concussion history (23%) and diazepam use (42%), proved most challenging, reflecting the difficulty of eliciting information that patients actively minimise. npj Digital Medicine Article in Press https://doi.org/10.1038/s41746-026-03372-0 Article Table 5 | Phase 1b post-training questionnaire results (1–5 after correction for multiple comparisons. We consider the two outcomes Likert scale, n = 21 respondents) in turn. The absence of a statistically significant between-group OSCE differ- Item Mean ± SD ence (p = 0.483; checklist dAB = 0.38) is consistent both with true equivalence Learning effectiveness 4.76 ± 0.44 and with a clinically meaningful difference that this pilot, with under 25 per Communication skill improvement 4.71 ± 0.46 cent power for an effect size of about 0.38, was unable to detect; it does not, Overall satisfaction 4.52 ± 0.60 by itself, constitute evidence of equivalence. A formal non-inferiority con- Comparable to human SP 4.33 ± 0.73 clusion would require a pre-specified margin, a one-sided confidence interval, and a demonstration that its lower bound excludes that margin, a Understanding of three-layer architecture 3.67 ± 0.97 test this pilot was neither designed nor powered to perform. We therefore Fig. 5 | CONSORT flow diagram for the Phase 2 pilot RCT. Sixty-one students were randomized into three arms; three withdrew before training commenced (attrition rate: 4.9%). All 58 remaining participants completed the study protocol and were included in intention-to-treat analysis. The Phase 1b ecological validation demonstrated this diagnostic report the observed effect sizes to inform the sample size of a future con- capacity: per-item discovery rates ranging from 23% (concussion history) to firmatory trial rather than as an equivalence claim. We did not conduct a 87% (porcelain crown) across 26 items revealed that students reliably formal economic evaluation; as an order-of-magnitude anchor, the total inquire about conditions they associate with anaesthetic risk but con- large language model application-programming-interface cost for the entire sistently miss information that patients minimise or normalise. The con- study (all phases, more than 500 simulated consultations) was approxi- centration of low-discovery items in S4, where a young male patient mately CNY 100 (US$14), reflecting marginal inference cost only and normalises recreational supplement use and conceals benzodiazepine excluding platform development, maintenance, and examiner time. A full dependence, identifies a specific curricular gap in training students to cost-effectiveness comparison against human SP programmes remains to be recognise and probe patient minimisation behaviours. These findings performed. The dominant source of improvement was a testing effect: extend the paradigm demonstrated by Luo et al.16 in ophthalmology by ANCOVA confirmed that baseline scores alone explained 48.5% of post- showing that AI-SP systems can provide not only training but also sys- training variance, leaving limited variance for training-specific effects to tematic diagnostic data about educational blind spots24. capture25. Importantly, the three-arm design allows contextualisation that In the Phase 2 pilot, all three groups (AI-SP, human SP, and untrained two-arm studies cannot: the largest published AI-SP RCT (Lavigne et al., controls) achieved comparable OSCE skill gains, whereas self-efficacy gains n = 247 17) reported a significant AI advantage but compared AI only against differed numerically between arms without reaching statistical significance standard training without a human SP arm, a design that cannot distinguish npj Digital Medicine Article in Press https://doi.org/10.1038/s41746-026-03372-0 Article Table 6 | Phase 2 RCT outcomes: baseline characteristics and pre–post changes across study arms Outcome AI-SP Human SP Control p Effect size (n = 21) (n = 18) (n = 19) Baseline (pre-training) OSCE checklist (/14) 7.5 ± 1.1 7.7 ± 1.3 7.2 ± 1.5 0.715 — OSCE total (/39) 18.5 ± 1.8 19.1 ± 2.7 18.3 ± 2.6 0.588 — Self-efficacy (/50) 25.1 ± 9.1 30.6 ± 10.1 27.3 ± 9.5 0.193 — Primary outcome Checklist change (ICC=0.72) +1.4 ± 1.0 +0.9 ± 1.3 +1.0 ± 1.2 0.483a dAB=0.38 Secondary outcomes OSCE total change (ICC=0.40) +8.6 ± 1.9 +7.3 ± 3.4 +7.6 ± 2.6 0.266a dAB=0.52 Self-efficacy change +10.2 ± 6.3 +6.8 ± 6.8 +6.1 ± 6.8 0.064a dAC=0.62c Satisfaction (/55) e 45.0 ± 7.2 44.4 ± 6.9 — 0.876 b dAB=0.09 Values are mean ± SD. Human SP: n = 18. Holm-adjusted across the three pairwise comparisons, p = 0.103. a Kruskal–Wallis H test across three arms. b Mann–Whitney U test (two arms). c AI-SP vs Control Mann-Whitney U test: unadjusted p = 0.034. dAB: Cohen’s d for AI-SP vs Human SP; dAC: AI-SP vs Control. e Satisfaction collected for intervention arms only (AI-SP: n = 21. Fig. 6 | Phase 2 score changes by outcome measure. Each panel shows pre–post score (/14; ICC = 0.72; primary outcome): all groups improved comparably score change by study arm (box plots with individual data points; diamonds indicate (p = 0.483). b Global rating score (/25; ICC = 0.15): poor inter-rater reliability means). Panel letters (a–c) denote the three outcome measures; within each panel motivated its exclusion as primary outcome. c Total OSCE score (/39; ICC = 0.40): the study arms are labelled A (AI-SP), B (Human SP), and C (Control). a Checklist no statistically significant between-arm difference (underpowered pilot) (p = 0.266). AI efficacy from the absence of an active comparator. Our pilot RCT not explanations of an established effect. Bandura’s self-efficacy theory29 (n = 58), though smaller, provides a structurally more informative com- identifies four sources of efficacy beliefs: mastery experience, vicarious parison: the inclusion of both human SP and no-intervention arms positions experience, verbal persuasion, and physiological/affective states. AI-SP a future adequately powered study to test superiority (vs control) and non- training may amplify two of these. First, the psychologically safe environ- inferiority (vs human SP) simultaneously; the present pilot supplies the ment, where errors carry no interpersonal consequences and practice can be effect-size estimates such a test requires. This is a qualitative design repeated at self-selected times, reduces performance anxiety (affective state), advantage, not merely a quantitative limitation. The pilot was designed to enabling students to accumulate positive mastery experiences without the estimate effect sizes for a future confirmatory trial26,27 rather than to detect social evaluation apprehension inherent in human SP encounters. Second, small differences; post-hoc power analysis (N≥216 needed for 80% power at unlimited repetition provides more frequent mastery experiences per unit d = 0.38 28) confirms that definitive equivalence testing requires a larger time than scheduled human SP sessions. The largest per-item AI-SP multicentre study. Critically, the three-phase design means the RCT is advantage was in managing time pressure, which is compatible with this contextualised by 350 expert assessments and 155 live student sessions; the account, although that item-level difference was not significant (p = 0.060, total evidence base supporting AI-SP validity is substantially larger than the uncorrected). Self-efficacy is not merely a subjective outcome; in clinical Phase 2 sample alone. education, it predicts willingness to attempt challenging patient interactions, The self-efficacy results should be read as exploratory. AI-SP students persistence when communication breaks down, and ultimately clinical gained more confidence than controls (mean change = +10.2 vs +6.1; performance in unsupervised settings4. Notably, the modality difference d = 0.62) and than human SP students (+10.2 vs +6.8; d = 0.51), but the (text vs voice) may also contribute: text-based interaction allows more three-arm comparison was not significant (p = 0.064) and neither pairwise deliberate response composition, potentially reinforcing a sense of com- difference survived Holm correction (adjusted p = 0.103 and 0.138). The municative competence that transfers to self-efficacy ratings. mechanisms below are therefore hypotheses for a confirmatory trial to test, npj Digital Medicine Article in Press https://doi.org/10.1038/s41746-026-03372-0 Article Fig. 7 | Phase 2 secondary outcomes. a Self-efficacy scores (/50) before and after p = 0.103) is exploratory, as the three-arm Kruskal–Wallis test was not significant training. Individual student trajectories (thin lines) and group means ( ± SEM; (p = 0.064). b Per-item training satisfaction comparison between AI-SP and human circles = pre, squares = post). All groups improved significantly (**p < 0.01; SP arms (1–5 Likert scale). No item showed significant between-group differences ***p < 0.001), and the AI-SP group showed the largest mean gain (mean change = (all p > 0.27); interaction naturalness numerically favoured human SP, while time +10.2); its difference from controls (d = 0.62; unadjusted p = 0.034, Holm-adjusted flexibility favoured AI-SP. Training satisfaction did not differ significantly between AI-SP and Second, the Phase 2 RCT (n = 58) was designed and powered as a pilot human SP arms (p = 0.876), with both scoring above 3.8/5.0 on all study; while baseline equivalence was confirmed, the sample was insufficient dimensions. This finding contrasts with early concerns that students would to detect small between-group OSCE differences (post-hoc power < 25% for resist AI-mediated clinical training, and suggests that the educational value observed d = 0.38; N≥216 needed for 80% power). The absence of a sig- and convenience of AI-SP practice compensate for any perceived limitations nificant OSCE difference is therefore compatible with either genuine in interaction naturalness. equivalence or limited statistical power, a distinction that only a larger Notably, the entire Phase 2 protocol (training, assessment, and data confirmatory trial can resolve. Secondary outcomes were compared without collection) was conducted fully remotely. AI-SP training used text-based pre-specified correction for multiple comparisons, so the self-efficacy chat; human SP training used voice calls via videoconferencing software findings are hypothesis-generating. with trained actors following identical scripts; OSCE assessments were Third, within-group variability in OSCE score changes was substantial video-recorded and scored asynchronously by blinded examiners. This (checklist change score SD = 1.0–1.3 on a 14-point scale), compounded by a shows that an AI-SP training programme can be delivered without dedi- strong testing effect that elevated all groups’ scores and compressed cated simulation facilities, on-site scheduling infrastructure, or synchronous between-group differences. We addressed this by: (a) using ANCOVA to examiner coordination, a deployment pathway particularly relevant for adjust for baseline scores, which accounted for 48.5% of post-training var- programmes serving geographically distributed learners and infeasible with iance; (b) pre-specifying the high-reliability checklist (ICC = 0.72) as the traditional in-person SP programmes. The practical implication is primary outcome; and (c) reporting both parametric (ANCOVA) and non- straightforward: if a confirmatory trial upholds these pilot estimates, namely parametric (Kruskal–Wallis, Mann-Whitney) tests. Fourth, using the same comparable skill gains, a possible self-efficacy advantage, and comparable five scenarios for both training and assessment introduced scenario-specific satisfaction at low marginal cost, then the principal barrier to wider adop- familiarity effects. This confound acts in a specific direction: it inflates all tion would shift from evidence toward implementation. post-training scores and compresses between-group differences, making a Several limitations warrant consideration. First, the AI-SP and human null between-arm result more likely regardless of true training effects, and, SP arms used different communication modalities (text-based chat vs voice unlike baseline imbalance, it cannot be removed by statistical adjustment. It calls), introducing a confound that prevents attributing outcome differences should therefore be read as an intrinsic limit on this design’s ability to detect solely to the AI vs human distinction. This was a deliberate design choice between-arm differences. Future implementations should employ a larger reflecting current deployment reality: text-based LLM interaction is the scenario bank with non-overlapping training and assessment scenarios to most mature, accessible, and reproducible modality available today, while isolate genuine communication skill transfer from scenario memorisation. human SP training is inherently voice-based; forcing it into text would Fifth, the study assessed immediate post-training outcomes only; misrepresent the comparator and reduce ecological validity. Matching longer-term skill retention (4–12 weeks) and transfer to real patient modalities would improve internal validity but at the cost of testing a encounters were not evaluated. Sixth, inter-rater reliability for the global comparison that no institution would actually implement. Notably, text- rating scale was poor (ICC = 0.15), necessitating reliance on the checklist based interaction is conventionally considered a communicative dis- score as the primary outcome. While binary checklist items are inherently advantage relative to voice, as it lacks paralinguistic cues, real-time turn- more reliable than subjective Likert scales, this limits assessment to obser- taking, and tonal modulation that are central to clinical communication30,31. vable behaviours rather than holistic communication quality. As Newble That text-based AI-SP reached comparable OSCE outcomes despite text’s notes, checklists and global ratings suit different stations, and global jud- recognised communicative disadvantage is suggestive, but we emphasise gements depend on assessor experience32, so lower global-rating agreement that this reasoning does not control the confound: modality (text vs voice) partly reflects this instrument’s known sensitivity to rater calibration. Low and agent (AI vs human) remain entangled in our design, and no rhetorical global rating reliability is a well-documented challenge in OSCE assessment: argument substitutes for a factorial comparison. This is the study’s most Regehr et al.33 reported global rating ICCs of 0.20–0.40 across clinical tasks, significant unresolved design limitation. A confirmatory study should cross and Gershov et al.34 recently reported comparable variability in AI-assisted the two factors, at minimum by adding a text-based human SP arm and a competency assessment. Our decision to report ICC transparently and voice-based AI-SP arm, so that modality and agent effects can be estimated conduct three layers of robustness analysis (leave-one-out, Supplementary separately. Table 9; sensitivity, Supplementary Table 10; examiner variability, npj Digital Medicine Article in Press https://doi.org/10.1038/s41746-026-03372-0 Article Supplementary Table 6; Supplementary Note 1, Supplementary Fig- ecological validity, and training efficacy. Future research should prioritise ures 6–8) represents a methodological strength. Enhanced examiner cali- adequately powered multicentre equivalence trials with extended follow-up, bration with video-anchored exemplars should be prioritised in future cross-specialty validation, investigation of optimal AI-SP–human-SP inte- studies. gration ratios in curricula, and development of multimodal AI-SPs incor- Additional methodological considerations include: the study was porating voice and video interaction. The exploratory self-efficacy signal conducted at a single institution with Chinese-language scenarios, limiting should be tested as a pre-specified outcome, with its psychological cross-linguistic and cross-cultural generalisability. We designed the dis- mechanisms examined directly. The three-layer information architecture closure rules to operate at the level of communication behaviour rather than may also be applicable beyond medical education to any professional language syntax, which we conjecture may support transfer across lan- training context where skilled questioning is central, including journalism, guages; however, all scenarios and prompt engineering were Chinese-lan- social work, and legal interviewing. guage, and whether Layer 3 trigger conditions transfer across languages and cultures is an untested empirical question, not an established property. The Methods five-model comparison does embed deployment diversity across US and Study design and ethics Chinese providers, open- and closed-weight platforms, and three accessi- This three-phase study was conducted at the Department of Anaes- bility tiers, but this speaks to model portability, not cross-linguistic gen- thesiology, The First Affiliated Hospital of Dalian Medical University, eralisability. The remote video-conferenced OSCE format, while validated China. Phase 1 validated AI-SP fidelity through blinded expert evaluation of in post-pandemic literature35,36, does not assess paralinguistic and non- scripted dialogues. Phase 1b extended validation to live student–AI-SP verbal communication skills. A multicentre, multilingual replication across interactions. Phase 2 employed a three-arm, parallel-group randomized specialties is planned as the next phase of this research programme. controlled trial. The study was reported following DECIDE-AI guidelines38 Several further limitations qualify the interpretation and deployment for AI system evaluation (completed checklist in Supplementary Table 11) of these findings. First, the “architecture over model” inference is partially and CONSORT guidelines for the RCT component (completed checklist in confounded by design: because the identical architecture was applied to Supplementary Table 12). The study was determined to be exempt from every model, the larger variance attributable to student skill than to model formal ethics review by the Ethics Committee of The First Affiliated Hos- identity (η2 = 0.31 vs 0.06) cannot by itself separate the specific contribution pital of Dalian Medical University (exemption reference: YJ-MS-2025-02, of the three-layer disclosure mechanism from that of prompt quality, per- dated 9 October 2025), as it involved comparison of standard educational sona coherence, or language register. A stronger test would compare the methods among medical students with no patient contact, no clinical same models with and without the architecture; we frame the present result intervention, and no collection of biological specimens. This was an edu- as consistent with, but not proof of, an architecture-dominant account. cational evaluation comparing teaching methods rather than a clinical trial Second, Phase 1 fidelity was assessed on scripted dialogues authored to of a health intervention; consistent with its educational nature and its represent competent and novice students; real novices ramble, misunder- exemption from clinical ethics review, it was not registered in a clinical trials stand questions, and pursue tangents in ways scripts do not, which limits the registry. We acknowledge the absence of prospective registration as a lim- predictive validity of scripted-dialogue fidelity for live performance. Relat- itation and address its implications for outcome selection in the Discussion. edly, Phase 1b was conducted with a single model (Qwen-2.5 Plus); its per- All student participants provided written informed consent. item discovery rates, session durations, and satisfaction data therefore characterise that deployment and should not be generalised to the other four Scenario design and AI-SP construction models evaluated in Phase 1. Third, the standardised patient, whether AI or Five preoperative anaesthesia consultation scenarios were developed human, is a deliberately bounded model of a person: a genuine patient is through iterative expert consensus, stratified by complexity: S1 (low: more than a set of partly hidden cues and risk factors, and skilled clinicians laparoscopic cholecystectomy, 60M), S2 (moderate: thyroid nodulectomy, “read between the lines” in ways that matter most outside the structured pre- 45F), S3 (high: total knee replacement, 75F), S4 (high: lumbar disc surgery, anaesthesia encounter (for example, in family medicine); whether current 30M), and S5 (moderate: mastectomy, 55F). Scenarios covered the spectrum models support this remains uncertain, and our task deliberately does not of preoperative communication challenges including medication reconci- test it. Fourth, all evaluated models are hosted services under continuous liation, substance use disclosure, psychosocial assessment, and management development and subject to silent updates, deprecation, and capability drift, of acute anxiety. so the published prompts may over time elicit different behaviour from the Each AI-SP was constructed with a patient persona and the three-layer models they were tuned for; deployments should pin model versions and information architecture described in the Introduction (Fig. 1a). Each sce- periodically re-validate. Fifth, in real deployment students aware that the nario contained 4–7 hidden Layer 3 items, including at least one safety- patient is an AI may attempt to elicit hidden information through direct critical item whose non-discovery would represent a clinically significant requests or prompt injection rather than skilled interviewing; adversarial assessment gap (Table 3). System prompts were structured in four sections: prompt hardening was not part of the present system design and is required patient identity (demographics, personality, speech patterns), Layer 1 items before high-stakes assessment use. Sixth, student–AI conversation logs are (volunteered spontaneously), Layer 2 items (disclosed when directly asked), recorded, analysed, and stored, raising data-governance obligations; for the and Layer 3 items with disclosure conditions (gated behind specific com- public repository, participant identifiers were replaced with anonymous munication behaviours). Additionally, each prompt specified an emotional codes and free-text comments and timestamps were removed prior to response model defining baseline anxiety, triggers for escalation or de- release, but a full governance framework (retention, access control, and escalation, defensive reactions to insensitive questioning, and gradual trust- jurisdiction-specific compliance) should accompany any operational building responses to empathetic communication1,39. All five LLMs received deployment. identical system prompts. Model parameters were standardised across Taken together, this study addresses three gaps in the emerging AI-SP models (temperature = 0.7, top_p = 0.9). The response token limit was 4096 literature. Unlike prior single-model studies16,17,37, the five-model compar- for Phase 1 scripted-dialogue generation; the live student-facing platform ison provides multi-model evidence on the portability of the pedagogical (Phases 1b and 2) applied a 2048-token per-turn limit appropriate to architecture. Unlike two-arm designs that compare AI-SP to standard interactive chat. training, our three-arm RCT includes the clinically meaningful human SP Translating this architecture into effective system prompts required comparator alongside a no-intervention control, enabling both superiority iterative prompt engineering (Fig. 2; Supplementary Table 13). Over four (vs control) and, in a future adequately powered study, non-inferiority (vs development rounds, a clinician–engineer team tested each draft prompt human SP) inferences. And unlike technical benchmarking studies18, the against all five LLMs across both competent and novice student interaction three-phase validation pipeline progressively establishes construct validity, patterns, identifying and resolving four recurring failure modes: (i) role npj Digital Medicine Article in Press https://doi.org/10.1038/s41746-026-03372-0 Article boundary violations, where the model offered differential diagnoses or equivalent to ≥3.33 mean per dimension) was established through a two- treatment suggestions rather than remaining in the patient role; (ii) pre- round modified Delphi process among the seven evaluators: the threshold mature information disclosure, where multiple Layer 3 items were revealed corresponds to the “adequate for unsupervised student use” anchor point on in a single turn regardless of conversational context; (iii) affective flattening, the rubric, and maps to the institutional OSCE borderline pass standard where the model responded with uniformly cooperative behaviour irre- (60% on a converted percentage scale) that the department applies to spective of the student’s communication quality; and (iv) stylistic artefacts, human SP encounters. This is a criterion-referenced standard rather than a including verbose, formal phrasing inconsistent with the patient persona data-derived cut-point: it was fixed by expert consensus before scores were and bracketed action descriptions (e.g., “[hesitates]”) absent from natural examined, following the principle that simulation fidelity standards should patient speech. Each failure mode was addressed through targeted beha- be anchored to the intended educational use of the tool4. We use it vioural constraints appended to the prompt: an explicit safety block for- descriptively to distinguish models that met the department’s “adequate for bidding medical advice, a “one hidden item per turn” disclosure rule, unsupervised practice” bar, not as an inferential test. conditional emotional response rules linking patient cooperation to inter- viewer empathy, and response length limits enforcing 1–3 sentence turns in Phase 1b: live student validation colloquial register (representative prompt in Supplementary Table 13). The Thirty-one medical students (fourth- and fifth-year) each completed all five final prompts averaged 78 lines ( ~ 1.8 KB) per scenario. scenarios using Qwen-2.5 Plus (selected as the deployment model because it achieved the highest Phase 1 fidelity score, offers a free-tier API with low Platform and model selection latency in China, and natively handles Chinese-language interaction with- The AI-SP system was deployed as a web-based platform (Fig. 1) comprising out translation artefacts), generating 155 live sessions. The platform a real-time chat interface, a scenario management module, an LLM inte- recorded all dialogue turns, timestamps, and session duration in real time. gration layer routing messages to the assigned model, and an automated Hidden-information discovery was assessed automatically: upon session data collection backend recording all dialogue turns, timestamps, and ses- completion, the dialogue transcript was submitted to a second Qwen-2.5 sion metadata. For Phase 2, a server-side whitelist access control mechanism Plus instance with an analysis prompt listing all Layer 3 items for that enforced group allocation: upon login, each participant’s student ID was scenario, returning a binary classification (discovered/not discovered) for verified against the randomization list stored as a server environment each item. Each hidden item has explicit trigger conditions hard-coded in variable, automatically assigning the correct study arm without self- the system prompt (Supplementary Table 13), and the detection task selection. Only Arm A participants were granted access to the AI-SP chat reduces to verifying whether the patient character’s response contained the interface; Arm B and Arm C participants received group-appropriate pre-specified disclosure content, a constrained binary classification with instructions but were technically unable to access the AI practice module, minimal ambiguity compared with open-ended assessment tasks. To vali- preventing cross-arm contamination at the platform level. date this automated classification rather than assert its reliability, a random Five frontier LLMs were selected to span three accessibility tiers rele- sample of 20 sessions (104 item-level judgements) was independently re- vant to global deployment: GPT-4o (OpenAI; commercial API)23, Claude coded by three anaesthesiology educators, each blinded to the automated 4.5 Sonnet (Anthropic; commercial API), Gemini 2.5 Flash (Google; free- labels and to one another (Supplementary Table 14). Agreement between tier API), Qwen-2.5 Plus (Alibaba; free-tier API), and DeepSeek-R1 the automated classifier and the three human coders was 98.1%, 86.5%, and (DeepSeek; an open-weight model that can be self-hosted but was accessed 95.2% (Cohen’s κ = 0.93, 0.63, and 0.84; mean κ = 0.80), while the three here through the official DeepSeek API). This selection enables comparisons coders agreed with one another at a mean κ of 0.79 (pairwise κ = 0.69–0.91). across performance tiers (premium, cost-efficient, open-weight), geo- Against the majority judgement of the three coders, the automated classifier graphic provenance (US, China), and provider ecosystems, representing the achieved 95.2% agreement (Cohen’s κ = 0.84; 95% CI 0.71–0.98). The spectrum of options available to medical education programmes worldwide. automated classifier therefore agreed with expert human judgement as For reproducibility, all models were accessed through their providers’ APIs closely as the experts agreed with one another, supporting the use of the during February–March 2026 using the following identifiers: gpt-4o, automated discovery rate as the Phase 1b measurement instrument. All claude-sonnet-4-5-20250929, gemini-2.5-flash, qwen- disagreements arose from items the classifier scored as disclosed but a coder plus (Qwen-2.5 Plus), and deepseek-reasoner (DeepSeek-R1). scored as not disclosed; no item judged disclosed by a coder was missed by Because all five were accessed as hosted services, model weights, parameter the classifier. Following all scenarios, students completed an anonymous 11- counts, and quantization were fixed by each provider and not under our item Likert-scale questionnaire (1–5) spanning AI-SP realism, learning control; we therefore report provider, identifier, and access window in place value, and adoption willingness, plus three open-ended questions. Internal of local deployment specifications, and note that hosted models are subject consistency was assessed using Cronbach’s α. to silent provider-side updates (addressed in the Discussion). Phase 2: randomized controlled trial Phase 1: dialogue generation and expert evaluation This phase was designed as a pilot RCT; the target sample size of approxi- For each scenario, two scripted student interaction patterns were developed mately 60 was determined by the single-cohort rotation size available during representing distinct skill levels: competent (empathetic, open-ended, sys- the study period rather than a formal a priori power calculation, consistent tematic) and novice (checklist-style, minimal empathy, jargon-heavy), with recommendations for pilot trials aimed at estimating effect sizes and based on documented communication patterns from institutional OSCE assessing feasibility26,27. Sixty-one medical students on anaesthesiology assessments40. Each scenario × student level × model combination was rotation were recruited. Participants were randomized using computer- generated once using standardised parameters, yielding 50 dialogues generated sex-stratified block randomization (seed = 20260308; sex dis- (5 scenarios × 2 levels × 5 models). tribution: χ2 = 0.07, p = 0.956) into three arms: AI-SP (n = 22), human SP Seven anaesthesiology faculty served as blinded evaluators (mean (n = 20), and control (n = 19). Three students withdrew before training clinical experience: 12.4 years, range: 8–18; mean simulation education commenced (one AI-SP, two human SP), yielding 58 analysed participants experience: 7.2 years). Evaluators were blinded to model identity and pre- (AI-SP: 21, human SP: 18, control: 19; attrition rate: 4.9%). Students who sentation order was randomised. Each evaluator assessed all 50 dialogues, had participated in Phase 1b were excluded to prevent contamination. yielding 350 total assessments. Six dimensions were scored on 1–5 Likert The entire protocol was designed as a fully remote intervention. Arm A scales with anchored rubrics: Clinical Realism, Emotional Authenticity, (AI-SP) completed 10 practice consultations over one week (five scenarios, Information Control, Communication Naturalness, Behavioural Con- each attempted twice) using the text-based AI-SP platform, with automated sistency, and Pedagogical Value. Prior to scoring, evaluators completed a post-session feedback. Arm B (Human SP) completed an identical schedule 3-hour calibration workshop. The educational viability threshold (≥20/30, of 10 practice consultations via voice calls over videoconferencing software, npj Digital Medicine Article in Press https://doi.org/10.1038/s41746-026-03372-0 Article with trained graduate student actors following identical scenario scripts and (between-group) and paired d (within-group). Inter-rater reliability was three-layer disclosure rules. The two arms intentionally used different assessed using ICC(2,1) (two-way random, single measures, absolute communication modalities: text-based chat for AI-SP reflects the current agreement). All analyses were intention-to-treat. Statistical significance: deployment reality of LLM-driven systems, while voice interaction for α = 0.05. Analyses used Python 3.9 (SciPy 1.11, statsmodels 0.14). human SP preserves the ecological validity of actor-based training35. This design prioritises external validity, comparing AI-SP and human SP as they Data availability would actually be deployed, over strict internal control of modality. The The datasets generated and analysed during the current study (anonymised implications of this modality difference are addressed in the Discussion. evaluation scores, AI-SP prompt templates, scenario specifications, scoring Arm C (Control) followed the standard curriculum with no additional rubrics, complete anonymised dialogue transcripts, and the DECIDE-AI practice. To prevent cross-arm contamination, the AI-SP platform enforced and CONSORT reporting checklists) are available in the GitHub repository server-side whitelist access control: only Arm A student identifiers were at https://github.com/RegAItool/ai-standardized-patient, which also con- authorised for the practice module. tains the source data underlying all figures and tables. The repository is not The OSCE comprised two scoring instruments: a 14-item binary publicly available during peer review because the manuscript is under checklist assessing observable consultation behaviours (e.g., “verified patient consideration; during this period the data are available from the corre- identity”, “explained anaesthetic options”) and a 5-item global rating scale sponding author on reasonable request. The repository will be made pub- assessing holistic communication quality on 1–5 Likert scales with licly available upon acceptance. descriptive anchors (e.g., 1 = “no attempt”, 3 = “adequate but formulaic”, 5 = “exemplary, patient-centred”). The pre–post change in the OSCE checklist Code availability score (/14) was designated the primary outcome on the basis of its markedly Analysis scripts (Python 3.9) for all statistical analyses and figure generation superior inter-rater reliability (ICC(2,1) = 0.72) relative to the global rating are available in the same GitHub repository at https://github.com/ scale (ICC = 0.15) and total score (ICC = 0.40). We note explicitly that this RegAItool/ai-standardized-patient, subject to the same access conditions designation was made after inter-rater reliability had been estimated rather during peer review. than pre-registered; the study was an educational evaluation and was not registered in a trials registry (see Study design and ethics). To guard against Received: 28 March 2026; Accepted: 26 September 2026; selective reporting, all three measures, the checklist, the global rating, and the total score, are reported in full (Table 6; Fig. 6), and their reliability is reported transparently. The poor global rating ICC likely reflects the References inherently subjective nature of holistic quality judgements in asynchronous 1. Makoul, G. Essential elements of communication in medical video scoring without real-time calibration; leave-one-out and sensitivity encounters: the Kalamazoo consensus statement. Academic Med. analyses (Supplementary Note 1, Supplementary Figures 6–8) confirmed 76, 390–393 (2001). that this was not driven by a single outlier examiner but reflected systematic 2. Ericsson, K. A. Deliberate practice and the acquisition and inter-examiner variability across all five dimensions. OSCEs were con- maintenance of expert performance in medicine and related domains. ducted at baseline (Week 0) and immediately post-training (Week 1) across Academic Med. 79, S70–S81 https://doi.org/10.1097/00001888- five stations (10 minutes each) via remote videoconference. All sessions were 200410001-00022 (2004). video-recorded and scored asynchronously by five blinded examiners, each 3. Barrows, H. S. An overview of the uses of standardized patients for assigned primary responsibility for one station with cross-scoring of 15 teaching and evaluating clinical skills. Academic Med. 68, 443–451 randomly selected recordings per station for ICC estimation (26% dual- (1993). rated). This video-recorded OSCE approach is consistent with validated 4. Issenberg, S. B., McGaghie, W. C., Petrusa, E. R., Gordon, D. L. & telehealth assessment protocols35,36. Secondary outcomes included total Scalese, R. J. Features and uses of high-fidelity medical simulations OSCE score change (/39), self-efficacy change (10-item questionnaire, /50; that lead to effective learning: a BEME systematic review. Med. Teach. assessed at Week 0 and Week 3), and training satisfaction (11-item Likert 27, 10–28 https://doi.org/10.1080/01421590500046924 (2005). scale, /55; collected for Arms A and B only, as Arm C received no inter- 5. Blitz, J. D. et al. Preoperative evaluation clinic visit is associated with vention to evaluate). decreased risk of in-hospital postoperative mortality. Anesthesiology 125, 280–294 https://doi.org/10.1097/ALN.0000000000001193 Statistical analysis (2016). Phase 1 analyses employed one-way ANOVA (five models) and two-way 6. Bosse, H. M., Nickel, M., Huwendiek, S., Schultz, J. H. & Nikendei, C. ANOVA (model × student level; model × complexity) with Bonferroni- Cost-effectiveness of peer role play and standardized patients in corrected post-hoc comparisons. Inter-rater reliability was assessed using undergraduate communication training. BMC Med. Educ. 15, 183 ICC (two-way random, absolute agreement). Effect sizes are reported as η2 https://doi.org/10.1186/s12909-015-0468-1 (2015). (ANOVA) and Cohen’s d (pairwise). 7. Singhal, K. et al. Large language models encode clinical knowledge. Phase 1b used descriptive statistics for per-scenario and per-item dis- Nature 620, 172–180 https://doi.org/10.1038/s41586-023-06291-2 covery rates. Sequential learning effects were assessed using paired t-tests (S1 (2023). vs S5). Pearson correlation assessed the engagement–performance 8. Savage, T., Nayak, A., Gallo, R., Rangan, E. & Chen, J. H. Diagnostic relationship. reasoning prompts reveal the potential for large language model Phase 2 employed non-parametric tests given the small sample and interpretability in medicine. npj Digital Med. 7, 20 https://doi.org/10. non-normal distributions. Between-group comparisons used 1038/s41746-024-01010-1 (2024). Kruskal–Wallis H tests with pairwise Mann–Whitney U tests. Pairwise 9. Lee, P., Bubeck, S. & Petro, J. Benefits, limits, and risks of GPT-4 as an comparisons were not pre-specified with a multiplicity correction; for the AI chatbot for medicine. N. Engl. J. Med. 388, 1233–1239 https://doi. pairwise self-efficacy comparisons reported in the text we therefore give org/10.1056/NEJMsr2214184 (2023). Holm-adjusted p values (across the three pairwise comparisons) alongside 10. Shanahan, M., McDonell, K. & Reynolds, L. Role play with large unadjusted values, and all secondary-outcome comparisons, including the language models. Nature 623, 493–498 https://doi.org/10.1038/ ten item-level self-efficacy tests (reported uncorrected), are treated as s41586-023-06647-8 (2023). exploratory. Within-group pre–post comparisons used Wilcoxon signed- 11. Kron, F. W. et al. Using a computer simulation for teaching rank tests. ANCOVA (post-training score ~ group + baseline score) was communication skills: a blinded multisite mixed methods randomized used to adjust for baseline differences. Effect sizes are reported as Cohen’s d npj Digital Medicine Article in Press https://doi.org/10.1038/s41746-026-03372-0 Article controlled trial. Patient Educ. Counseling 100, 748–759 https://doi. 31. Walther, J. B. Computer-mediated communication: Impersonal, org/10.1016/j.pec.2016.10.024 (2017). interpersonal, and hyperpersonal interaction. Commun. Res. 23, 3–43 12. Lane, H. C., Hays, M. J., Core, M. G. & Auerbach, D. Learning https://doi.org/10.1177/009365096023001001 (1996). intercultural communication skills with virtual humans: feedback and 32. Newble, D. Techniques for measuring clinical competence: objective fidelity. J. Educ. Psychol. 105, 1026–1035 https://doi.org/10.1037/ structured clinical examinations. Med. Educ. 38, 199–203 (2004). a0031506 (2013). 33. Regehr, G., MacRae, H., Reznick, R. K. & Szalay, D. Comparing the 13. Thesen, T. & Park, S. H. A generative AI teaching assistant for psychometric properties of checklists and global rating scales for personalized learning in medical education. npj Digital Med. 8, 627 assessing performance on an OSCE-format examination. Acad. Med. https://doi.org/10.1038/s41746-025-02022-1 (2025). 73, 993–997 https://doi.org/10.1097/00001888-199809000-00020 14. Mandal, S. et al. Utilization of generative AI-drafted responses for (1998). managing patient-provider communication. npj Digital Med. 8, 591 34. Gershov, S., Mahameed, F., Raz, A. & Laufer, S. Towards accurate https://doi.org/10.1038/s41746-025-01972-w (2025). and interpretable competency-based assessment: enhancing clinical 15. Stanceski, K. et al. The quality and safety of using generative AI to competency assessment through multimodal AI and anomaly produce patient-centred discharge instructions. npj Digital Med. 7, detection. npj Digit. Med. 9, 219 https://doi.org/10.1038/s41746-025- 329 https://doi.org/10.1038/s41746-024-01336-w (2024). 02299-2 (2026). 16. Luo, M.-J. et al. A large language model digital patient system 35. Boursicot, K. et al. Conducting a high-stakes OSCE in a COVID-19 enhances ophthalmology history taking skills. npj Digital Med. 8, 502 environment. MedEdPublish 9, 54 https://doi.org/10.15694/mep. https://doi.org/10.1038/s41746-025-01841-6 (2025). 2020.000054.1 (2020). 17. Lavigne, E. et al. AI-standardized clinical examination training on 36. Chan, S. M., Choa, G., Kelly, J., Maru, D. & Rashid, M. A. OSCE performance. NEJM AI 2 https://doi.org/10.1056/AIoa2500066 Implementation of virtual OSCE in health professions education: a (2025). systematic review. Med. Educ. 57, 833–843 https://doi.org/10.1111/ 18. Yu, H. et al. Simulated patient systems powered by large language medu.15089 (2023). model-based AI agents offer potential for transforming medical 37. Brügge, E. et al. Large language models improve clinical decision education. Commun. Med. 6, 27 https://doi.org/10.1038/s43856- making of medical students through patient simulation and structured 025-01283-x (2025). feedback: a randomized controlled trial. BMC Med. Educ. 24, 1391 19. Haltaufderheide, J. & Ranisch, R. The ethics of ChatGPT in medicine https://doi.org/10.1186/s12909-024-06399-7 (2024). and healthcare: a systematic review on large language models. npj 38. Vasey, B. et al. Reporting guideline for the early-stage clinical Digital Med. 7, 183 https://doi.org/10.1038/s41746-024-01157-x evaluation of decision support systems driven by artificial intelligence: (2024). DECIDE-AI. Nat. Med. 28, 924–933 (2022). 20. Nestel, D. & Tierney, T. Role-play for medical students learning about 39. Kurtz, S., Silverman, J. & Draper, J. Teaching and learning communication: guidelines for maximising benefits. BMC Med. Educ. communication skills in medicine. (2003). 7, 3 https://doi.org/10.1186/1472-6920-7-3 (2007). 40. Frank, J. R. et al. Competency-based medical education: theory to 21. Collins, A., Brown, J. S. & Holum, A. Cognitive apprenticeship: making practice. Med. Teach. 32, 638–645 https://doi.org/10.3109/ thinking visible. Am. Educator 15, 6–11 (1991). 0142159X.2010.501190 (2010). 22. Zhang, G. et al. Closing the gap between open source and commercial large language models for medical evidence summarization. npj Acknowledgements Digital Med. 7, 239 https://doi.org/10.1038/s41746-024-01239-w The authors thank the expert evaluators for their contributions to the fidelity (2024). assessment, the graduate student standardised patient actors in Phase 2, 23. Wu, C. Towards evaluating and building versatile large language and all 92 participating medical students from Dalian Medical University. models for medicine. npj Digital Med. 8, 58 https://doi.org/10.1038/ This work was supported by the Centre for School Development and s41746-024-01390-4 (2025). Planning under the Ministry of Education (Grant No. CSDP26FC5A425) and 24. Jin, Q. et al. Hidden flaws behind expert-level accuracy of multimodal by the General Project of the Undergraduate Teaching Reform Research GPT-4 vision in medicine. npj Digital Med. 7, 190 https://doi.org/10. Programme of Liaoning Provincial Institutions of Higher Education (Grant 1038/s41746-024-01185-7 (2024). No. 2025YBXM0491). The funders had no role in study design, data 25. McGaghie, W. C., Issenberg, S. B., Cohen, E. R., Barsuk, J. H. & collection and analysis, decision to publish, or preparation of the manuscript. Wayne, D. B. Does simulation-based medical education with deliberate practice yield better results than traditional clinical Author contributions education? A meta-analytic comparative review of the evidence. P.W. conceptualised the study, led the clinical methodology, designed the Acad. Med. 86, 706–711 https://doi.org/10.1097/ACM. clinical scenarios and patient personas, curated the data, conducted the 0b013e318217e119 (2011). investigation, and wrote the original draft. Y.H. designed the study and its 26. Lancaster, G. A., Dodd, S. & Williamson, P. R. Design and analysis of three-layer information architecture, developed the cross-model prompt- pilot studies: recommendations for good practice. J. Evaluat. Clin. engineering framework, built the web-based AI-SP platform and automated Pract. 10, 307–312 (2004). analytics pipeline, and led the formal analysis, statistical analysis, and 27. Thabane, L. et al. A tutorial on pilot studies: the what, why, and how. visualisation. J.Z., Y.L., and M.J. developed and validated the clinical sce- BMC Med. Res. Methodol. 10, 1 https://doi.org/10.1186/1471-2288- narios, recruited participants, and delivered the training programme. X.L., 10-1 (2010). H.Z., D.X., H.M., and L.W. served as blinded examiners for the OSCE and the 28. Faul, F., Erdfelder, E., Lang, A.-G. & Buchner, A. G*Power 3: a flexible expert fidelity evaluation. Q.W. conceptualised and supervised the project, statistical power analysis program for the social, behavioral, and acquired funding, and reviewed and edited the manuscript. All authors biomedical sciences. Behav. Res. Methods 39, 175–191 https://doi. reviewed and approved the final manuscript. org/10.3758/BF03193146 (2007). 29. Bandura, A. Self-efficacy: toward a unifying theory of behavioral Funding change. Psychol. Rev. 84, 191–215 https://doi.org/10.1037/0033- The authors received no specific funding for this work. 295X.84.2.191 (1977). 30. Mehrabian, A. Silent Messages: Implicit Communication of Emotions Competing interests and Attitudes (Wadsworth, 1971). The authors declare no competing interests. npj Digital Medicine Article in Press https://doi.org/10.1038/s41746-026-03372-0 Article Additional information Open Access This article is licensed under a Creative Commons Supplementary information The online version contains Attribution-NonCommercial-NoDerivatives 4.0 International License, supplementary material available at which permits any non-commercial use, sharing, distribution and https://doi.org/10.1038/s41746-026-03372-0. reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Correspondence and requests for materials should be addressed to Commons licence, and indicate if you modified the licensed material. You Qingping Wen. do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party Reprints and permissions information is available at material in this article are included in the article's Creative Commons http://www.nature.com/reprints licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended Publisher’s note Springer Nature remains neutral with regard to use is not permitted by statutory regulation or exceeds the permitted use, jurisdictional claims in published maps and institutional affiliations. you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by- nc-nd/4.0/. © The Author(s) 2026 This Article in Press is shared early to give you faster access to new research. It is citable and carries a permanent DOI. The final edited version will replace it automatically. npj Digital Medicine Article in Press