Randomisierte Studie eines generativen KI-Chatbots für die psychische Gesundheitsbehandlung

NEJM AI 2025;2(4) DOI: 10.1056/AIoa2400802 ORIGINAL ARTICLE Randomized Trial of a Generative AI Chatbot for Mental Health Treatment Michael V. Heinz , M.D.,1,2 Daniel M. Mackin , Ph.D.,1,2 Brianna M. Trudeau , B.A.,1 Sukanya Bhattacharya , B.A.,1 Yinzhou Wang , M.S.,1 Haley A. Banta ,1 Abi D. Jewett , B.A.,1 Abigail J. Salzhauer , B.A.,1 Tess Z. Griffin , Ph.D.,1 and Nicholas C. Jacobson , Ph.D.1,2,3,4 Received: August 11, 2024; Revised: November 18, 2024; Accepted: February 2, 2025; Published March 27, 2025 Abstract BACKGROUND Generative artificial intelligence (Gen-AI) chatbots hold promise for build- ing highly personalized, effective mental health treatments at scale, while also addressing user engagement and retention issues common among digital therapeutics. We present a randomized controlled trial (RCT) testing an expert–fine-tuned Gen-AI–powered chatbot, Therabot, for mental health treatment. METHODS We conducted a national, randomized controlled trial of adults (N=210) with clinically significant symptoms of major depressive disorder (MDD), generalized anxi- ety disorder (GAD), or at clinically high risk for feeding and eating disorders (CHR-FED). Participants were randomly assigned to a 4-week Therabot intervention (N=106) or waitlist control (WLC; N=104). WLC participants received no app access during the study period but gained access after its conclusion (8 weeks). Participants were stratified into one of three groups based on mental health screening results: those with clinically significant symptoms of MDD, GAD, or CHR-FED. Primary outcomes were symptom changes from baseline to postintervention (4 weeks) and to follow-up (8 weeks). Secondary outcomes included user engagement, acceptability, and therapeutic alliance (i.e., the collaborative patient and thera- pist relationship). Cumulative-link mixed models examined differential changes. Cohen’s d effect sizes were unbounded and calculated based on the log-odds ratio, representing differ- ential change between groups. RESULTS Therabot users showed significantly greater reductions in symptoms of MDD (mean changes: −6.13 [standard deviation {SD}=6.12] vs. −2.63 [6.03] at 4 weeks; −7.93 [5.97] vs. −4.22 [5.94] at 8 weeks; d=0.845–0.903), GAD (mean changes: −2.32 [3.55] vs. −0.13 [4.00] at 4 weeks; −3.18 [3.59] vs. −1.11 [4.00] at 8 weeks; d=0.794–0.840), and CHR-FED (mean changes: −9.83 [14.37] vs. −1.66 [14.29] at 4 weeks; −10.23 [14.70] vs. −3.70 [14.65] at 8 weeks; d=0.627–0.819) relative to controls at postintervention and fol- The author affiliations are listed at low-up. Therabot was well utilized (average use >6 hours), and participants rated the thera- the end of the article. peutic alliance as comparable to that of human therapists. Dr. Heinz can be contacted at CONCLUSIONS This is the first RCT demonstrating the effectiveness of a fully Gen-AI ther- michael​.v.heinz@dartmouth​.edu apy chatbot for treating clinical-level mental health symptoms. The results were promising for or at the Center for Technology and Behavioral Health, Dartmouth MDD, GAD, and CHR-FED symptoms. Therabot was well utilized and received high user rat- College, 46 Centerra Pkwy, ings. Fine-tuned Gen-AI chatbots offer a feasible approach to delivering personalized mental Lebanon, NH 03766. NEJM AI is produced by NEJM Group, a division of the Massachusetts Medical Society. Downloaded from ai.nejm.org by a a on March 27, 2025. For personal use only. No other uses without permission. Copyright © 2025 Massachusetts Medical Society. All rights reserved. health interventions at scale, although further research with Recent advances in computing and machine learning larger clinical samples is needed to confirm their effective- now allow for sophisticated systems capable of learning, ness and generalizability. (Funded by Dartmouth College; adapting, and understanding context in natural language, ClinicalTrials.gov number, NCT06013137.) removing the necessity for explicit programming. Pushing these bounds even further in the language domain, the advent of generative AI (Gen-AI), recently popularized by ChatGPT,20 has enabled the automated production of novel Introduction and highly personalized responses to human input. To date, T conversational agents using Gen-AI have fallen under gen- he prevalence and burden of mental health dis- eral purpose, wellness, or companion applications,21 rather orders have increased significantly over the past than software intended for the diagnosis and treatment of three decades.1 Despite the adverse impact of mental health disorders. Although some Gen-AI-powered these disorders,2 mental health infrastructure is inade- chatbots have shown both broad appeal and the capacity to quately resourced to meet the current and growing demand form human-like bonds,22 they are not intended and have for care.3-5 Although empirically validated psychosocial not been evaluated for mental health treatment. The non- treatments exist,6-8 they are resource intensive, and limited deterministic and open-ended nature of Gen-AI, enabling in scalability and accessibility, leading to fewer than half of the possibility of harmful responses, has paused adop- the people with a mental health disorder receiving care.5,9 tion in mental health. Such risks associated with Gen-AI Digital therapeutics (DTx) — automated, evidence-based and related chatbots underscore the need for a systematic software for the treatment or diagnosis of medical condi- approach to exploring the safety of chatbots for use in men- tions10 — offer a solution to bridge this gap. tal health. While DTx aim to improve the accessibility and scalabil- Despite significant risks, there is potential benefit from ity of evidence-based mental health interventions,11 these the use of therapeutic Gen-AI–powered chatbots. Paired approaches have been plagued by attrition and low rates of with existing frameworks for DTx, Gen-AI chatbots have engagement.12 Within established psychotherapies, there is unprecedented potential to address existing problems with evidence for the benefit of nonspecific factors, such as ther- engagement while powering the development of new and apeutic alliance (i.e., the collaborative relationship between personalized interventions. Although the literature sup- patient and therapist), empathy, and shared goals.13 The rel- ports the effectiveness of cognitive behavioral therapy ative lack of personalization and alliance in DTx, compared (CBT)–based DTx and rule-based AI chatbots for depres- with human-delivered psychosocial interventions, is likely sion and anxiety, and Gen-AI chatbots exhibit promise for to contribute to reduced engagement.14 These nonspecific addressing issues of accessibility, scalability, engagement, factors have been difficult to emulate via automated tech- and personalization in mental health care, to our knowl- nologies and may be fundamentally different, or unachiev- edge, no prior RCTs have investigated the effectiveness able, in automated software.15 and safety of a Gen-AI chatbot for the treatment of mental health symptoms. Beginning in 2019, we started develop- Artificial intelligence (AI) represents a promising direction ing Therabot, a Gen-AI chatbot trained using expertly writ- for improving DTx. Chatbots hold particular promise given ten therapist–patient dialogues based on third-wave CBT,23 their capacity to imitate human conversation and dialogue, integrating empirically grounded contextual and functional long known to be integral parts of psychotherapeutic treat- approaches to mental health problems. Developed with ments — that is, the talking cure.16 In fact, mental health over 100,000 human hours comprising software develop- and wellness chatbots are not a new phenomenon, with ment, training dialogue creation, and refinement, Therabot ELIZA,17 an early rule-based chatbot, used to emulate a is designed to augment and enhance conventional mental Rogerian therapist. The study of chatbots for mental health health treatment services by delivering personalized, evi- remains nascent, with evaluation limited to exclusively denced-based mental health interventions at scale. rule-based conversational agents to date. Although such chatbots (e.g., Woebot) have shown benefits in clinical tri- In this RCT, we examined Therabot’s effectiveness for the als,18 and in some cases a capacity to promote a therapeu- treatment of major depressive disorder (MDD) symptoms, tic alliance,19 they are inherently limited by their reliance generalized anxiety disorder (GAD) symptoms, and clin- on an explicitly programmed decision trees and restricted ically high-risk feeding and eating disorder (CHR-FED) inputs. symptoms in a large, nationally representative sample of NEJM AI 2 NEJM AI is produced by NEJM Group, a division of the Massachusetts Medical Society. Downloaded from ai.nejm.org by a a on March 27, 2025. For personal use only. No other uses without permission. Copyright © 2025 Massachusetts Medical Society. All rights reserved. participants. We hypothesized that participants assigned to Therabot. Study data were collected and managed using to a 4-week intervention with Therabot would measur- REDCap electronic data capture tools hosted at Dartmouth ably improve in mental health symptoms across all clinical College.25,26 domains relative to patients assigned to the waitlist control (WLC) condition at both postintervention (4 weeks) and MODEL TRAINING AND INFERENCE follow-up (8 weeks). Furthermore, we hypothesized partic- We utilized transformer-based, decoder-only architectures. ipants would demonstrate a high level of engagement with The system employed both Falcon-7B and LLaMA-2-70B Therabot, rate Therabot positively, and develop a therapeu- models in tandem. The models were trained and deployed tic alliance with Therabot. on AWS SageMaker. During training, we applied quantized low-rank adaptation (QLoRA) for efficient fine-tuning to optimize the models’ generation of appropriate responses Methods based on conversation history. For inference, the conversa- tion history was used to prompt the fine-tuned models via TRIAL DESIGN SageMaker end points. The study was designed as a randomly assigned, WLC trial with a 1:1:1 allocation ratio across the MDD, GAD, PARTICIPANTS and CHR-FED symptom groups. A Meta Ads campaign The participants were required to be at least 18 years of was used to recruit adults across the United States. Given age and to screen positive for clinical-level symptoms of its broad and diverse reach, samples gathered via Meta at least one of the following: MDD, GAD, or CHR-FED. Ads have been shown to represent varied age, education, Participants who screened positive for CHR-FED were and gender groups.24 Based on self-reported responses given priority assignment until recruitment goals were met to a baseline questionnaire, with instruments detailed for that group. Otherwise, participants were assigned to the in the Supplementary Appendix, participants screening pathology group coinciding with their most severe screen- positive for MDD, GAD, or CHR-FED symptoms were ing measure. Exclusion criteria included active suicidality, stratified accordingly into one of the three groups and mania, and psychosis (see the Supplementary Appendix). then randomly assigned to either the control or interven- tion group. We assumed participant identity to be truth- ful unless we detected irregularities in identity data. We INTERVENTION automatically blocked identical email addresses, phone Therabot is a text-based multithread chat application for numbers, and IP addresses to prevent duplicates during iOS and Android that can interact with participants regard- sign-up. Furthermore, we used a custom-designed two-fac- ing their mental health problems in natural language. The tor authentication system built into REDCap to ensure that intervention utilizes a generative large language model participants indeed owned the phone number that they pro- (LLM) fine-tuned on expert-curated mental health dia- vided. We included a manual review in cases of uncertainty. logues. The dialogues were developed by our research Comorbidity was allowed, and outcomes were analyzed team, including a board-certified psychiatrist and a clinical based on participants scoring within the respective groups psychologist, and peer-reviewed using evidence-based (pri- at baseline, regardless of their primary presenting problem. marily CBT) modalities. Given the potential risks associ- ated with Gen-AI, we added multiple guard rails, including Participants randomly assigned to the intervention group a crisis classification model. were prompted daily to interact with Therabot during the treatment phase (first 4 weeks). During the subsequent For the first 4 weeks, participants in the intervention group postintervention follow-up phase (weeks 4–8), participants were prompted daily to engage with Therabot. During were not prompted but were permitted access to Therabot. the subsequent follow-up phase (weeks 4–8), participants Both groups, intervention, and WLC, received assessments engaged with Therabot as frequently as they desired. The at baseline, postintervention (4 weeks), and follow-up (8 application allowed users to either initiate a session directly weeks). Therabot access was disabled in the treatment in the chat interface or to respond to scheduled notifications. group after 8 weeks; owing to a 72-hour grace period and In response to a user prompt, conversation history, and the variations across time zones, the actual access duration most recent user message were combined and sent to the was potentially up to 60 days. After completing their final LLM. Common output included empathetic responses, assessment, the WLC group was also provided full access validation, targeted interventions, or questions prompting NEJM AI 3 NEJM AI is produced by NEJM Group, a division of the Massachusetts Medical Society. Downloaded from ai.nejm.org by a a on March 27, 2025. For personal use only. No other uses without permission. Copyright © 2025 Massachusetts Medical Society. All rights reserved. A B C D Figure 1. Key Design Features of the Therabot Application. Panel A shows the Therabot login screen. Panel B shows the main chat interface. Panel C shows the emergency module deployed in response to model detection of high-risk content (e.g., suicidal ideation). Panel D shows the conversation thread interface for users to initiate a thread or return to a prior thread. elaboration (see Fig. 1). All responses from Therabot were baseline risk stratification tool to ensure we represented supervised by trained clinicians and researchers post-trans- participants with CHR-FED. mission. In the event of an inappropriate response from The PHQ-9 sum score ranges from 0 to 27, with a decrease Therabot (e.g., providing medical advice), we contacted of 5 or more shown to constitute a clinically meaningful the participant to provide correction. In the event of a par- change30 and a score of 10 or more often used as a clinical ticipant raising safety concerns (e.g., suicidal ideation), we threshold for MDD screening. The GAD-Q-IV, designed contacted the participant to provide safety guidance and to assess GAD based on DSM-IV and DSM-5 criteria, can emergency resources. be scored via binary criterion method or a total sum score ranging from 0 to 13. The WCS score ranges from 0 to 100 MEASURES and constitutes a mean of five Likert-style items, normal- Primary outcome measures were administered to both ized from 0 to 100.29 Although the PHQ-9 has established groups at baseline, postintervention (4 weeks), and clinically meaningful change thresholds, the GAD-Q-IV follow-up (8 weeks) and included the Patient Health and WCS have not; therefore, we use Cohen’s d values for Questionnaire 927 (PHQ-9), the Generalized Anxiety primary effect sizes, given their widespread use in evaluat- Disorder Questionnaire for the Diagnostic and Statistical ing psychiatric trials. Manual of Mental Disorders, Fourth Edition (DSM-IV)28 Secondary outcomes included therapeutic alliance (GAD-Q-IV), and the Weight Concerns Scale (WCS) within (Working Alliance Inventory — Short Revised31 [WAI-SR]), the Stanford–Washington University Eating Disorder29 engagement with Therabot (number of messages sent), and (SWED), as measures of depression, anxiety, and weight satisfaction with Therabot (self-developed survey). Each concerns, respectively. The SWED was used in full as a domain of the WAI-SR (goal, task, and bond) is scored as the NEJM AI 4 NEJM AI is produced by NEJM Group, a division of the Massachusetts Medical Society. Downloaded from ai.nejm.org by a a on March 27, 2025. For personal use only. No other uses without permission. Copyright © 2025 Massachusetts Medical Society. All rights reserved. mean of four Likert-style items, resulting in a score range of individual differences in the outcome. The logit link func- 1–5. Additional details on the primary and secondary out- tion was used, which was well suited for proportional odds comes are presented in the Supplementary Appendix. models, and thresholds were assumed to be symmetric around the latent mean of zero. Formally, the model is SAMPLE SIZE JUSTIFICATION described as follows: Based on the comorbidity between diagnoses, we expected logit(P( Y ≤ j Time,Group,ID)) to have 100 participants in each analysis. A Monte-Carlo = α j −(β1 ×Time + β2 × Group + β3 × Time× Group + uID ) simulation study was used to estimate the statistical power (1) for the differential change in treatment response. For each simulated dataset (N=80–150), we generated ordi- where Y represents the ordinal outcome (MDD, GAD, or nal data such that the treatment group showed differential WCS score), j denotes the threshold category, αj are the changes (d=0.3–0.5) over time. We assumed 10% missing threshold parameters, and β1, β2, and β3 are the fixed-effect data. We fit cumulative-link mixed models (CLMMs; see coefficients for time, group, and their interaction, respec- Statistical Methods below), with individual differences as tively. The coefficients of primary interest are the β3 terms random intercepts. The interaction effects between time representing the differential change between groups from and randomly assigned group determined power, with the baseline at each follow-up time point. Here, uID signifies the proportion of times the interaction terms were significant random intercepts for the participants. Odds ratios were representing power. The results suggested that we had calculated from these estimates, and effect sizes were cal- greater than 90% power to detect a differential response of culated using d=log(OR)×sqrt(3)/π.32 All Cohen’s d effect 0.3 and 0.5 sizes were calculated between groups representing differen- tial change. The effect sizes are unbounded. All treatment RANDOM ASSIGNMENT effects reported in our analyses are marginal effects derived Random assignment was performed using a computer-gen- from CLMMs. These models provide predicted probabilities erated random sequence with a fixed block size of six. The for each possible outcome score under each condition (treat- random assignment sequence was generated prior to the ment/control) at each time point. To obtain interpretable trial start, and the assignment process was fully automated. statistics, we drew 10,000 samples from these model-pre- Once group assignment occurred, neither researchers nor dicted probability distributions, which provided stable esti- participants were blinded to their group membership. mates of means and standard deviations rather than using raw scores.33,34 For example, with depression symptoms (PHQ-9), the model predicts the probability of each possi- STATISTICAL METHODS ble score (0–27) for both groups at baseline, postinterven- To examine the effectiveness of Therabot relative to the tion and follow-up. The change scores were calculated by waitlist control group, we examined the effect of time and comparing these sampled distributions across time points. treatment assignment on depression, anxiety, and weight This approach allows us to honor the ordinal nature of our concerns among participants at a clinical level of MDD, measures while also providing clinically meaningful statis- GAD, and CHR-FED at baseline. The participants were tics, with effect sizes computed directly from the model’s analyzed within the groups to which they were randomly log-odds ratios and supporting statistics (means, standard assigned. Owing to the trivial percentage of missing out- deviations [SDs]) derived from the sampling procedure. come data, we used complete case analysis. To ensure To control for multiple comparisons across our six primary complete case analysis provided an unbiased estimate, we end points (three outcomes measured at two time points), also analyzed the data using multiple imputation for miss- we implemented the Holm–Bonferroni sequential proce- ing data handling. The results were nearly identical, with dure for familywise error rate control. Adjusted P values are no changes to the interpretation, and are displayed in the reported for all primary analyses (see Table 1). Supplementary Appendix. Descriptive results for working alliance and user satisfac- To address the ordinal, nonequidistant nature in the tion were visualized using box plots. In each plot, the box response categories of our symptom measures and elimi- represents the interquartile range (IQR), with the median nate potential distortion of effect size estimates and inflated presented as a horizontal line. Whiskers extend to the small- error rates, we used CLMMs to analyze the effects of time est and largest values within 1.5×IQR from the hinges. Data and random assignment as fixed effects, and random beyond the whiskers are plotted as individual outliers. NEJM AI 5 NEJM AI is produced by NEJM Group, a division of the Massachusetts Medical Society. Downloaded from ai.nejm.org by a a on March 27, 2025. For personal use only. No other uses without permission. Copyright © 2025 Massachusetts Medical Society. All rights reserved. Table 1. Mean Primary Outcomes (Standard Deviation) at Each Observation Time Point across the Study.* Intervention Control Between-Group Differences in Change Adjusted Outcome and Time Point M (SD) ΔM (SD) N M (SD) N ΔM (SD) β (SE) d (95% CI) P Value† MDD symptoms (as measured by PHQ-9) Baseline 15.63 (4.33) 73 15.91 (4.45) 69 Postintervention 9.50 (7.50) −6.13 (6.12) 70 13.28 (7.50) 69 −2.63 (6.03) −1.533 (0.404) 0.845 <0.001 (4 weeks) (0.409 to 1.282) Follow-up (8 weeks) 7.70 (7.38) −7.93 (5.97) 70 11.69 (7.43) 69 −4.22 (5.94) −1.639 (0.410) 0.903 <0.001 (0.460 to 1.347) GAD symptoms (as measured by GAD-Q-IV) Baseline 10.43 (1.14) 60 10.42 (1.13) 56 Postintervention 8.11 (3.73) −2.32 (3.55) 53 10.30 (4.16) 56 −0.13 (4.00) −1.523 (0.424) 0.840 0.001 (4 weeks) (0.382 to 1.298) Follow-up (8 weeks) 7.24 (3.77) −3.18 (3.59) 52 9.31 (4.16) 56 −1.11 (4.00) −1.441 (0.432) 0.794 0.003 (0.328 to 1.261) Weight concerns (as measured by WCS) Baseline 54.13 (11.38) 42 54.65 (16.41) 47 Postintervention 44.30 (18.33) −9.83 (14.37) 42 52.99 (21.76) 47 −1.66 (14.29) −1.485 (0.513) 0.819 0.008 (4 weeks) (0.264 to 1.373) Follow-up (8 weeks) 43.90 (18.60) −10.23 (14.70) 42 50.95 (22.00) 47 −3.70 (14.65) −1.137 (0.514) 0.627 0.027 (0.072 to 1.182) * Marginal effects were estimated using cumulative-link mixed models based on model-derived probabilities. Sample sizes analyzed at each time point are provided in the column immediately to the right of the mean. CI denotes confidence interval; GAD-Q-IV, Generalized Anxiety Disorder Questionnaire for the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition; MDD, major depressive disorder; PHQ-9, Patient Health Questionnaire 9; SD standard deviation; and WCS, Weight Concerns Scale. † P values were adjusted using the Holm–Bonferroni method for multiple comparisons across six primary end points. COMPENSATION PRIMARY OUTCOMES Participants were compensated $25 (U.S. dollars) for each of the three assessments completed. Major Depressive Disorder Symptoms Participants receiving the Therabot intervention showed large and greater improvement in depression symptoms compared with the control participants across both time Results points (see Table 1), with both comparisons remain- ing significant after adjusting for multiple comparisons. PARTICIPANTS The mean change (SD) in PHQ-9 score from baseline to Participants included 210 adults, randomly assigned to inter- postintervention was –6.13 (6.12) in the intervention vention and WLC. By the 4-week assessment, four partici- group and −2.63 (6.03) in the control group. The change pants had withdrawn from the study (treatment group), and from baseline to follow-up was −7.93 (5.97) in the inter- three were lost to follow-up (treatment group); six of these vention group and −4.22 (5.94) in the control group participants had opted to discontinue the Therabot inter- (Fig. 3, Row 1). vention. By the 8-week assessment, one participant was lost to follow-up (treatment group). Detailed aggregate demo- Generalized Anxiety Disorder Symptoms graphics are displayed in Table 2. Figure 2 displays the flow Similar patterns of improvement were observed for anx- of participants from screening to analysis, including counts iety symptoms (Table 1), where participants receiving of participants meeting clinical screening thresholds for each the Therabot intervention showed a large and differential group. The study spanned March through May 2024, with response in anxiety symptoms across both postintervention recruitment from March 15–31, 2024. Recruitment ended and follow-up. The mean change (SD) in GAD-Q-IV score when the enrollment target was reached. from baseline to postintervention was −2.32 (3.55) in the NEJM AI 6 NEJM AI is produced by NEJM Group, a division of the Massachusetts Medical Society. Downloaded from ai.nejm.org by a a on March 27, 2025. For personal use only. No other uses without permission. Copyright © 2025 Massachusetts Medical Society. All rights reserved. Table 2. Baseline Characteristics of the Study Sample (N=210), by Intervention Group (n=106) and Waitlist Control Group (n=104).* Characteristic Waitlist Control (n=104) Intervention (n=106) Overall (N=210) Age (years) — mean (SD) 33.63 (10.56) 34.09 (11.41) 33.86 (10.97) Gender — n (%) Male 37 (35.58) 41 (38.68) 78 (37.14) Female 62 (59.62) 63 (59.43) 125 (59.52) Nonbinary 4 (3.85) 2 (1.89) 6 (2.86) Other 1 (0.96) 0 (0.00) 1 (0.48) Transgender — n (%) Yes 5 (4.81) 3 (2.83) 8 (3.81) No 99 (95.19) 103 (97.17) 202 (96.19) Sexual orientation — n (%) Heterosexual 82 (78.85) 84 (79.25) 166 (79.05) Homosexual/gay 3 (2.88) 9 (8.49) 12 (5.71) Bisexual 11 (10.58) 10 (9.43) 21 (10.00) Pansexual 3 (2.88) 1 (0.94) 4 (1.90) Asexual 0 (0.00) 1 (0.94) 1 (0.48) Bicurious 1 (0.96) 0 (0.00) 1 (0.48) Other 4 (3.85) 1 (0.94) 5 (2.38) Race or Ethnicity — n (%)* Non-Hispanic White 56 (53.85) 56 (52.83) 112 (53.33) Hispanic White 7 (6.73) 9 (8.49) 16 (7.62) Black 28 (26.92) 26 (24.53) 54 (25.71) American Indian 0 (0) 1 (0.94) 1 (0.48) Asian 5 (4.81) 6 (5.66) 11 (5.24) Multiple/other 8 (7.69) 8 (7.55) 16 (7.62) Highest level of education — n (%) High school 4 (3.85) 7 (6.60) 11(5.24) Some college 14 (13.46) 19 (17.92) 33 (15.71) Associate’s degree 23 (22.12) 17 (16.04) 40 (19.05) Bachelor’s degree 45 (43.27) 44 (41.51) 89 (42.38) Master’s degree 15 (14.42) 16 (15.09) 31 (14.76) Doctoral degree 3 (2.88) 3 (2.83) 6 (2.86) Current student, n (%) No 78 (75.00) 77 (72.64) 155 (73.81) Yes — part time 12 (11.54) 11 (10.38) 23 (10.95) Yes — full time 14 (13.46) 18 (16.98) 32 (15.24) Current treatment, n (%) Medication 23 (22.12) 24 (22.64) 47 (22.38) Psychotherapy 16 (15.38) 13 (12.26) 29 (13.81) Medication and psychotherapy 10 (9.62) 7 (6.60) 17 (8.10) No treatment 75 (72.12) 79 (74.53) 154 (73.33) *Race was reported by the participants. intervention group and −0.13 (4.00) in the control group. exhibiting a large differential response across both time The change from baseline to follow-up was −3.18 (3.59) points, with comparisons remaining significant after in the intervention group and −1.11 (4.00) in the control adjusting for multiple comparisons. The mean change (SD) group (Fig. 3, Row 2). in WCS score from baseline to postintervention was −9.83 (14.37) in the intervention group and −1.66 (14.29) in Weight Concerns the control group. The change from baseline to follow-up The intervention group showed significantly greater was −10.23 (14.70) in the intervention group and −3.70 improvements in weight concerns than the control group, (14.65) in the control group. (Fig. 3, Row 3). NEJM AI 7 NEJM AI is produced by NEJM Group, a division of the Massachusetts Medical Society. Downloaded from ai.nejm.org by a a on March 27, 2025. For personal use only. No other uses without permission. Copyright © 2025 Massachusetts Medical Society. All rights reserved. THERABOT CONSORT FLOW DIAGRAM SCREENED INELIGIBLE BY INCLUSION CRITERIA n=1707 Did not complete screening (n=435) n=743 Ineligible by inclusion criteria (n=308) INELIGIBLE BY EXCLUSION CRITERIA High risk for clinical eating D/O (n=480) ELIGIBLE n=754 High risk for suicide (n=215) n=210 68% with MDD (n=142) High risk for mania (n=58) 55% with GAD (n=116) High risk for psychosis (n=1) 42% with CHR-FED (n=89) RANDOM ASSIGNMENT WAITLIST CONTROL INTERVENTION 4-WEEK DID NOT COMPLETE ASSESSMENT n=104 n=99 n=7 Lost to follow-up (n=3) ASSESSMENT Discontinued assessments (n=4) 8-WEEK DID NOT COMPLETE ASSESSMENT n=104 n=98 n=1 ASSESSMENT Lost to follow-up (n=1) 66% with MDD (n=69) 69% with MDD (n=73) 54% with GAD (n=56) 57% with GAD (n=60) 45% with CHR-FED (n=47) 40% with CHR-FED (n=42) DISCONTINUED INTERVENTION (n=6) Figure 2. CONSORT Flow Diagram Showing Movement of Participants through the Study, with Associated Counts. As comorbidity is the norm rather than the exception, data were analyzed based on each participant’s pathology group membership at baseline. CHR-FED denotes clinically high risk for feeding and eating disorders; CONSORT, Consolidated Standards of Reporting Trials; D/O, disorder; GAD, generalized anxiety disorder; and MDD, major depressive disorder. SECONDARY OUTCOMES mean score for Goal was 3.59 (1.35). Box plots for work- ing alliance scores are displayed in Figure 4. Participants, User Working Alliance on average, reported a therapeutic alliance comparable to The user working alliance, measured via the WAI-SR and norms reported in an outpatient psychotherapy sample.31 offered to all participants in the treatment group who inter- acted at least once with Therabot at 4 weeks, was com- User Satisfaction pleted by 96 participants. The overall mean WAI score (SD) The user satisfaction survey was offered to all those in the was 3.59 (1.27); the mean (SD) score for Bond was 3.71 treatment group who used Therabot, with 96 participants (1.28); the mean score for Task was 3.47 (1.30); and the completing the survey. Participants rated their experience NEJM AI 8 NEJM AI is produced by NEJM Group, a division of the Massachusetts Medical Society. Downloaded from ai.nejm.org by a a on March 27, 2025. For personal use only. No other uses without permission. Copyright © 2025 Massachusetts Medical Society. All rights reserved. Treatment Control PHQ-9: Probability of Change from Preintervention to Postintervention PHQ-9: Probability of Change from Preintervention to Follow-Up Between Group Pre–Post Randomly Assigned Group: Between Group Pre–Follow-up Randomly Assigned Group: Odds Ratio=0.216, P=0.000, and d=–0.845 Odds Ratio=0.194, P=0.000, and d=–0.903 0.06 0.06 Probability Probability 0.04 0.04 0.02 0.02 0.00 0.00 –20 –10 0 10 –20 –10 0 10 Change in PHQ-9 Change in PHQ-9 GAD-Q-IV: Probability of Change from Preintervention to Postintervention GAD-Q-IV: Probability of Change from Pre–intervention to Follow-Up Between Group Pre–Post Randomly Assigned Group: Between Group Pre–Follow-up Randomly Assigned Group: Odds Ratio=0.218, P=0.000, and d=–0.840 Odds Ratio=0.237, P=0.001, and d=–0.794 0.12 0.09 Probability Probability 0.09 0.06 0.06 0.03 0.03 0.00 0.00 –15 –10 –5 0 5 10 –15 –10 –5 0 5 10 Change in GAD-Q-IV Change in GAD-Q-IV WCS: Probability of Change from Preintervention to Postintervention WCS: Probability of Change from Preintervention to Follow-Up Between Group Pre–Post Randomly Assigned Group: Between Group Pre–Follow-up Randomly Assigned Group: Odds Ratio=0.227, P=0.004, and d=–0.819 Odds Ratio=0.321, P=0.027, and d=–0.627 0.05 0.05 Probability Probability 0.04 0.04 0.03 0.03 0.02 0.02 0.01 0.01 0.00 0.00 –50 –25 0 25 –50 –25 0 25 Change in WCS Change in WCS Figure 3. Distributions Representing Smoothed Probability of Changes in Clinical Outcomes (Depression, Anxiety, Weight Concerns, Row-Wise) Postintervention (4 Weeks, Left Column) and at Follow-Up (8 Weeks, Right Column). The probabilities shown in these plots are derived directly from the CLMMs through predicted probabilities for each possible change score under treatment and control conditions. The treatment group is visualized in blue, and the control group is visualized in red. CLMM denotes cumulative-link mixed model; GAD-Q-IV, Generalized Anxiety Disorder Questionnaire for the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition; PHQ-9, Patient Health Questionnaire 9; and WCS, Weight Concerns Scale. on a seven-point Likert scale, with seven being the high- with Therabot being 24 days (min.=1, max.=60). The est. Users rated Therabot as easy to learn to use (6.42, mean total amount of time participants interacted with SD=1.18) and intuitive (5.58, SD=1.58). On average, Therabot was 6.18 hours across the course of the study. users liked the interface (5.46, SD=1.93) and design (5.53, Participant engagement over the 4-week treatment phase SD=1.98). Users reported feeling better after interaction is depicted in Figure 6. Participant engagement over the (5.39, SD=1.84) and found the Therabot sessions help- full study period, including follow-up, is depicted in the ful (5.44, SD=1.82). Users reported that they would use Supplementary Appendix. Post-transmission staff interven- Therabot on their own (5.12, SD=2.02) and rated Therabot tion was required 15 times for participant safety concerns as similar to a real therapist (4.90, SD=2.21). On average, (e.g., expressions of suicidal ideation) and 13 times to cor- overall satisfaction was 5.30 (SD=1.89). Box plots for user rect inappropriate responses provided by Therabot (e.g., satisfaction ratings are shown in Figure 5. providing medical advice). User Engagement Of participants randomly assigned to the Therabot group, Discussion 101 (95%) interacted with Therabot. The mean num- As the first RCT of its kind, our study supports the fea- ber of messages sent by participants was 260 (min.=1, sibility, acceptability, and effectiveness of a fine-tuned, max.=1557), with the mean number of days interacting fully GenAI–powered chatbot for treating mental health NEJM AI 9 NEJM AI is produced by NEJM Group, a division of the Massachusetts Medical Society. Downloaded from ai.nejm.org by a a on March 27, 2025. For personal use only. No other uses without permission. Copyright © 2025 Massachusetts Medical Society. All rights reserved. 5 4 Average Score 3 2 1 WAI–Bond WAI–Task WAI–Goal Average Score Subscale Figure 4. Box Plots of Aggregate Working Alliance Inventory Subscale Scores with Average Subscale Score (Right). Box plots display the median (line), interquartile range (IQR) (box), and values within 1.5×IQR (whiskers). For comparison, the outpatient norms from Munder et al.31 are as follows: Bond (M=4.0, SD=0.78), Task (M=3.4, SD=0.77), Goal (M=4.0, SD=0.68), and Average (M=3.8, SD=0.63). IQR denotes interquartile range; SD, standard deviation; and WAI, Working Alliance Inventory. symptoms. Users demonstrated sustained engagement see such a bond as inherently limited compared with that and rated their alliance with Therabot as comparable possible with Gen-AI–powered agents; Gen-AI provides to human therapists during the 4-week trial. Critically, greater capacity for personalized adaptation and more as compared with the WLC, Therabot users showed a closely resembles human–human interaction. Our results greater reduction in depression, anxiety, and CHR-FED suggest that, within 4 weeks, participants were able to symptoms at postintervention (4 weeks) and at follow-up develop a working alliance comparable to that shown in (8 weeks). We posit that Therabot’s success is driven by an outpatient psychotherapy sample,32 with use at consis- three main factors. First, akin to effective rule-based con- tently high rates. versational agents,18 Therabot is rooted in evidence-based psychotherapies for anxiety,35 depression,36 and weight Although existing companion Gen-AI chatbots can be concerns.37 Second, users had unrestricted access to highly engaging, they are not trained or evaluated for treat- Therabot, allowing for any time–anywhere interactions. ing clinical-level mental health symptoms. Such chatbots Notably, the ability to access therapeutic support when may also be compromised by competing interests, such as most needed, regardless of the time or location, may be user engagement or profit, which may be at odds with best one of the most significant advantages of DTx. Third, practices for treatment.39 Therefore, Gen-AI conversational unlike existing chatbots for mental health treatment, agents tailored to integrate both evidence-based techniques Therabot was powered by Gen-AI, allowing for natural, and important nonspecific factors contributing to psycho- highly personalized, open-ended dialogue. Moreover, we therapy outcomes represent a significant opportunity to argue that the Gen-AI approach promoted the therapeu- provide scalable, on-demand, and effective mental health tic alliance, a critical nonspecific mediator of change in treatment. The nascency of Gen-AI and associated risks psychotherapy.38 Although some evidence supports devel- have likely contributed to the absence of a clinically vali- oping a therapeutic alliance with rule-based agents,19 we dated Gen-AI chatbot for mental health treatment. Indeed, NEJM AI 10 NEJM AI is produced by NEJM Group, a division of the Massachusetts Medical Society. Downloaded from ai.nejm.org by a a on March 27, 2025. For personal use only. No other uses without permission. Copyright © 2025 Massachusetts Medical Society. All rights reserved. 6 4 Rating 2 e e ce n r n ist n d fte us us fie ig sio ow fa p s ra tis ra er to to de es n he tte t to sa ys in rn e ke v lt be bo a am iti ap e Li le a th tu re at el er to ll In ke ch Fe th ke ra sy Li ve Li e ul Ea us pf O el ld H ou W Variable Figure 5. Box Plots Showing the Distribution of User Ratings across Satisfaction Variables. Box plots display the median (line), interquartile range (IQR) (box), and values within 1.5×IQR (whiskers). Points beyond the whiskers represent outliers. IQR denotes interquartile range the nondeterministic nature of Gen-AI models introduces Our results have important implications, forming the early the possibility of hallucinations and incorrect or potentially foundational evidence for the use of fine-tuned Gen-AI– harmful content.22 While human-delivered therapy is not powered chatbots in mental health treatment. Therabot immune to patient iatrogenesis,40 such effects in Gen-AI shows promise as a means to scale evidence-based thera- models have the potential to impact more people and are pies in a way that maintains a high degree of personaliza- less regulated. tion and engagement. Furthermore, our approach enables novel translations of therapeutic techniques that are not We emphasize the need to understand Gen-AI’s potential possible in rule-based agents. Consider, for instance, role and risks associated with mental health treatment and detailed and personalized imaginal exposures prompted by also the need for guardrails and close human supervision Gen-AI agents. Interventions dependent on the therapeutic while testing such methods. All content was closely super- alliance or specific patient–therapist interactions may also vised for quality and safety in our trial, with rapid expert benefit from the integration of Gen-AI chatbots. intervention available. This approach may continue to be necessary when testing similar future models to ensure Our study has notable strengths, including a nationally safety. In addition, given the inscrutable black-box nature recruited, demographically diverse, moderate sample size. of Gen-AI models, the inner processes are difficult or Furthermore, unlike many digital mental health studies,41 impossible to understand analytically. In this way, Gen-AI Therabot ran on both Android and iOS devices, increasing models are similar to human minds — intractable in com- generalizability. However, we also acknowledge several plexity and predominantly studied by the data they produce limitations. First, given our recruitment strategy, there — and thus may require extensive observation to obtain a was potential for selection bias toward younger, more reliable assessment. technologically minded participants who were open to AI. NEJM AI 11 NEJM AI is produced by NEJM Group, a division of the Massachusetts Medical Society. Downloaded from ai.nejm.org by a a on March 27, 2025. For personal use only. No other uses without permission. Copyright © 2025 Massachusetts Medical Society. All rights reserved. Number of Messages Per Day 100+ Participant ID 30 10 1 0 10 20 Days in Study Figure 6. Heat Map Representing User Engagement across Days in the Study (x-Axis), by Participant (y-Axis) The color represents the number of messages sent per day. However, this may also resemble the most likely end users. range of psychopathologies treated (e.g., obsessive–com- Relatedly, up-front disclosure of a monetary incentive may pulsive disorders), the settings in which Gen-AI chatbots are have introduced selection bias, favoring participants who provided (e.g., emergency rooms), and the role of the Gen- were motivated financially. Second, the characteristic of AI chatbot (e.g., adjunctive to in-person psychotherapy). WLC RCTs is that there was potential for differential con- Furthermore, future work should build on our descriptive tact between the intervention and control group. We helped user engagement metrics to define clinical thresholds per- mitigate this by planning equivalent contact between taining to minimal doses of Gen-AI–driven therapies. Our groups whenever possible. Last, our follow-up period was study provides key groundwork for the development of limited to 4 weeks postintervention (weeks 4–8). While this Gen-AI chatbots for mental health treatment. allowed for testing early effectiveness and safety, longer studies are needed to assess the durability of Therabot’s Disclosures effectiveness. Author disclosures and other supplementary materials are available at Overall, results from the Therabot RCT are highly prom- ai​.nejm​.org. ising. We found high engagement and acceptability of the Supported by Dartmouth College. intervention, as well as symptom decreases while main- taining a therapeutic alliance comparable to that of human We are extraordinarily grateful for the efforts of the many dedicated therapists and their patients. Future work may extend the people who made Therabot possible. We thank those who contributed NEJM AI 12 NEJM AI is produced by NEJM Group, a division of the Massachusetts Medical Society. Downloaded from ai.nejm.org by a a on March 27, 2025. For personal use only. No other uses without permission. Copyright © 2025 Massachusetts Medical Society. All rights reserved. in meaningful ways to the creation and curation of training data, 5. Kohn R, Ali AA, Puac-Polanco V, et al. Mental health in the including Victor A. Moreno, Chloe S. Park, Jimena Abejon Fuertes, Americas: an overview of the treatment gap. Rev Panam Salud Jonathan J. Cartwright, Anna C. St. Jean, Erica L. Simon, Isabel R. Pública 2018;42:e165. DOI: 10.26633/RPSP.2018.165. Hillman, Enoc A. Garza, Alexandra N. Limb, Dawson D. Haddox, 6. Monteleone AM, Pellegrino F, Croatto G, et al. Treatment of eating Mingyue Zha, Camilla M. Lee, Rachita Batra, MK Song, Cameron disorders: a systematic meta-review of meta-analyses and network M. Hasund, Avijit Singh, Daniel W. Shen, Rachel E. Quist, Kaitlyn I. meta-analyses. Neurosci Biobehav Rev 2022;142:104857. DOI: Romanger, Chaehyun Lee, Anjali G. Dhar, Ivy N. Mayende, Eleanor 10.1016/j.neubiorev.2022.104857. M. Rodgers, Rachel Zhang, Jenny Song, Veronica E. Abreu, Russell T. Rapaport, Mary M. Basilious, Sofia M. Yawand-Wossen, Nathan J. 7. Cuijpers P, Karyotaki E, Eckshtain D, et al. Psychotherapy for Kung, Jenny Y. Oh, Ashna J. Kumar, Eda Naz Gokdemir, Janelle E. depression across different age groups: a systematic review and Annor, Ganza, Belise Aloysie Isingizwe, Chloe N. Malave, Ezinne E. meta-analysis. JAMA Psychiatry 2020;77:694. DOI: 10.1001/ Anozie, Tara L. Karim, Nhi D. Nguyen, Krista E. Schemitsch, Helen jamapsychiatry.2020.0164. M. Young, Mia G. Russo, Rachel E. Quist, Tonya I. Tolino, Mckenzi 8. Van Dis EAM, Van Veen SC, Hagenaars MA, et al. Long-term B. Popper, Daniel G. Amoateng, and Dr. Seo Ho (Michael) Song. We outcomes of cognitive behavioral therapy for anxiety-related dis- thank those who contributed in meaningful ways to the software devel- orders: a systematic review and meta-analysis. JAMA Psychiatry opment, including Jason Kim, John F. Keane, Dr. George D. Price, Dr. 2020;77:265. DOI: 10.1001/jamapsychiatry.2019.3986. Matthew D. Nemesure, Ore E. James, Caroline C. Hall, Brendan W. Keane, Lisa Aeri Oh, Ly H. Nguyen, Dr. William R. Haslett, Vivian 9. Coombs NC, Meriwether WE, Caringi J, Newcomer SR. Barriers to N. Tran, Alexander M. Ye, Atziri Enriquez, Sarah M. Chacko, Sofia healthcare access among U.S. adults with mental health challenges: Jayaswal, D.J. M. Matusz, Jose Hernandez Barbosa, Alyssia M. Salas, a population-based study. SSM Popul Health 2021;15:100847. Ella J. Gates, and Tianwen Chen. We thank the team from Amazon DOI: 10.1016/j.ssmph.2021.100847. Web Services (AWS), especially Stefan Mationg and Dr. Jianjun Xu, 10. Fürstenau D, Gersch M, Schreiter S. Digital therapeutics who provided valuable technical support for Therabot. (DTx). Bus Inf Syst Eng 2023;65:349-360. DOI: 10.1007/ All participants provided informed written consent prior to their partic- s12599-023- ​00804-z. ipation. The Dartmouth Hitchcock Institutional Review Board approved 11. Wang C, Lee C, Shin H. Digital therapeutics from bench to bedside. the research protocol. The trial was preregistered through ClinicalTrials. NPJ Digit Med 2023;6:38. DOI: 10.1038/s41746-023-00777-z. gov as NCT06013137. 12. Nwosu A, Boardman S, Husain MM, Doraiswamy PM. Digital therapeutics for mental health: is attrition the Achilles heel? Author Affiliations Front Psychiatry 2022;13:900615. DOI: 10.3389/fpsyt.2022.​ 1 Center for Technology and Behavioral Health, Geisel School of 900615. Medicine, Dartmouth College, Lebanon, NH 2 Department of Psychiatry, Geisel School of Medicine, Dartmouth 13. Huibers M, Cuijpers P. Common (nonspecific) factors in psychother- College, Hanover, NH apy. 2015:1–6. In: Cautina RL, Lilienfeld SO, eds. The encyclope- 3 Quantitative Biomedical Sciences Program, Dartmouth College, dia of clinical psychology. 2nd Edn. Hoboken, NJ: Wiley-Blackwell. Hanover, NH DOI:10.1002/9781118625392.wbecp272 4 Department of Biomedical Data Science, Geisel School of Medicine, Dartmouth College, Lebanon, NH 14. Henson P, Peck P, Torous J. Considering the therapeutic alli- ance in digital mental health interventions. Harv Rev Psychiatry 2019;27:268-273. DOI: 10.1097/HRP.0000000000000224. References 15. Tong F, Lederman R, D’Alfonso S, Berry K, Bucci S. Digital ther- 1. GBD 2019 Mental Disorders Collaborators. Global, regional, and apeutic alliance with fully automated mental health smartphone national burden of 12 mental disorders in 204 countries and ter- apps: a narrative review. Front Psychiatry 2022;13:819623. DOI: ritories, 1990–2019: a systematic analysis for the Global Burden 10.3389/fpsyt.2022.819623. of Disease Study 2019. Lancet Psychiatry 2022;9:137-150. DOI: 10.1016/S2215-0366(21)00395-3. 16. Breuer J, Freud S. Studies on hysteria. Oxford: Basic Books, 1957. 2. Cloninger CR. The science of well-being: an integrated approach to 17. Weizenbaum J. ELIZA — a computer program for the study of natu- mental health and its disorders. World Psychiatry 2006;5:71-76. ral language communication between man and machine. Commun 3. National Center for Health Workforce Analysis. Behavioral health ACM 1966;9:36-45. DOI: 10.1145/365153.365168. workforce, 2023. Rockville, MD: Bureau of Health Workforce, 18. Fitzpatrick KK, Darcy A, Vierhile M. Delivering cognitive behav- December, 2023. ior therapy to young adults with symptoms of depression and 4. Health Resource & Service Administration. Workforce projections. anxiety using a fully automated conversational agent (Woebot): a Rockville, MD: Bureau of Health Workforce, 2024 (https://data. randomized controlled trial. JMIR Ment Health 2017;4:e19. DOI: hrsa.gov/topics/health-workforce/workforce-projections). 10.2196/mental.7785. NEJM AI 13 NEJM AI is produced by NEJM Group, a division of the Massachusetts Medical Society. Downloaded from ai.nejm.org by a a on March 27, 2025. For personal use only. No other uses without permission. Copyright © 2025 Massachusetts Medical Society. All rights reserved. 19. Darcy A, Daniels J, Salinger D, Wicks P, Robinson A. Evidence of 30. McMillan D, Gilbody S, Richards D. Defining successful treat- human-level bonds established with a digital conversational agent: ment outcome in depression using the PHQ-9: a comparison of cross-sectional, retrospective observational study. JMIR Form Res methods. J Affect Disord 2010;127:122-129. DOI: 10.1016/j. 2021;5:e27868. DOI: 10.2196/27868. jad.2010.04.030. 20. Gill SS, Kaur R. ChatGPT: vision and challenges. Internet 31. Munder T, Wilmers F, Leonhart R, Linster HW, Barth J. Working Things Cyber-Phys Syst 2023;3:262-271. DOI: 10.1016/j. alliance inventory-short revised (WAI-SR): psychometric prop- iotcps.2023.05.004. erties in outpatients and inpatients. Clin Psychol Psychother 2010;17:231-239. DOI: 10.1002/cpp.658. 21. De Freitas J, Cohen IG. The health risks of generative AI-based wellness apps. Nat Med 2024;30:1269-1275. DOI: 10.1038/ 32. Sánchez-Meca J, Marín-Martínez F, Chacón-Moscoso S. Effect- s41591-024-02943-6. size indices for dichotomized outcomes in meta-analysis. Psychol Methods 2003;8:448-467. DOI: 10.1037/1082-989X.8.4.448. 22. Pentina I, Hancock T, Xie T. Exploring relationship development with social chatbots: a mixed-method study of replika. Comput 33. Christensen RHB. Cumulative link models for ordinal regression Hum Behav 2023;140:107600. DOI: 10.1016/j.chb.2022.107600. with the R package ordinal. 2018 (https://cran.r-project.org/web/ packages/ordinal/vignettes/clm_article.pdf). 23. Hayes SC, Hofmann SG. “Third-wave” cognitive and behavioral therapies and the emergence of a process-based approach to inter- 34. UCLA: Statistical Consulting Group. Ordinal logistic regression. R vention in psychiatry. World Psychiatry 2021;20:363-375. DOI: data analysis examples. 2024. (https://stats.oarc.ucla.edu/r/dae/ 10.1002/wps.20884. ordinal-logistic-regression/). 35. Covin R, Ouimet AJ, Seeds PM, Dozois DJA. A meta-analysis of 24. Neundorf A, Öztürk A. How to improve representativeness and CBT for pathological worry among clients with GAD. J Anxiety cost-effectiveness in samples recruited through meta: a compar- Disord 2008;22:108-116. DOI: 10.1016/j.janxdis.2007.01.002. ison of advertisement tools. PLoS One 2023;18:e0281243. DOI: 10.1371/journal.pone.0281243. 36. Lepping P, Whittington R, Sambhi RS, et al. Clinical rele- vance of findings in trials of CBT for depression. Eur Psychiatry 25. Harris PA, Taylor R, Minor BL, et al. The REDCap consortium: 2017;45:207-211. DOI: 10.1016/j.eurpsy.2017.07.003. building an international community of software platform partners. J Biomed Inform 2019;95:103208. DOI: 10.1016/j.jbi.2019.103208. 37. Jarry JL, Ip K. The effectiveness of stand-alone cognitive-be- havioural therapy for body image: a meta-analysis. Body Image 26. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. 2005;2:317-331. DOI: 10.1016/j.bodyim.2005.10.001. Research electronic data capture (REDCap)–a metadata-driven methodology and workflow process for providing translational 38. Baier AL, Kline AC, Feeny NC. Therapeutic alliance as a medi- research informatics support. J Biomed Inform 2009;42:377-381. ator of change: a systematic review and evaluation of research. DOI: 10.1016/j.jbi.2008.08.010. Clin Psychol Rev 2020;82:101921. DOI: 10.1016/j.cpr.2020.​ 101921. 27. Kroenke K, Spitzer RL, Williams JBW. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001;16:606- 39. Haque MDR, Rubya S. An overview of chatbot-based mobile men- 613. DOI: 10.1046/j.1525-1497.2001.016009606.x. tal health apps: insights from app description and user reviews. JMIR MHealth UHealth 2023;11:e44838. DOI: 10.2196/44838. 28. Newman MG, Zuellig AR, Kachin KE, et al. Preliminary reli- ability and validity of the generalized anxiety disorder question- 40. Boisvert CM, Faust D. Iatrogenic symptoms in psychother- naire-IV: a revised self-report diagnostic measure of generalized apy. Am J Psychother 2002;56:244-259. DOI: 10.1176/appi. anxiety disorder. Behav Ther 2002;33:215-233. DOI: 10.1016/ psychotherapy.2002.56.2.244. S0005-7894(02)80026-0. 41. Bryan AC, Heinz MV, Salzhauer AJ, Price GD, Tlachac ML, 29. Graham AK, Trockel M, Weisman H, et al. A screening tool for Jacobson NC. Behind the screen: a narrative review on the trans- detecting eating disorder risk and diagnostic symptoms among col- lational capacity of passive sensing for mental health assess- lege-age women. J Am Coll Health J ACH 2019;67:357-366. DOI: ment. Biomed Mater Devices 2024;2:778–810. DOI: 10.1007/ 10.1080/07448481.2018.1483936. s44174-023-00150-4. NEJM AI 14 NEJM AI is produced by NEJM Group, a division of the Massachusetts Medical Society. Downloaded from ai.nejm.org by a a on March 27, 2025. For personal use only. No other uses without permission. Copyright © 2025 Massachusetts Medical Society. All rights reserved.