Generative KI ohne Schutzmaßnahmen beeinträchtigt das Lernen in der Mathematik

RESEARCH ARTICLE ECONOMIC SCIENCES Generative AI without guardrails can harm learning: Evidence from high school mathematics Hamsa Bastania,b,1 ID , Osbert Bastanic,1 , Alp Sungua,1,2 ID , Haosen Geb ID , Özge Kabakcıd , and Rei Marimane Affiliations are included on p. 7. Edited by Emma Brunskill, Stanford University, Stanford, CA; received November 3, 2024; accepted May 5, 2025 by Editorial Board Member Mark Granovetter Generative AI is poised to revolutionize how humans work, and has already demonstrated promise in significantly improving human productivity. A key question Significance is how generative AI affects learning—namely, how humans acquire new skills as they perform tasks. Learning is critical to long-term productivity, especially since generative While generative AI has been AI is fallible and users must check its outputs. We study this question via a field shown to enhance productivity, experiment where we provide nearly a thousand high school math students with access its influence on learning to generative AI tutors. To understand the differential impact of tool design on learning, new skills remains unclear. Our we deploy two generative AI tutors: one that mimics a standard ChatGPT interface research examines the impact of (“GPT Base”) and one with prompts designed to safeguard learning (“GPT Tutor”). generative AI, specifically GPT-4, Consistent with prior work, our results show that having GPT-4 access while solving on student learning in math problems significantly improves performance (48% improvement in grades for GPT education. Through a large-scale Base and 127% for GPT Tutor). However, we additionally find that when access is field experiment in a high school, subsequently taken away, students actually perform worse than those who never had access (17% reduction in grades for GPT Base)—i.e., unfettered access to GPT-4 can our study reveals that although harm educational outcomes. These negative learning effects are largely mitigated by AI-based tutoring improves the safeguards in GPT Tutor. Without guardrails, students attempt to use GPT-4 performance during practice as a “crutch” during practice problem sessions, and subsequently perform worse on sessions, students relying on the their own. Thus, decision-makers must be cautious about design choices underlying technology may underperform generative AI deployments to preserve skill learning and long-term productivity. when access to AI is subsequently removed, indicating reduced generative AI | education | skill acquisition | personalized tutoring skill acquisition. However, we also find that carefully designed Generative AI, such as OpenAI’s ChatGPT, has rapidly emerged as a disruptive safeguards, especially asking technology capable of achieving human-level performance on a broad range of tasks (1–5). the AI tutor to provide teacher- In many applications, they are expected to augment humans to help them perform tasks designed hints instead of giving effectively and efficiently (6). Recent studies have sought to better understand how away answers, can mitigate these humans work in collaboration with these tools (7–9). Broadly speaking, these studies have focused on productivity, finding that workers can perform knowledge-intensive negative effects. Our findings tasks significantly more efficiently when given access to generative AI. highlight the need for thoughtful A key question that remains is how generative AI affects how humans learn novel skills, integration of generative AI both in educational settings and through the course of performing their jobs. This process in educational settings to ensure of skill acquisition is critical for safeguarding long-term productivity (10). However, that human learning is preserved. many generative AI deployments are designed to automate tasks without consideration for impact on learning. When technology automates a task, humans can miss out on valuable experience, inducing a tradeoff where the technology improves performance Author contributions: H.B., O.B., and A.S. designed on average but introduces new failure cases due to reduced human skill. For example, research; H.B., O.B., and A.S. performed research; O.B. overreliance on autopilot led the Federal Aviation Administration to recommend that and H.G. contributed new analytic tools; H.B., O.B., A.S., H.G., and R.M. analyzed data; A.S. and Ö.K. managed field pilots minimize their use of this technology (11); their precautionary guidance ensures activities; and H.B., O.B., and A.S. wrote the paper. that pilots have the necessary skills to maintain safety in situations where autopilot fails to The authors declare no competing interest. function correctly. The potential for generative AI to interfere with learning is especially This article is a PNAS Direct Submission. E.B. is a guest concerning due to the inconsistent reliability of this technology; for instance, while editor invited by the Editorial Board. generative AI has demonstrated tremendous capabilities such as strong performance Copyright © 2025 the Author(s). Published by PNAS. This article is distributed under Creative Commons on medical exams (2) and competitive programming (3), it continues to suffer from Attribution-NonCommercial-NoDerivatives License 4.0 hallucinations where it provides confident but factually incorrect responses (12). As a (CC BY-NC-ND). consequence, users must vigilantly check its outputs and fix any issues present; if they 1 H.B., O.B., and A.S. contributed equally to this work. fail to learn the underlying skills, then they may lack the expertise required to do so. 2 To whom correspondence may be addressed. Email: Simultaneously, there has also been interest in leveraging generative AI to improve alpsungu@wharton.upenn.edu. learning (13, 14), e.g., by incorporating it into existing chatbots for personalized This article contains supporting information online at https://www.pnas.org/lookup/suppl/doi:10.1073/pnas. tutoring (15, 16). Given the broad capabilities of generative AI for natural language 2422633122/-/DCSupplemental. understanding of tasks (1), the hope is that generative AI tutors can automatically Published June 25, 2025. PNAS 2025 Vol. 122 No. 26 e2422633122 https://doi.org/10.1073/pnas.2422633122 1 of 8 identify concepts that students misunderstand based on their learning or subsequent performance as a consequence of copying attempts at solving practice problems, and provide hints to help solutions, suggesting they are not aware of how generative AI can clarify these misunderstandings so they can make progress. As impede their learning. Our results have significant implications a consequence, there is a critical need for field evidence to help for generative AI tools—while such tools have the potential understand whether generative AI can aid or impede learning. to improve human performance, they must be deployed with Based on mixed findings from prior deployments of technologies appropriate guardrails when learning is important. such as laptops (17) and iPads (18) in educational settings, we naturally expect learning outcomes to depend critically on the Experimental Design specific design of the generative AI tool being deployed as well as the deployment context (19). One of the key design decisions is We created a custom math tutoring program based on OpenAI’s the choice of prompt, which is a set of instructions for how the GPT-4 (1); our tutor is designed to help students solve a series tool should respond to user queries. While standard prompts of practice problems provided by the teachers (described below). ask the tool to assist the user without regard for impact on Our tool has two variants. The first variant, “GPT Base,” is learning, they can be augmented with guardrails designed to a simple chat interface similar to ChatGPT, with a prompt facilitate learning. including the current practice problem and indicating that GPT- In this paper, we study how the design of generative AI 4 should serve as a tutor and help the student solve the problem. tools affects learning in an educational setting. In collaboration The second variant, called “GPT Tutor,” uses the same chat with a high school in Turkey, we conducted a large-scale interface, but the prompt additionally implements safeguards to randomized controlled trial (RCT) evaluating the impact of mitigate two key challenges. First, the prompt instructs GPT-4 GPT-4 based tutors on student learning.* Specifically, focusing to provide hints to the student without directly giving them the on mathematics, we study the impact of GPT-4 based tutors on answer, to encourage learning (20). Second, the prompt provides in-class study sessions designed to help students review material a significant amount of problem-specific information provided previously covered in the course. Each study session proceeds in by teachers,§ including one or more (correct) solutions to the two phases. In the first phase, students have the opportunity to practice problem, as well as common student mistakes and how solve a number of practice problems. In this phase, students are to provide feedback. This problem-specific construction is labor- given access to standard resources (their course notes and the intensive, but ensures that GPT-4 does not provide incorrect course textbook), as well as additional generative AI resources feedback to the student. Fig. 1 shows representative prompts for determined based on a randomly assigned arm; the arms are as both variants; additional details on our tool as well as example follows: i) access to a standard chat interface based on GPT-4, interactions are in SI Appendix, Appendix A.1. Note that students designed to mimic the widely used ChatGPT tool (called GPT do not see our system prompts in either variant of our tool. Base), ii) access to a specialized chat interface built on GPT-4 We performed a RCT to evaluate the impact of this tutoring using guardrails designed based on teacher input (called GPT program on student performance. The study took place at a large Tutor),† and iii) no access to generative AI resources. In the high school in Turkey during the Fall semester of the 2023–2024 second phase, students must complete an exam on their own academic year. This study was approved by the University of without access to any resources. Pennsylvania IRB (#853745) and was deemed exempt under 45 Our main results are two-fold. First, students in the GPT CFR 46.104, category 1.¶ We conducted four 90-min sessions Tutor (resp., GPT Base) arm perform 127% (resp., 48%) better for about fifty 9th, 10th, and 11th-grade classes, comprising on the practice problems compared to students in the control nearly 1,000 students. For each grade, our sessions collectively arm. This finding is consistent with prior work on the benefits comprised about 15% of the math curriculum covered during of generative AI on improving human abilities on a variety of the semester. Each session has three contiguous parts: tasks. Second, on the exam, students in the GPT Base arm perform statistically significantly worse than students in the 1. In the first part, teachers review a topic (e.g., combinatorics) control arm by 17%; this negative effect is essentially eradicated previously covered in the course, and solve one or more in the GPT Tutor arm, though we still do not observe a positive examples on the board. This part is identical to a standard effect. These results suggest that while access to generative AI high school lecture. can improve performance, it can substantially inhibit learning 2. The second part is an assisted practice period, where students without appropriate guardrails. Importantly, the detrimental solve exercises designed by teachers to reinforce the concepts impact of GPT Base on learning is of immediate concern since covered. Our randomized intervention (detailed below) only the similar ChatGPT tool is already widely used by students affects this second, self-study part. After students submit their outside of class for help with assignments. Furthermore, an answers, the teacher briefly reviews the correct answers with analysis of student interactions shows that students often use the entire class. GPT Base as a “crutch” by asking for and copying solutions, 3. The third part is an unassisted evaluation, where students but they use GPT Tutor in more substantive ways like asking take a closed-book, closed-laptop exam. Importantly, each for help or independently attempting answers.‡ Finally, we find problem in the exam corresponds to a conceptually very evidence that students do not perceive any reduction in their similar practice problem from the previous part—this design was chosen to help students practice the key concepts needed to perform well on the exam. * We distinguish ChatGPT, the chat interface, from GPT-4, the underlying language model. † In GPT Tutor, GPT-4 is given a prompt including the solution to each problem (to mitigate hallucinations) as well as instructions to avoid giving away the entire solution; furthermore, the prompt includes common student mistakes and corresponding hints to provide if a § We hired two math teachers part-time to provide these inputs; see SI Appendix, student makes one of these mistakes. Fig. 1 shows an example prompt construction, with Appendix A.3. details provided in SI Appendix, Appendix A.1. ¶ The study team provided the school with draft information and consent forms, allowing ‡ Our analysis focuses on short-term learning (estimated via exam performance) rather students to opt out of having their data shared with the research team. The school was than long-term learning—while the two can be significantly different (20), our finding of a responsible for distributing this information and obtaining consent since the study team “crutch” mechanism suggests that both types of learning would be negatively impacted. did not directly interact with students or parents. 2 of 8 https://doi.org/10.1073/pnas.2422633122 pnas.org Fig. 1. Prompts used in GPT Base and GPT Tutor for the first 11th grade practice problem in the first session. The first and third parts are identical across all treatment arms. bias. Graders evaluated the scores based on a teacher-designed Teachers did not interact with students during the second rubric; see details in SI Appendix, Appendix A.5. At the end and third parts, and all students submitted both practice and of each session, we surveyed the students on their experience exam answers on paper to maintain consistency across arms. and preferences. Third, in the GPT Base and GPT Tutor arms, Furthermore, to ensure incentive compatibility, performance on we collected all student messages and corresponding GPT-4 both the second and third parts contributed to students’ final responses from interactions with our tutoring program. grades. Details on the session material and experimental protocol We preregistered|| this RCT with a designated primary analysis is provided in SI Appendix, Appendix A. of comparing students’ unassisted exam outcomes across arms— At this school, students are randomly assigned to classrooms this translates to our main findings that generative AI without (with the exception of honors-designated classrooms, which we guardrails can harm learning. exclude from our main sample). We assigned each classroom to one of three treatment arms—control, GPT Base, and GPT Main Results Tutor.# The control arm is business-as-usual, having students work through the practice problems with access to course books Our primary regression specifications evaluate student perfor- and notes with no devices provided. For classes in the GPT Base mance in the assisted practice problems and unassisted exam, and GPT Tutor arms, we provide a laptop to each student, and respectively: they have the opportunity to use our respective tutoring program. A teacher and a staff member were present in each experimental (j) Outcomeics = 𝛽1 GPT Basec + 𝛽2 GPT Tutorc [1] class session to ensure that students did not use other applications + 𝛽3 Prev GPAi + 𝜃s + 𝛿g + 𝛼y + 𝜆t + 𝜀ics . or websites during the session. Students in the GPT arms were also shown a short instructional video introducing our tool and (j ) illustrating prompts designed to aid learning. Students are free to Here, Outcomeics ∈ [0, 1] is the normalized grade of student i in move between different problems during the session. classroom c and session s ∈ {1, .., 4} for the assisted (j = 0) The study has three avenues of data collection. First, at the start or unassisted (j = 1) portion; GPT Basec and GPT Tutorc of the semester, we sent out a 10-min survey to students, collect- are binary variables indicating treatment assignments for class ing data on their demographics and educational background. c; Prev GPAi controls for student performance, and captures We report balance of these covariates across arms in SI Appendix, student i’s normalized GPA from the previous year;** and 𝜃s , Appendix A.4. Second, we collected performance data from both 𝛿g , 𝛼y , and 𝜆t are session, grader, grade level, and teacher the assisted practice problems and the unassisted exams. We hired fixed effects, respectively. SEs are clustered at the classroom independent graders to evaluate student performance to reduce level (which is the unit of randomization). Our main sample potential teacher bias (e.g., self-fulfilling prophecy), and ensured excludes students in honors-designated classrooms (which are that each grader was assigned a similar number of papers across all not populated randomly, unlike regular classrooms).†† three arms in each grade–session pair to reduce potential grader || Our preregistration is available at https://aspredicted.org/4DL_Q3J; see SI Appendix, Appendix A.6 for a discussion. # Class assignments were made based on an integer program that matched observable ** Normalized GPA has a mean of 0.82 and a SD of 0.11 among students in our main sample. characteristics while satisfying scheduling constraints. Since students were randomly assigned to classrooms within our main sample, the assignment of students to arms †† We performed robustness checks that included these students and found similar is random; see SI Appendix, Appendix A.4 for details. results; see SI Appendix, Appendix B.2. PNAS 2025 Vol. 122 No. 26 e2422633122 https://doi.org/10.1073/pnas.2422633122 3 of 8 Table 1. Regression results on normalized student We preregistered a simplistic variation of our main specifica- performance in the practice (assisted) and exam (unas- tion in Eq. 1 using pairwise t tests; the results are reported in SI sisted) problems across grades and sessions; fixed ef- Appendix, Appendix A.6 and are qualitatively similar. fects are suppressed These results demonstrate an inherent tradeoff in access Dependent variable: to generative AI tools: While these tools can substantially Practice perf Exam perf improve human performance when access is available, they can (1) (2) also degrade human learning (particularly when appropriate safeguards are absent), which may have a long-term impact on GPT base 0.137∗∗ -0.054∗ human performance. (0.031) (0.022) Problem-Level Specification. We also examine an alternative GPT tutor 0.361∗∗ -0.004 (0.032) (0.013) regression specification that examines problem-level outcomes rather than student-level outcomes: Prev GPA 0.802∗∗ 1.334∗∗ (j) (0.076) (0.069) Outcomeicps = 𝛽1 GPT Basec + 𝛽2 GPT Tutorc [2] + 𝛽3 Prev GPAi + 𝜃s + 𝛿g + 𝛼y + 𝜆t + 𝜀ics . Control arm mean 0.284 0.321 Control arm SD 0.287 0.277 (j) Here, Outcomeicps ∈ [0, 1] is the normalized grade of student i in Observations 2,848 2,848 classroom c for problem p and session s ∈ {1, .., 4} for the assisted R2 0.389 0.386 (j = 0) or unassisted (j = 1) portions. SI Appendix, Table B.6 Adjusted R2 0.382 0.379 reports the results, which are very similar to the previous student- Robust SEs are clustered at the classroom level. Note: HC1 robust SEs clustered by class. level specification. The problem-level specification is useful in the ∗ P < 0.05; ∗∗ P < 0.01. next section, where we examine how the error rate in solutions generated by GPT Base in the assisted problems affects student performance and learning in the subsequent unassisted exam. Table 1 reports intention-to-treat estimates from this regres- sion. We find that GPT Base and GPT Tutor substantially Student Perception. Interestingly, students’ own self-reported increased student scores in the GPT-assisted practice sessions by perceptions of the effects of GPT tutors on their exam per- 0.137 and 0.361 (out of 1), respectively, relative to the control formance and learning are overly optimistic. While students in arm that only had access to textbooks (mean performance of the GPT Base arm performed worse on the exam (relative to 0.28). These results imply that GPT Base and GPT Tutor would the control arm), they did not perceive that they performed increase performance on the assisted practice sessions by 48% worse or learned less. Similarly, while students in the GPT and 127%, respectively, on average relative to the control arm. Tutor arm did not perform better on the exam (relative to the These results are consistent with the existing literature on the control arm), they perceived that they performed significantly productivity effects of generative AI (7, 8). Furthermore, the gap better. This mismatch between perceived and actual learning has between GPT Tutor and GPT Base illustrates the added benefits been observed in other settings (20, 21). Additional details are of problem-specific teacher inputs in the prompt. Specifically, in SI Appendix, Appendix B.3. GPT Base often provides incorrect answers due to hallucinations; in contrast, the prompt for GPT Tutor includes the solution Heterogeneity. We look for heterogeneous treatment effects as along with recommended hints, which enables students to both a function of preregistered variables, capturing students’ ability, obtain useful hints and check their answers once they have them. resources, and effort. In general, we find limited to no statistically Our mechanism analysis in the next section provides a more significant support for heterogeneous treatment effects with detailed analysis supporting this hypothesis. either treatment, particularly with respect to unassisted exam per- In stark contrast, in the subsequent unassisted exam, student formance. Additional details are in SI Appendix, Appendix B.4. performance in the GPT Base arm degraded by 0.054 (out of 1) relative to that of the control arm. In other words, GPT Grade Dispersion. We examine a measure of dispersion in Base diminished the average control student’s performance on student performance—the Herfindahl–Hirschman index (HHI). the unassisted exam by 17%. Student performance in the GPT Both GPT Base and GPT Tutor reduced grade dispersion in the Tutor arm was statistically indistinguishable from that of the assisted practice sessions, matching prior findings that generative control arm, and the point estimate was smaller by an order AI assistance reduces the “skill gap” by providing the largest of magnitude (−0.004). The fact that students in the GPT benefits for the weakest students (7–9). However, we find no Tutor arm performed similarly to students in the control arm significant effect on HHI for the unassisted exam—i.e., the on the unassisted exam may be surprising since they performed reduction in the skill gap does not persist when access to AI so much better on the practice problems. This difference may is removed. Additional details are in SI Appendix, Appendix B.5. be partly explained by the fact that students with access to GPT Tutor during the practice session could ask it to check Robustness Checks. Five class sessions did not use the assigned their answers,‡‡ whereas students in the control arm were only treatment due to external circumstances (e.g., laptops did not shown the solutions after they had already submitted their arrive on time). Our primary specifications use an intention-to- answers. treat analysis—i.e., to preserve randomization, we consider all ‡‡ As shown in SI Appendix, Fig. C.4, “Attempted Answers” and “Ask for help” are the students in a treatment arm as treated, regardless of whether they dominant message types in the GPT Tutor arm; see discussion in “Student Engagement” actually received that treatment. In SI Appendix, Appendix B.1, subsection. we provide details on noncompliance and perform a regression 4 of 8 https://doi.org/10.1073/pnas.2422633122 pnas.org Logical Error Arithmetic Error 40 Number of Problems 30 20 10 0 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 Fraction of Incorrect Answers Fig. 2. Fraction of times GPT Base returns an incorrect answer due to a logical (Left) or arithmetic (Right) error for 57 practice problems across grades and sessions. (Note that GPT Tutor does not return an answer by design, and is further given the correct answer in the prompt). omitting noncompliers, finding the same insights. Next, we significant problem-specific heterogeneity in error rates. Details examine several variations to our main specification in Eq. 1 are in SI Appendix, Appendix C.1. to assess the robustness of our results. These results provide We now assess how errors made by GPT Base affect student qualitatively similar insights as our main analysis; see SI Appendix, performance on both the practice problems and the unassisted Appendix B.2. Last, we check whether differential student exam. To this end, we add interaction terms between the error absenteeism may impact our results, and find no differential rate of GPT Base and the treatment arm to the problem- attrition in student attendance across arms or sessions; see SI level regression specification in Eq. 2. When assessing exam Appendix, Appendix B.6. performance, we leverage our paired design of the session material—i.e., for each exam problem, the teachers included a conceptually similar practice problem to help students learn how Potential Mechanism: Asking for Solutions to solve that exam problem. Thus, for a given exam problem, we use GPT Base’s error rate on the corresponding practice In general, students may be adversely affected by GPT Base’s problem. The regression specification is given in SI Appendix, assistance in two ways: 1) errors made by GPT Base mislead Eq. S5, and the results are shown in Table 2. Both types of GPT students in the subsequent unassisted problems, or 2) using GPT Base errors negatively impact practice problem performance for Base as a “crutch” prevents them from fully engaging with or students in the GPT Base arm (i.e., the coefficients of “GPT Base understanding the material prior to attempting the unassisted × Logical Error Rate” and “GPT Base × Arithmetic Error Rate” problems. Recall that the design of GPT Tutor avoids both of are statistically significantly negative in the practice performance these issues: 1) it rarely makes mistakes since its prompt includes regression).§§ the solution, and 2) it is hard for students to use it as a crutch since Two key observations support our hypothesis that students its prompt asks it to avoid giving them the answer and instead are using GPT Base as a crutch. First, if students are being guide them in a step-by-step fashion (SI Appendix, Fig. A.3). We misled by logical errors made by GPT Base, we would expect perform two analyses that help determine which explanation is these errors to affect performance on the corresponding exam more likely. First, we analyze how the error rate of GPT Base on problems in the unassisted exam. However, while GPT Base’s a practice problem affects students’ subsequent performance on logical errors affect performance on the practice problems, we a highly similar exam problem, and second, we analyze student find no evidence that this effect spills over to the corresponding engagement with the tool. Our findings suggest that the second exam problems (i.e., “GPT Base × Logical Error Rate” does explanation (i.e., students using GPT Base as a crutch) is the not have a statistically significant effect on exam performance). main mechanism by which GPT Base impedes student learning. An analogous regression on the total error rate (combining both logical and arithmetic errors), yields similar insights (SI Appendix, GPT Errors vs. Student Performance. GPT Base often makes Table C.14). Second, if students were reading and understanding mistakes on math problems (22). We first quantify error rates the solutions provided by GPT Base in the practice session, by repeatedly querying GPT Base using the most common we might expect arithmetic errors to have a smaller impact message used by students in the GPT Base arm—i.e., “What on practice problem performance than logical errors. This is is the answer?” For each of the 57 total practice problems, we because students know arithmetic relatively well, and should be ask GPT Base for the answer ten times (resetting the system better able to catch these errors. However, arithmetic and logical between queries), and then manually categorize any errors in errors appear to have similar effects on practice performance (i.e., the response as arithmetic (steps followed were correct, but the “GPT Base × Logical Error Rate” and “GPT Base × Arithmetic resulting computation was incorrect) or logical (steps followed Error Rate” have similar coefficients in the practice performance were partially or fully incorrect). We find that GPT Base gives a correct answer only 51% of the time on average; it makes logical errors 42% of the time and arithmetic errors 8% of §§ Note that we separately control for logical and arithmetic error rates. As expected, the corresponding coefficients in the practice performance regression are both statistically the time. Fig. 2 shows the histogram of how often GPT Base significantly negative, since higher GPT Base error rates are correlated with higher problem returns an incorrect answer for different problems, demonstrating difficulty. PNAS 2025 Vol. 122 No. 26 e2422633122 https://doi.org/10.1073/pnas.2422633122 5 of 8 Table 2. Regression results on student performance in in SI Appendix, Fig. C.4, students in GPT Base most often simply the practice and corresponding exam problems across ask for the answer; in contrast, students in GPT Tutor learn grades and sessions; this regression is at the problem to interact more substantively with the tutoring tool over time level, and includes interaction terms for the logical and by asking for help and independently attempting to solve the arithmetic error rates of GPT Base on practice problems problem. We observe this learning effect even within the first (SI Appendix, Eq. S5) session—if we restrict to the very first interaction students have Dependent variable: with our tool in the first session, 56% in the GPT Base arm Practice perf Exam perf and 42% in the GPT Tutor arm## either repeat the question text or ask for the answer; in contrast, when considering the first GPT base 0.362∗∗ -0.035 interaction across all problems in the first session, this number (0.032) (0.027) increases to 67% for GPT Base but decreases to 37% for GPT Tutor. GPT tutor 0.337∗∗ 0.029 We consider clusters where the student simply asks for (0.037) (0.023) the answer (specifically, “Repeat Question Text” and “Ask for Answers”) to be superficial, and the remaining clusters Logical error rate -0.075∗ 0.178∗∗ (specifically, “Attempted Answers” and “Ask for Help”) to be (0.030) (0.028) nonsuperficial. For a given student in a given session, we consider Arithmetic error rate -0.172∗ -0.063 the corresponding conversation superficial if the student asked (0.082) (0.032) any superficial messages, and nonsuperficial otherwise. Intu- itively, nonsuperficial conversations are ones where the student Prev GPA 0.789∗∗ 1.330∗∗ constructively interacts with the tutoring tool and never asks (0.074) (0.068) for the answer. Fig. 3B shows the aggregate rate of nonsuperficial conversations for both GPT Base and GPT Tutor. As can be seen, GPT base × logical error rate -0.448∗∗ -0.029 across all sessions, in the GPT Base arm, only a small fraction (0.036) (0.040) of conversations are nonsuperficial; in contrast, a substantially larger fraction of conversations in the GPT Tutor arm were GPT tutor × logical error rate 0.022 -0.086 nonsuperficial. These results suggest that the vast majority of (0.038) (0.044) students are using GPT Base to obtain solutions, whereas a significant fraction of students are using GPT Tutor in a purely GPT base × arithmetic error rate -0.492∗∗ -0.099 substantive way. (0.117) (0.056) GPT tutor × arithmetic error rate 0.329∗∗ 0.095∗ Discussion (0.107) (0.044) Our results provide a cautionary tale regarding both the existing use of the freely available ChatGPT tool by students as well as Observations 13,484 11,392 the potential deployment of GPT-based tutors in educational R2 0.214 0.212 settings. While generative AI tools such as ChatGPT can make Adjusted R2 0.212 0.209 tasks significantly easier for humans, they come with the risk of We use a correspondence between the exam and practice problems to estimate how deteriorating our ability to effectively learn some of the skills errors on practice problems affect performance on exam problems. Fixed effects are required to solve these tasks. These shortcomings have been suppressed. Note: HC1 robust SEs clustered by class. ∗ P < 0.05; ∗∗ P < 0.01. anecdotally reported for tools such as Khanmigo (23), a GPT-4 based tutoring application. Our findings support both the need for educators to find ways to safeguard student learning in the regression). Both these results suggest that students are simply face of freely available tools such as ChatGPT as well as the need copying answers from GPT Base. to design more effective guardrails for generative AI tutors. In some ways, ChatGPT is not the first technology to exhibit Student Engagement. Next, we analyze the messages that stu- this tradeoff—for instance, typing diminishes the need for hand- dents sent to GPT Base or GPT Tutor to better understand how writing, and calculators diminish the need for arithmetic, etc. they are interacting with these tools. Fig. 3A shows the average However, we believe ChatGPT differs from prior technologies number of messages each student had with their respective GPT in two significant ways. First, the capabilities of ChatGPT are tool (Base or Tutor) in a given session. As can be seen, the substantially broader and more intellectual compared to prior number of messages in GPT Tutor is significantly higher, and examples; for instance, our experiments focus on a broad variety further increases with experience using the tool.¶¶ The fact that of mathematical topics, which encompass fundamental skills students interact substantially less with GPT Base is consistent required by a wide range of knowledge-intensive professions. with our hypothesis that GPT Base simply provides students with Second, unlike many prior technologies, ChatGPT is highly solutions. unreliable and often provides incorrect responses. Our results For a more fine-grained understanding of the content of the suggest that students are either unable to detect these failures or student messages, we use natural language processing and cluster- unwilling to spend the effort needed to check correctness. ing to group student messages (see SI Appendix, Appendix C.2 for While the guardrails implemented in GPT Tutor appear details). We manually associate each cluster with a text description to largely mitigate these negative effects, substantial work is summarizing the content of the messages in that cluster. As shown required to enable generative AI to positively enhance rather than ¶¶ We also find that students spend more time with GPT Tutor than with GPT Base; see ## The initial discrepancy may be due to the fact that GPT Tutor, unlike GPT Base, offers SI Appendix, Appendix C.3. suggested prompts; see SI Appendix, Fig. A.2B for details. 6 of 8 https://doi.org/10.1073/pnas.2422633122 pnas.org Fraction of Conversations that are Non−Superficial 5 0.5 4 0.4 Messages per Problem 3 Treatment 0.3 Treatment GPT Base GPT Base GPT Tutor GPT Tutor 2 0.2 1 0.1 0 0.0 1 2 3 4 1 2 3 4 Session Session A B Fig. 3. Student engagement—given by (A) average number of student messages per problem, and (B) average fraction of student session conversations that have no superficial messages (simply restating the question or asking for the answer) per session—by treatment (GPT Base and GPT Tutor) over time. diminish education. GPT Tutor remains passive, responding then. Additional studies are required to assess generalizability to students when they ask questions but failing to proactively to other tutor designs and deployment contexts. Moreover, we engage students with the material. Effective human tutors ask focus on short-term outcomes due to limitations imposed by our probing questions to uncover student misconceptions, and then partner school; studying long-term outcomes is a key direction for clarify these misconceptions by providing tailored explanations. future research. Last, while we have performed several analyses to Combining existing software tutors (15, 24) with generative uncover mechanisms underlying our findings, our approach can AI may be a promising path to achieving this goal, since it be complemented with controlled experiments in settings where balances the pedagogical principles baked into existing algorithms more fine-grained interventions and monitoring are possible to with the capability of generative AI to understand and respond better understand these mechanisms. to complex student queries. One way to do so is to leverage agent approaches (25, 26), which compose models with different Data, Materials, and Software Availability. To support further research, prompts to achieve complex goals; for instance, we might use we have made anonymized data available at https://github.com/obastani/ one prompt to ask the model to identify student misconceptions, GenAICanHarmLearning (28). Data include student performance on practice and then use another prompt to ask it to generate a useful and exam questions for all sessions, as well as student-, class-, and grader- hint. Beyond designing better tutors, promising avenues include level covariates; they also include anonymized and time-stamped student educating students and teachers about how to more effectively conversations with our GPT tutors. All code used in this paper was written in a use generative AI, and leveraging generative AI to help teachers combination of R, Python, and Stata and can be found in the same repository. instead of focusing on students (13). For instance, recent evidence suggests that “co-pilots” that work with a human tutor instead of ACKNOWLEDGMENTS. We acknowledge invaluable research partnership with replacing them might be a more promising path toward effective Tamer Atacan and TED Ankara Koleji, as well as research assistance from Tuba use of generative AI in education (27). Tas and Dogus Berk Kocak. We are grateful for funding from the Wharton Finally, while our study takes a step toward understanding AI & Analytics Initiative, the Fishman-Davidson Center, and Wharton Global the potential harms of generative AI on learning, it focuses on Initiatives. We are also grateful for helpful feedback from Eric Bradlow, Angela a specific deployment context, and substantial work is needed Duckworth, Benjamin Luttges, Lilach Mollick, Ananya Sen, Christian Terwiesch, to better understand the nature and scope of these effects. First, Lyle Ungar, as well as seminar participants at the London School of Economics, our study focuses on two generative AI tutors for a single topic Indian School of Business, Google, Microsoft, Workshop on Unstructured Data (mathematics) deployed in a single high school in Turkey. We and Language Models, Conference on Digital Experimentation, and Workshop have objective evaluation criteria for math problems, which is on AI & Analytics for Social Good. unavailable in other important subjects such as writing. Our deployment was also carried out in Fall 2023, when generative Author affiliations: a Department of Operations, Information, and Decisions, Wharton AI was still very new; modes of interaction may have changed School, University of Pennsylvania, Philadelphia, PA 19104; b Wharton AI & Analytics, now that users are more familiar with effective uses of these tools Philadelphia, PA 19104; c Department of Operations, Information, and Decisions, School of Engineering and Applied Science, University of Pennsylvania, Philadelphia, PA 19104; and their shortcomings. Furthermore, both closed and open- d Department of Mathematics, Budapest British International School, Budapest 1125, weight models have significantly improved in performance since Hungary; and e Independent, Philadelphia, PA 19104 1. J. Achiam et al., Gpt-4 technical report. arXiv [Preprint] (2023). https://arxiv.org/abs/2303.08774 7. S. Noy, W. Zhang, Experimental evidence on the productivity effects of generative artificial (Accessed 5 March 2025). intelligence. Science 381, 187–192 (2023). 2. K. Singhal et al., Large language models encode clinical knowledge. Nature 620, 172–180 8. E. Brynjolfsson, D. Li, L. R. Raymond, “Generative AI at work” in National Bureau of Economic (2023). Research Working Paper (2023). 3. Y. Li et al., Competition-level code generation with AlphaCode. Science 378, 1092–1097 (2022). 9. F. Dell’Acqua et al., “Navigating the jagged technological frontier: Field experimental evidence 4. W. Geerling, G. D. Mateer, J. Wooten, N. Damodaran, ChatGPT has aced the test of understanding of the effects of AI on knowledge worker productivity and quality” in Harvard Business School in college economics: Now what? Am. Econ. 68, 233–245 (2023). Technology & Operations Management Unit Working Paper (2023). 5. C. Terwiesch, “Would chat GPT get a Wharton MBA” in Mack Institute White Paper (Wharton School, 10. G. S. Becker, Human Capital: A Theoretical and Empirical Analysis, with Special Reference to University of Pennsylvania, 2023). Education (University of Chicago press, 2009). 6. T. Eloundou, S. Manning, P. Mishkin, D. Rock, GPTs are GPTs: Labor market impact potential of 11. Safety alert for operators: Manual flight operations (Tech. Rep. SAFO 13002, Federal Aviation LLMs. Science 384, 1306–1308 (2024). Administration, 2013). PNAS 2025 Vol. 122 No. 26 e2422633122 https://doi.org/10.1073/pnas.2422633122 7 of 8 12. J. Maynez, S. Narayan, B. Bohnet, R. McDonald, “On faithfulness and factuality in abstractive 20. E. L. Bjork, R. A. Bjork, “Making things hard on yourself, but in a good way: Creating desirable summarization” in Proceedings of the 58th Annual Meeting of the Association for Computational difficulties to enhance learning” in Psychology and the Real World: Essays Illustrating Fundamental Linguistics, D. Jurafsky, J. Chai, N. Schluter, J. Tetreault, Eds. (Association for Computational Contributions to Society, M. A. Gernsbacher, R. W. Pew, L. M. Hough, J. R. Pomerantz, Eds. (Worth Linguistics, 2020), pp. 1906–1919. Publishers, 2011), pp. 56–64. 13. E. R. Mollick, L. Mollick, New modes of learning enabled by AI chatbots: Three methods and 21. L. Deslauriers, L. S. McCarty, K. Miller, K. Callaghan, G. Kestin, Measuring actual learning versus assignments. http://dx.doi.org/10.2139/ssrn.4300783. Accessed 5 March 2025. feeling of learning in response to being actively engaged in the classroom. Proc. Natl. Acad. Sci. 14. A. Extance, ChatGPT has entered the classroom: How LLMs could transform education. Nature 623, U.S.A. 116, 19251–19257 (2019). 474–477 (2023). 22. S. Frieder et al., Mathematical capabilities of chatGPT. Adv. Neural Inf. Process. Syst. 36, 13867 15. S. Ruan et al., “Supporting children’s math learning with feedback-augmented narrative tech- (2024). nology” in Proceedings of the Interaction Design and Children Conference (Association for 23. N. Singer, In classrooms, teachers put A.I. tutoring bots to the test. N. Y. Times (2023). Computing Machinery, New York, NY, 2020), pp. 567–580. 24. S. Doroudi, V. Aleven, E. Brunskill, Where’s the reward? A review of reinforcement learning for 16. S. Ruan et al., Reinforcement learning tutor better supported lower performers in a math task. instructional sequencing Int. J. Artif. Intell. Educ. 29, 568–620 (2019). Mach. Learn. 113, 3023–3048 (2024). 25. S. Yao et al., “React: Synergizing reasoning and acting in language models” in The Eleventh 17. J. Cristia, P. Ibarrarán, S. Cueto, A. Santiago, E. Severín, Technology and child development: International Conference on Learning Representations (ICLR) (2023). Evidence from the one laptop per child program. Am. Econ. J. Appl. Econ. 9, 295–320 26. X. Wang et al., “Executable code actions elicit better LLM agents” in Forty-First International (2017). Conference on Machine Learning (JMLR.org, Vienna, Austria, 2024). 18. A. J. Lamb, J. M. Weiner, Institutional factors in iPad rollout, adoption, and implementation: 27. R. E. Wang, A. T. Ribeiro, C. D. Robinson, S. Loeb, D. Demszky, Tutor copilot: A human-AI approach Isomorphism and the case of the Los Angeles unified school district’s iPad initiative. Int. J. Educ. for scaling real-time expertise. arXiv [Preprint] (2024). https://arxiv.org/abs/2410.03017 (Accessed Math. Sci. Technol. 6, 136–154 (2018). 5 March 2025). 19. M. Escueta, A. J. Nickow, P. Oreopoulos, V. Quan, Upgrading education with 28. H. Bastani et al., Data and Code for “Generative AI without guardrails can harm learning: Evidence technology: Insights from experimental research. J. Econ. Lit. 58, 897–996 from high school mathematics.” GitHub. https://github.com/obastani/GenAICanHarmLearning. (2020). Deposited 11 June 2025. 8 of 8 https://doi.org/10.1073/pnas.2422633122 pnas.org