AI, problem solving, and the future of STEM education

About the author: José Luis Granados Mateo is a former secondary-school and sixth-form STEM teacher, having taught physics, chemistry and mathematics. He is currently a postdoctoral researcher in the epistemology of science at the University of the Basque Country (UPV/EHU).

When I taught physics, chemistry and mathematics to secondary-school and sixth-form students, a correct answer was sometimes where the interesting part began. I would ask why they had used a particular equation. A solution that had looked perfectly sound could become much less convincing; occasionally the reverse happened, and a student whose algebra had gone wrong could explain the physics rather well. Nothing on the page had changed. My estimate of what the student understood had.

I have been thinking about those exchanges while reading the research on generative AI in education. Students can now submit a convincing explanation, solve a numerical problem or produce working code with considerable help from a language model. It is easy to see the achievement and much harder to work out what the student has learnt along the way.

That difficulty cannot be settled by appealing to the virtues of an unaided mind. Scientific work depends on instruments, notation, diagrams, software and other people. Delegating part of a calculation may be entirely sensible. Yet teaching has a complication that is easy to overlook from the perspective of an expert: the operation being delegated may be one of the things through which a beginner is meant to acquire expertise.

An experiment by Hamsa Bastani and colleagues brings this problem into unusually sharp focus.1 Nearly a thousand pupils at a secondary school in Turkey took part in a study comparing two GPT-4 systems with a control condition without AI. GPT Base offered assistance resembling a general-purpose chatbot. GPT Tutor had additional material from teachers, including solutions and common mistakes, and instructions to guide pupils with hints rather than give away the answer.

During practice, both systems helped. The estimated scores were 48% higher than the control mean with GPT Base and 127% higher with GPT Tutor. These are relative differences, not percentage-point gains. On the tests that followed, however, the pupils had to work without assistance. The GPT Base group then scored an estimated 17% below the control mean. GPT Tutor largely mitigated that penalty, but its users showed no statistically significant advantage over the control group. The tests took place within the study sessions, so they tell us about immediate independent performance rather than retention months later.

Had the researchers stopped at the practice exercises, they would have had an encouraging result for both systems. They did not stop there, and the educational interpretation changed. A pupil working with AI had become more successful at the task. Whether that success belonged to the pupil in any lasting sense was another matter.

Psychologists have long distinguished performance during acquisition from learning: a relatively durable change in knowledge or skill. Soderstrom and Bjork review the evidence for that distinction, including cases in which fluent practice leads to disappointing retention and others in which slower, more error-prone practice produces better results later.2 Watching someone learn, we inevitably see performance. To find out what remains, we need to look again after a delay, withdraw some of the support or ask for an application in a different context. Even then, what we have is evidence to interpret, not a direct view of understanding.

In STEM, much of the important work happens before the calculation. A mechanics problem requires a choice of system, a judgement about the relevant interactions and some representation of the situation. Chemistry involves moving between observable phenomena, molecular accounts and symbols on a page. A program that runs is the endpoint of decisions about what to separate, what to abstract and how to test it. Nersessian’s work on model-based reasoning gives representations a central place in scientific thought: constructing and revising a model can be part of arriving at an explanation, rather than merely a way of presenting one.3

A language model can help at almost any of these points. Perhaps the student has made the difficult modelling decisions and wants an algebraic check. Perhaps the model has chosen the principle, drawn up the representation, done the algebra and supplied an explanation before the student has formed a view of the problem. A questionnaire asking whether both students “used AI” would record the same answer.

There is an obvious temptation here to defend struggle. I am wary of it. Spending half an hour looking for a formula can be frustrating without being especially instructive, and a teacher who withholds every useful clue has not necessarily protected the student’s learning.

Worked examples offer a familiar counterexample. A novice studying a solution can attend to the structure of a procedure without having to search blindly for it. Barbieri and colleagues’ meta-analysis, covering 43 articles, 55 studies and 181 effect sizes, found a medium benefit for mathematics performance (g = 0.48).4 Removing some work can help a learner do other work more productively. The judgement lies in deciding what to remove.

Nor will the right decision remain right indefinitely. In a 2025 meta-analysis of the expertise reversal effect, Tetzlaff and colleagues brought together 60 experiments involving 5,924 participants and 176 effect sizes. Learners with little prior knowledge benefited from greater instructional assistance (d = 0.505); for those with high prior knowledge, lower-assistance conditions were preferable (d = −0.428).5 These are heterogeneous results across educational contexts, not estimates specifically for generative AI. Still, they help explain a familiar experience: an explanation that once made a procedure possible can later get in the way of carrying it out.

This is part of what Koedinger and colleagues call the assistance dilemma: when should instruction provide information, and when should it require the learner to generate it?6 With a conversational model, the practical difficulty is that help need never run out. Another prompt can produce another hint, or an answer, or a complete solution to a problem the student has scarcely begun to consider.

Pablo Rodríguez Sánchez raised a related question in Mapping Ignorance, in Why teach calculus in the age of AI. Software can compute many standard derivatives and integrals almost immediately, and AI interfaces make such tools easier to consult. That does change the case for teaching manual procedures. It does not dispose of it. Working through integration by parts can make its relation to the product rule visible; substitution can bring the chain rule into focus from the other direction. Some of the value of a procedure lies in the relationships a student encounters while carrying it out. Requiring the procedure without regard to those relationships would miss the point just as thoroughly as abandoning it because a machine is faster.

We already move parts of our thinking into the world. We write a number down so that we do not have to remember it, or set a reminder for something we expect to forget. Risko and Gilbert describe such actions as cognitive offloading: changing the processing demands of a task through physical action and external resources.7 Its consequences are worth studying, but the fact that someone has offloaded a task is not in itself evidence of cognitive decline.

For history and philosophy of science (HPS), this should be familiar ground. Giere and Moffatt describe scientific cognition as distributed across people and artefacts.8 On this account, instruments, representations and social arrangements can be parts of the system through which scientific knowledge is produced. A physicist who uses a computer to integrate a differential equation has not thereby revealed a failure to understand physics.

She does, though, need some grounds for trusting the result. She must know what calculation is appropriate, which assumptions it involves and when the output ought to make her suspicious. A beginner may still be acquiring those resources. If every mechanics problem arrives already represented and ready for calculation, there may be very little occasion to learn how to represent one.

“Which formula do I use?” was a question I heard often. Usually, in the cases I remember, the difficulty was not simply a forgotten equation. The student had not yet made sense of the situation in a way that gave any equation a purpose. Supplying the formula could rescue the exercise while leaving that difficulty untouched. A question about the forces, or about what was being treated as the system, could sometimes help much more.

The records from Bastani’s experiment are suggestive in this respect. GPT Base users more often requested answers, and the authors interpreted their interaction and error analyses as consistent with copying solutions. GPT Tutor elicited more attempted answers and requests for help. The surveys, administered at the end of the sessions, revealed a further complication. GPT Base users did not perceive the disadvantage visible in their unassisted scores; GPT Tutor users rated their performance more favourably even though no significant exam advantage had been detected.1 Students’ impressions were not a dependable guide to what the assistance had done for them.

But the evidence does not all point towards disappointment. In an introductory physics course at Harvard, Kestin and colleagues compared a purpose-built AI tutor with an established active-learning class.9 Their randomised crossover study covered two lessons, with students experiencing both conditions. The tutor incorporated scaffolding, targeted feedback and self-pacing. Median post-test scores were 4.5 in the AI condition and 3.5 in the classroom condition. Measured against the pooled pre-test median, the gain in the AI condition was more than twice as large. Seventy per cent of students spent less than the 60 minutes allocated to classroom learning on the AI activity; the median was 49 minutes. They also reported greater engagement and motivation.

That is an interesting result on its own terms. It concerns particular content, a carefully prepared tutor and tests taken after the lessons. The intervention also combined several differences, including the setting, pace and prerecorded introductions, so the study cannot tell us how much of the benefit came from the language model alone. Nor does it establish long-term retention. Within those limits, it is good reason to take the design of AI-supported instruction seriously. Efficiency and learning need not be enemies.

Programming provides a less reassuring case. Bassner and colleagues reported a three-arm randomised trial at the Technical University of Munich, with 275 introductory programming students in the reported sample.10 During a 90-minute exercise, students used either the scaffolded tutor Iris, unrestricted ChatGPT or conventional web resources without AI. Both AI groups achieved higher scores on automated tests of their code and reported less frustration. Neither showed a statistically significant advantage in knowledge gains or code comprehension. Only Iris showed an increase in intrinsic motivation. There was clearer evidence of better task performance than of better understanding, although the absence of a significant learning difference does not establish that the groups learnt exactly the same amount.

Three studies cannot carry a general theory of AI in education. Their students, tasks and measurements are too different, and their time horizons too short. What they do make difficult to ignore is the need to decide whose competence an assessment is measuring. If a tool remains available throughout, successful performance may tell us a great deal about the person–tool combination and rather less about a change in the person.

The larger reviews do not make this much easier. Boolzen and colleagues’ 2026 review identified 85 eligible controlled studies of cognitive outcomes in STEM; 49 studies, yielding 59 effect sizes, entered the meta-analysis.11 A conventional random-effects model gave a positive mean effect, alongside extraordinary heterogeneity (I² = 96.32%). Its prediction interval ran from g = −1.52 to g = 3.20. For comparable settings, the model therefore allows for effects extending from substantial harm to substantial benefit. It would be a mistake to read the positive average as a reliable forecast for a new classroom intervention.

The authors also found funnel-plot asymmetry, consistent with possible publication bias. Using a robust Bayesian adjustment, they obtained an overall estimate close to zero (μ = 0.076), with substantial uncertainty. The heterogeneity remained. Asymmetry need not be caused by publication bias, and the adjusted estimate depends on modelling assumptions; it is not a final answer about whether AI helps students learn.

Their moderator analysis is more helpful for thinking about instruction. They coded the learning outcome and the change in students’ activity relative to the control condition. The patterns favoured augmentation over substitution, but these were comparisons across studies, not randomised tests of those design choices. Much variation remained unexplained, and no combination of the examined moderators reliably supplied a sufficient condition for a large effect.11 The distinction is promising as a research question. We do not yet have a rule for applying it to every lesson.

Wu and colleagues’ 2026 meta-analysis sounds more optimistic: 35 studies, 4,193 participants and a positive mean effect of g = 0.670, with a 95% confidence interval of 0.495–0.844.12 The two reviews are not estimating quite the same thing. Wu’s synthesis ranges across disciplines and includes motivation, engagement and self-efficacy alongside cognitive outcomes. It also reports very high heterogeneity (I² = 91.444%). Its tests did not detect publication bias, which is different from establishing its absence. Most included studies concerned higher education.

I would hesitate to choose between those reviews on the strength of their averages. An answer-giving chatbot, a tutor that asks for explanations and a debugging assistant may use closely related models while organising the student’s activity very differently. Asking for “the effect of ChatGPT on learning” begins to sound like asking for the effect of books. We would immediately want to know which books, who was reading them and what they were being asked to do.

Meanwhile, students are getting on with using the tools. PISA 2025 cannot resolve the causal questions raised by the experiments, but it gives some sense of their scale. On average across OECD countries, 46% of 15-year-olds reported using AI chatbots at least weekly to help them learn. In Spain, the figure was 54%. The Spanish percentages reported for preliminary research, summarising assigned texts and drafting written work were 42%, 40% and 33%, respectively. Only 8% reported never or almost never using AI for any of the schoolwork purposes examined.13,14

The achievement patterns deserve attention, though they are easily overread. After adjustment for students’ socioeconomic profile, the OECD-average results generally favoured non-users over those reporting AI use for specific tasks such as drafting or summarising. Weekly use for the broader purpose of helping with learning was associated with roughly similar science performance to non-use. For some specific tasks, moderate users did better than very frequent users.13

A pupil who is struggling may seek more help. Schools may encourage AI use in response to difficulties that already exist. And frequency tells us very little about the encounter itself: requesting an answer, interrogating a hint and challenging an explanation can all count as using a chatbot. Those possibilities are not findings that explain away PISA’s associations; they are reasons why the associations cannot establish what AI caused.

PISA also examines opportunities to evaluate AI-generated information at school. Some descriptive patterns look favourable, but the overall positive association with that activity is not observed, on average, after adjustment for students’ and schools’ socioeconomic profiles.13 The report gives us no simple causal claim about the benefits of teaching students to check AI output.

There is nevertheless an educational question here that deserves more than a slogan. Mapping Ignorance has discussed it in Critical thinking for the unstoppable use of artificial intelligence. In STEM, asking a student to check an answer presupposes that something is available with which to check it. Does the student know what order of magnitude to expect? Which assumption matters? What evidence would bear on the explanation? A fluent paragraph can be physically impossible, but fluency alone will not tell a beginner where it goes wrong.

It is also worth keeping the chronology straight. Generative AI cannot be the sole explanation for declines in educational achievement that began before ChatGPT was publicly available. What PISA establishes more usefully here is that the practices under discussion are already part of ordinary schoolwork. Teachers have to make decisions while the evidence is still developing.

Some of those decisions will be quite small. A worked solution may be exactly what a pupil needs on Monday. A few weeks later, providing the whole solution may remove an opportunity to reconstruct the method. AI could make it easier to vary the support: ask for a representation before proposing an equation, invite a prediction before showing a simulation, or offer a flawed argument for the student to diagnose. These are plausible uses, not capabilities we should assume a system can deliver reliably simply because it holds a convincing conversation.

Assessment poses a related problem. Students will need to work competently with tools, as scientists and engineers do, and that competence deserves to be assessed. But if we also want to know whether they can perform a particular operation themselves, there must be an occasion when the tool is not doing it for them. We have to say what belongs in the learner’s own repertoire, and then find an assessment that gives us reasonable evidence of it.

For someone working in History and Philosophy of Science, the interest extends beyond the ownership of a correct answer. Scientific competence involves judgement about models, evidence, instruments and the reach of a procedure. A student may know a formula and still not know when it applies. An experienced scientist may quite reasonably delegate its numerical solution while retaining responsibility for the assumptions and interpretation. Learning to make those distinctions is part of becoming competent.

That brings me back to the classroom question. When I asked why a student had used an equation, I wanted access to the reasoning behind the calculation. With generative AI, the visible work may belong to a temporary partnership, and locating that reasoning becomes harder. The partnership itself need not be objectionable. What matters educationally is what repeated participation in it does to the student.

Over time, can the learner represent a problem more confidently, choose a method, recognise an implausible result or decide when help is needed? If so, the tool may be contributing to the acquisition of expertise. If those capacities scarcely change while the submitted answers become more polished, the same outward success has a different meaning.

I do not think the most useful question is whether students should use AI in STEM. It is what they should still be learning to do, how the assistance bears on that learning, and when it ought to recede. A correct answer was never a guarantee of understanding. We now have to be more deliberate about finding out what lies behind it.

References

1. Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö. & Mariman, R. (2025). ‘Generative AI without guardrails can harm learning: Evidence from high school mathematics’. Proceedings of the National Academy of Sciences, 122(26), e2422633122. 10.1073/pnas.2422633122
2. Soderstrom, N. C. & Bjork, R. A. (2015). ‘Learning versus performance: An integrative review’. Perspectives on Psychological Science, 10(2), 176–199. 10.1177/1745691615569000
3. Nersessian, N. J. (2002). ‘The cognitive basis of model-based reasoning in science’. In P. Carruthers, S. Stich & M. Siegal (eds), The Cognitive Basis of Science, pp. 133–153. Cambridge University Press. 10.1017/CBO9780511613517.008
4. Barbieri, C. A., Miller-Cotto, D., Clerjuste, S. N. & Chawla, K. (2023). ‘A meta-analysis of the worked examples effect on mathematics performance’. Educational Psychology Review, 35, Article 11. 10.1007/s10648-023-09745-1
5. Tetzlaff, L., Simonsmeier, B., Peters, T. & Brod, G. (2025). ‘A cornerstone of adaptivity – A meta-analysis of the expertise reversal effect’. Learning and Instruction, 98, 102142. 10.1016/j.learninstruc.2025.102142
6. Koedinger, K. R., Pavlik, P. I. Jr., McLaren, B. M. & Aleven, V. (2008). ‘Is it better to give than to receive? The assistance dilemma as a fundamental unsolved problem in the cognitive science of learning and instruction’. In B. C. Love, K. McRae & V. M. Sloutsky (eds), Proceedings of the 30th Annual Conference of the Cognitive Science Society, pp. 2155–2160. Full text
7. Risko, E. F. & Gilbert, S. J. (2016). ‘Cognitive offloading’. Trends in Cognitive Sciences, 20(9), 676–688. 10.1016/j.tics.2016.07.002
8. Giere, R. N. & Moffatt, B. (2003). ‘Distributed cognition: Where the cognitive and the social merge’. Social Studies of Science, 33(2), 301–310. 10.1177/03063127030332017
9. Kestin, G., Miller, K., Klales, A., Milbourne, T. & Ponti, G. (2025). ‘AI tutoring outperforms in-class active learning: An RCT introducing a novel research-based design in an authentic educational setting’. Scientific Reports, 15, 17458. 10.1038/s41598-025-97652-6
10. Bassner, P., Lenk-Ostendorf, B., Beinstingel, R., Wasner, T. & Krusche, S. (2026). ‘Less stress, better scores, same learning: The dissociation of performance and learning in AI-supported programming education’. Computers and Education: Artificial Intelligence, 10, 100537. 10.1016/j.caeai.2025.100537
11. Boolzen, C., Kuhn, J., Flegr, S., Rott, E.-M., Stausberg, N. & Küchemann, S. (2026). ‘Evidence of impact and interpretational limits of generative AI in STEM education: a systematic review and meta-analysis on cognitive learning outcomes’. Artificial Intelligence Review. Advance online publication. 10.1007/s10462-026-11665-9
12. Wu, X., Zhu, P., Zhang, J., Yin, M. & Wang, Y. (2026). ‘ChatGPT’s impact on student learning outcomes: a meta-analysis of 35 experimental studies’. Humanities and Social Sciences Communications, 13, 684. 10.1057/s41599-026-07019-z
13. OECD (2026). PISA 2025 Results (Volume I): Future-Ready Students. Paris: OECD Publishing. 10.1787/73451bc5-en
14. OECD (2026). PISA 2025 Results (Volume I): Spain. Country note. OECD. Country note

The post AI, problem solving, and the future of STEM education appeared first on Mapping Ignorance.