Is ChatGPT Really Ruining Education?

There has been a lot of discussion about a recent article in NY Magazine, titled Everyone Is Cheating Their Way Through College. It contains many anecdotes about the nearly omnipresent use of ChatGPT and other AI tools on college campuses. Even two years ago, a survey found that nearly all students had used ChatGPT in some form, and the use has just expanded. “Generative-AI chatbots — ChatGPT but also Google’s Gemini, Anthropic’s Claude, Microsoft’s Copilot, and others — take their notes during class, devise their study guides and practice tests, summarize novels and textbooks, and brainstorm, outline, and draft their essays.”

One of the students hit the nail on the head: “Most assignments in college are not relevant,” he told me. “They’re hackable by AI, and I just had no interest in doing them.”

One teacher interviewed claims that “Massive numbers of students are going to emerge from university with degrees, and into the workforce, who are essentially illiterate. … It’s short-circuiting the learning process, and it’s happening fast.”

In this episode, we’ll examine whether this is true: is the use of AI in schools really short-circuiting the learning process? We look at two repositories of incredible research that help us answer that question.

First, we look at the meta-analysis published on May 6 in Nature: The effect of ChatGPT on students’ learning performance, learning perception, and higher order thinking. The authors looked at 51 experimental studies from around the world, and they found “that ChatGPT has a large positive impact on improving learning performance and a moderately positive impact on enhancing learning perception and fostering higher-order thinking.”

I like meta-analyses because they tell the broader story, but I find it helpful to go into the trees and see what studies they referenced. Most of the research took place in colleges, but there were a few gems from the K-12 world. Here are some of the gems.

Game based learning with ChatGPT

In Taiwan, researchers tried ChatGPT embedded in a game to help seventh graders learn science concepts. They created a game, Summon of Magicrystal, and embedded a chatbot alongside the students to help them along their way. The control group just played the game, while the two experimental groups were augmented with assistance from GPT-3.5-turbo model. The experience looked like this:

They measured the student’s learning using a post-test. “Students who used the ChatGPT augmentation outperformed those in the Game only group. … The findings from learning behavioral analytics and interviews suggest that AI-assisted game-based learning can enhance students’ intrinsic motivation, reduce cognitive load, and promote effective learning behavior in science learning.”

ChatGPT helps students learn physics

In Dubai, a set of eleventh graders were studying electromagnetism in their physics class. Half of them were given access to ChatGPT to use whenever they needed help. This study was conducted two years ago in 2023, and was also using the older version of gpt-3.5.

The kids with access to ChatGPT did much better on the post-test than the kids without. The authors conclude that “ChatGPT as a learning tool resulted in significantly better achievement in electronic magnetism compared to traditional teaching methods.”

The Dubai study echoes similar findings found at Harvard. In their intro to physics class, freshman college students were given access to a ChatGPT-based assistant; similarly, post-test scores were much higher for those who made use of the tool in class.

The Harvard students were just a few years older than the Dubai 11th-graders, and they utilized a similar toolset in a similar environment.

Writing

One study out of Monash University in Australia tried to answer the question of how AI helps with writing. They analyzed the transcripts from 1,445 ChatGPT-assisted writing sessions. They didn’t just ask whether AI boosts scores. They tracked how students used AI - did they ask for lines and paste them verbatim, or ask and then remix those suggestions into their own words?

Students who simply copied ChatGPT’s text into their papers saw their lexical and syntactic quality plunge. Cohesion suffered too - until they actively edited the AI output. Writers who accepted suggestions and then rewrote them produced essays with richer vocabulary, more complex grammar, and smoother flow.

This study suggests our focus should shift to teaching students how to engage critically with those AI drafts—how to treat them like a rough draft of ideas, not a final product. In other words, it’s not ChatGPT that ruins writing — it’s skipping the work and effort of editing the output that hurts.

Not all rosy

Of course, not all the studies are showing such positive results. For example, at one school in Taiwan, high school students were offered the use of ChatGPT to help them study Intro to C++. The researchers in this case concluded “the direct use of ChatGPT to assist students in learning programming may not yield substantive benefits and may diminish their flow experience, self-efficacy, and learning achievement in the context of programming courses.”

So we find some mixed results. Still, the Nature meta-analysis concluded that on average the use of ChatGPT produces significant learning gains.

Stanford GenAI Research Repository

So the Nature article is a great snapshot of the research of the last few years specifically on ChatGPT - and most of that research was done on now antiquated models like GPT-3.5 and GPT-4 (I saw only one that included 4o and none that studied the more recent “thinking” models like o3). But that’s just one angle. There are many tools available beyond those offered by OpenAI, and the field continues to evolve rapidly.

Stanford has organized a repository of research papers. Every month, they curate the recent research, label the papers by their type (experimental, descriptive or whatever) and publish them into a searchable database. This dataset includes many of the papers talked about today. Visit the hub here. I particularly appreciated the intro to this research hub given by Chris Agnew on the AI Education Conversation podcast.

Takeaway

What we see here is that with any new technology, there will be risks and concerns as people have to adapt to those changes. We saw this with the rise of Wikipedia and Google, where suddenly students had access to sources on any topic that were previously hard to reach. And of course, students will try to use the latest tools to solve hard problems.

But educators also now have access to incredible tools that can help facilitate learning in many ways - helping teachers with lesson plans, as well as directly tutoring students. I think the evidence that has come out from recent studies points in a very positive direction that having individualized help provided by an LLM can be really positive for the learning experience.