AI in Education: What’s Actually Improving Learning?

Michelle Marlowe, the Director of Education at Pocket Prep, shares her insights into which aspects of AI in education are actually improving the learning experience for students. This article originally appeared in Insight Jam, an enterprise IT community that enables human conversation on AI.

As an educator of busy working professionals, I know learners often assume quick answers equate to learning and knowledge retention. AI’s speed in providing information makes it an appealing way to learn; however, it often harms the learning process and knowledge retention.

At Pocket Prep, we help people prepare for certification exams that determine whether they get licensed, promoted, or hired, and most already use AI in some form. As the person responsible for ensuring our learners pass exams and retain the knowledge they need to apply it in the field, I am always researching best practices for implementing AI in education. AI improves learning when it’s designed to make the learner do the thinking, and it gets in the way when it does the thinking for them.

What’s Working: AI That Makes Learners Think

Tennessee middle schools followed more than 6,000 middle schoolers through computer-based math practice, and the students with an AI tutor worked more slowly and answered fewer questions than classmates using standard software. Based on output alone, the AI tutor appeared less productive; however, one week later, the AI group scored higher on a follow-up test. The biggest improvements came when the tutor used a mastery rule that required students to correctly answer three questions in a row after any mistake before they could proceed. In these findings, the AI tutor helped because it made students do more of the thinking themselves.

That finding aligns with what learning science has shown for decades. Retrieval practice, spaced review, and desirable difficulties work because the effort of pulling something from memory makes it stick in a way that rereading or being handed the answer doesn’t. Well-designed Socratic questioning also works by surfacing a learner’s assumptions and asking them to reason forward instead of handing them the conclusion. When an AI tool is built to ask rather than tell, it strengthens the processes we already know produce learning. AI tutors can increase learning gains or lower scores, and it all depends on how educators design the tutor.

What Isn’t Working: AI That Hands Out Answers

In a randomized trial at the University of Maryland, nearly 2,400 undergraduate students took courses where some groups had access to a GPT-4o study assistant designed around the instructor’s materials. In those groups, students earned lower final grades than classmates in the same course without the study assistant.

In reviewing the student interactions with the AI study assistant, researchers found that nearly three out of four student requests asked the assistant for answers, information, or explanations; fewer than one percent asked for feedback on their work; and use of the tool spiked right before exams. The study assistant did have a guided tutoring mode option, but only two instructors turned it on, so nearly everyone used the default setting that gave direct answers, which students mistook as learning.

RAND’s August 2026 survey found that about two-thirds of young people believe that relying on AI for schoolwork hurts critical thinking. However, knowing a shortcut hurts long-term learning has never been enough to stop someone who’s tired and short on time, which is why the responsibility falls on educators to design effective AI tools.

What we’ve learned building a Socratic AI tutor

Our learners are nurses, IT professionals, emergency medical providers, and people in dozens of other fields, and many of them have short windows of time to study. In those moments, an AI tool that simply explains why C is correct feels like learning, even when very little is being retained.

That’s why we built our AI Tutor around Socratic questioning. When a learner misses a question, the tutor doesn’t lead with the explanation; it asks what led them to their answer, probes the assumption behind it, and nudges them toward the concept they’re missing, so they can put the reasoning into their own words.

In reviewing our learners’ tutor logs, I found a learner studying for the CompTIA Network+ missed a question about which tool an attacker would most likely use, and after a few exchanges, they typed, “Oh, I get it.” Many tools would end the conversation and the learning process there, but our tutor dug deeper and asked them to explain, in their own words, why the tool they’d chosen wouldn’t be an attacker’s first choice. “I get it” tells us how a learner feels, while explaining it shows us what they actually understand. Putting an answer into your own words helps the knowledge stick, and it’s what lets you recognize the same concept when it shows up in a different form on exam day.

Educators must design AI tools with a limit on productive struggle. Questioning only works until it tips into frustration, and a good tutor has to recognize when a learner needs a stronger hint or simply the missing fact to keep reasoning. AI works best as a layer on top of retrieval practice and spaced review rather than a replacement for them.

What to Consider Before Adopting or Creating an AI Learning Tool

These questions will tell you a lot about whether an AI tool is designed to improve learning:

1. Does it ask before it tells? Watch what happens when a learner gets something wrong. If the first thing they see is a full explanation, the thinking has already been done for them.
2. Is it aligned to what the learner is actually tested on? A tutor can be accurate and still teach the wrong things, so ask how the content maps to the current exam blueprint or competency framework and how quickly it’s updated when that changes.
3. What does it measure? Engagement and completion are easy to inflate, so ask whether the tool can show delayed retention or performance on new problems, not just activity.
4. What happens when the AI is wrong? Every generative tool makes mistakes, so ask how errors get caught, who reviews flagged responses, and how quickly fixes reach learners.

Learning is still the learner’s work

So how is AI actually improving learning? When it’s designed well, it does what effective teachers have always done: it asks questions, waits for the learner to answer, and gives just enough support for them to get there on their own. Whether the educator is a person or an AI, the job is the same: keep the learner thinking critically at every stage, from the first time they encounter a concept to the moment they need to recall and apply it. That means building AI tools around how people learn, not around how quickly they can get an answer. For those of us building in this space, the question that matters most isn’t how much AI can do for learners, but how much it can help them do for themselves.