When AI Tutoring Helps: Timing Support Around the Learner

Meryll Dindin, SVP of Product & Engineering at Parallel Learning, examines the situations when AI tutoring is most valuable. This article originally appeared in Insight Jam, an enterprise IT community that enables human conversation on AI.

A few months ago, I made the case that districts should test what students can do once the software is taken away. District leaders have been asking me the follow-up question: if the tutor stays, when should it step in?

In kindergarten through grade 12, the work of instruction is building cognition itself: attention, working memory, the capacity to hold an argument long enough to write it down. An adult who hands a spreadsheet formula to an assistant loses little. A 12-year-old who hands over the reasoning step skips the exercise the lesson was built around, which makes timing a central design decision in a tutor.

Help After the Mistake, Not Before the Attempt

Philip Oreopoulos and colleagues randomized more than 6,000 middle schoolers in Hamilton County, Tennessee, to AI tutoring or standard practice software, and, separately, to mastery progression or free progression, and then tested them a week later. Requiring mastery on its own produced no detectable gain on the delayed test. The clearest benefit the authors found came after mistakes: the tutor improved the student’s next attempt. Because it also offered help before a first attempt, the study shows where the benefit appeared without testing whether earlier help hurts. The best delayed results came when the tutor was built into the mastery workflow, and on the material students had practiced, that gain was only marginally significant.

The same team’s two-year trial of Khanmigo in 18 schools shows how rarely students reach for a tutor at that moment. The tutor was configured to coach, and gains were modest, similar to separate estimates for Khan Academy practice without AI. In the second year, the median student messaged it in only 17 percent of the practice sessions in which they made a mistake.

Keep a Reviewer in the Loop

In an exploratory trial in UK secondary schools, Eedi and Google DeepMind had a human tutor review every message the model drafted before a student saw it. Tutors approved ~74 percent without edits, and the researchers estimated a ~94 percent probability that supervised AI tutoring beat human tutoring alone on the first question of the next topic, measured inside Eedi. The sample was 165 students, which makes it a signal to test rather than a settled result, and Eedi has since launched a 1,525-student follow-up.

At Parallel Learning, we build the same way for clinicians: AI drafts and a licensed clinician decides, because the person who knows the student should judge whether a given message helps. The trial makes the same arrangement worth testing for students: the model drafts most of the tutoring, and a person approves, edits, or replaces each message before it goes out.

What I Would Use a Mastery Estimate For

Adaptive products commonly rely on knowledge-tracing models, which are trained and scored on one target: whether the student gets the next item right inside the same program. That makes a mastery estimate a good tool for choosing the next exercise, and a district can value it for that. It says far less about whether the student can do the work in March with the app closed. In Hamilton County, requiring three correct in a row did not, by itself, move the delayed test.

When a vendor shows a “mastered” badge, ask what it was validated against. If the answer is next-item prediction, call it a sequencing signal.

Try Before You Ask

My own working rule, for my teams and for me, is to use generative AI everywhere, but produce your own answer first when it’s one you want to stay able to produce.

Hamsa Bastani and colleagues reported in the Proceedings of the National Academy of Sciences that Turkish high school students who practiced math using an unrestricted GPT-4 interface scored 17 percent lower on an unaided exam than classmates without access, while a version that withheld answers showed no detectable penalty. Both versions ran on the same model; the guarded one also drew on teacher-written solutions and common errors. In a randomized trial with 120 undergraduates, André Barcaui found that students who studied with ChatGPT scored 57.5 percent on a surprise retention test 45 days later, against 68.5 percent for students who studied without it. Of the 120, 85 sat that delayed test.

Neither study followed children over years, and the undergraduate result may not hold for a nine-year-old. Both point toward an attempt-first design worth testing: the tutor asks for the student’s attempt before it offers the answer.

Scaffold or Crutch

A diagnosis does not tell you what a given student needs. Take two hypothetical students with the same dyslexia diagnosis and the same text-to-speech feature. For a seventh grader reading a science chapter well above their decoding level, listening to the text supports comprehension and frees them for grade-level thinking. For a second grader whose intervention goal is decoding, the same feature replaces the practice the goal exists to build.

AI is a scaffold when it supports comprehension, organization, and expression, and a crutch when it replaces synthesis, reasoning, and voice. Reading a passage aloud, breaking an assignment into steps, or turning a spoken idea into a first sentence the student then revises are scaffolds. Writing the paragraph for the student is a crutch. Because one feature can be either, depending on the student’s current goal, the setting belongs with the educator who wrote that goal.

What I Would Ask a Vendor For

Personalized learning, in most product definitions I see, adapts to what a student does inside the system: which item comes next, when to offer a hint, when to move on. That adaptation is also measured inside the system. The American Psychological Association (APA) expert report on learning with educational technology, published in September, states: “The real test of learning happens after the app is closed.” Its procurement guidance reads: “Ask vendors to provide independent data or validation showing knowledge retention or skill transfer that persists at least one week after use.”

Alongside that evidence, ask for:

• The answer boundary. Where in an interaction the tutor produces an answer instead of asking for an attempt, and whether a teacher can move that point for a given student.
• Subgroup results. Outcomes reported separately for students with individualized education programs (IEPs), English learners, and students reading below grade level, since an average can hide the students a tool fails.

Toward Precision Education

Precision medicine matches treatment to a specific diagnosis and checks the response early. Education can work the same way: identify what a student can and cannot yet do, target the intervention, and monitor progress against a measure outside the tool. When monitoring shows no progress after a few weeks, the plan changes that month. An end-of-year test reports the same problem after the year is gone. Special education already has the structure for this in IEP goals and progress reporting, and AI makes frequent measurement cheap enough to run.

A tutor that waits for the attempt and responds to the mistake is one a district can test for before signing.