Can an AI Tutor Help Students Learn? A University Trial Suggests It Isn't That Simple

When a university builds its own AI tutor, the obvious question is whether it helps students learn. The University of Maryland decided to find out before offering its tool across the whole campus. The answer, published as a working paper at the start of October, is more complicated than a yes or a no.

The trial

In the autumn 2025 semester, researchers at the university ran a randomised controlled trial with 2,379 undergraduates and 30 instructors, in subjects ranging from biology and computer science to English and public policy.

The tool, called the Virtual Study Assistant, is a chatbot built by the university's IT division on OpenAI's GPT-4o. It sits inside each course's online page and answers questions using the materials the instructor has posted there.

Instructors were grouped with colleagues teaching the same or similar courses. Within each group, some were picked at random to have the assistant switched on for all their students, and the rest had it switched off. That design lets the researchers compare like with like.

What they found

Grades did not improve. Where the comparison is fairest, between sections of exactly the same course, students who had the assistant finished the term about 4 points lower on a 100-point scale. Across all the participating classes the gap was about 3 points, but the researchers say that wider result is not strong enough statistically to count as firm evidence.

Students with access also took part less in their course's online activities: fewer page views, fewer active days, and much less recorded participation in things like discussions and quizzes. They still handed in homework at the same rate.

For first-generation students, the first in their family to go to university, the estimated drop in final grades was more than twice that of other students. The authors note that how precise these group-by-group estimates are varies.

Most students never used it

Only about one student in seven who was offered the assistant used it even once. Those who did mostly wanted answers: about three-quarters of their 3,460 requests asked for information, explanations or solutions. Fewer than one request in a hundred asked for feedback on the student's own work.

The settings mattered too. By default, the assistant simply gave the answer. A tutor mode that guides students towards the answer was available, but only two instructors ended the term using it.

So why did grades fall?

That is the open question. Lower grades showed up among students in those classes who never touched the tool, not only among those who used it. The researchers suggest that bringing the assistant in may have changed the classroom more broadly. In an end-of-term survey, which few students answered, those in the classes with the assistant were more likely to say their instructor encouraged AI use. Every student in the trial, in both groups, already had free access to ChatGPT and Gemini through the university. Students who used the assistant also reported asking their instructors fewer questions.

The lead author, Jing Liu, who directs the university's Center for Educational Data Science and Innovation, told Inside Higher Ed that instructors need much more training in how to use tools like this with their students. Justin Reich of the Massachusetts Institute of Technology, who was not involved, told the publication that a trial in real classrooms bundles many changes together, so the exact cause cannot be fully pinned down.

The university's provost, Jennifer King Rice, who is one of the paper's authors, said the study “raises questions that haven't yet been answered”. The university is holding off on making the assistant more widely available.

What it does and doesn't show

This is one trial at one university, in a working paper that has not yet been peer reviewed. It measured course grades, not independent tests of learning. It tested this assistant as it was set up, mostly in answer-giving mode. It does not show that all AI tutors lower grades: the paper itself points to other trials, such as one in a Harvard physics course, where a purpose-built tutor led to gains.

In The Age of Intelligence, I imagined AI tutors in 2036 that reach each learner in their own way and give teachers back time to teach (Chapter 12, “Education Reimagined”). The same chapter worries that making learning “friction-free” could lose something essential. The Maryland trial is an early, real-world reminder that switching a tool on is not the same as learning with it well, and that the teacher's role matters as much as the technology.

Sources

• Jing Liu and colleagues, The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University Trial, EdWorkingPaper No. 26-1598, University of Maryland, October 2026 (dated 30 September 2026). Also at doi.org/10.26300/y3f8-vh05.
• Kathryn Palmer, A University's Research on Its Own AI Tool Raised More Questions Than Answers, Inside Higher Ed, 7 October 2026.
• Watch the video: Can an AI Tutor Help Students Learn? on The Age of Intelligence YouTube channel.