Grade the Critique, Not the Artifact: Assessment Once Everyone Generates

In the summer of 2023, researchers at the University of Reading in England tested their own university’s exams. Through 33 fake student accounts, they submitted answers written entirely by an artificial intelligence (AI) chatbot to take-home exams in five undergraduate psychology modules. None of the markers knew. The markers flagged 6 percent of the AI submissions, and in four of the five modules the AI answers earned higher grades on average than the real students’ work.

A finished answer used to be fair evidence that the person who handed it in could do the work. Once a tool can produce a passable one on request, grading the answer tells a teacher or an employer very little. If I were redesigning assessment, I would hand people a plausible piece of work with something wrong in it. Then I would grade how well they find the flaw and defend the finding.

What the practice scores hid

A randomized trial published in the Proceedings of the National Academy of Sciences in 2025 shows how far the work and the learning can drift apart. Hamsa Bastani and colleagues worked with nearly 1,000 mathematics students at a high school in Turkey. Every student practiced with textbooks and notes, and some classes also had one of two AI tutors. Those given a version that worked like a standard chatbot scored 48 percent higher on the practice problems than students with no tutor. On the closed-book exam that followed, without the tool, they scored 17 percent lower than students who never had it.

In the researchers’ own tests, that chatbot gave a correct answer to the practice problems only 51 percent of the time. The authors suggest students were “either unable to detect these failures or unwilling to spend the effort needed to check correctness.” The students using it did not perceive that they had learned less. A second version, built to give hints and withhold answers, largely removed the exam penalty.

What I see in engineering

At TalentAid, roughly 95 percent of new code is written by AI. Our process got stricter over the same period. Every change now passes a mandatory human review plus automated checks, where before most changes were merged by their author within minutes. I used to average about three pull requests a week. Now it is around 15, at the same or better quality.

A review is a critique. The reviewer has to find what is wrong with a plausible change and argue for the fix. The skill that matters now is knowing when the AI is wrong, and you cannot outsource that.

A request can be plausible and wrong as well. I once promoted a DevOps engineer to DevOps lead because, when you gave him a task, he asked questions and pushed back. He tried to understand what you really wanted and how that would develop over the next few years, and built around it.

Employers and universities are changing the test

Some hiring tests have hit the same wall. Canva tested the questions from its Computer Science Fundamentals interview on AI tools, and the tools “produced correct, well-documented solutions in seconds,” the company wrote on its engineering blog in June 2025. The interview it piloted to replace that screen expects candidates to use AI. One of the things it assesses is: “Can they identify and fix issues in AI-generated code?” GitLab’s careers page lets candidates research take-home assignments with AI if they cite it, and asks for answers “in your own words that reflect your original synthesis and perspective.” In technical assessments, it welcomes transparent use of AI “with the expectation that you can explain your prompts and technical decisions.”

At the University of Sydney, Adam Bridgeman, pro vice-chancellor for educational innovation, said in November 2024 that “any take-home assessment can be completed to a high level by AI.” The university announced a two-lane approach from the second semester of 2025: secure, in-person assessments without external aids, and open assessments that allow AI and other tools. A staff guide by Bridgeman and Danny Liu describes open assessments as focused on “acquiring skills and critical thinking processes, rather than grading the outputs of such tasks.” It lists a “Q&A following presentation, submission or placement” among the secure formats proposed for 2025.

How to set a critique

Start with a plausible piece of work containing flaws you planted. That might be an essay that misreads its main source, or a function that breaks on an empty input. Ask people to find the problems and rank them by how much they matter. Score each answer against the planted flaws and deduct for false alarms. A real problem you missed when you built the task earns credit too. Then have each person defend one finding in conversation while you push back. Sydney’s guide reports that educators using interactive oral assessments find a 15-minute conversation explores student understanding more deeply than reading a written report.

There is a learning case as well. Bruce McLaren, Deanne Adams, and Richard Mayer studied 390 sixth-grade students learning decimals at two Pittsburgh-area schools. One group worked through a fictional student’s solutions, finding and fixing the errors in them. The other group solved the same problems itself. On an immediate test, the two groups did not differ. A week later, the error-finding group did significantly better. They also liked their lesson less.

The cost moves to whoever sets the task. A planted flaw has to survive a casual read, and the assessor has to understand it well enough to argue about it.