Why ‘Human-in-the-Loop’ Fails Agentic AI, and How to Build Institution-Level Safeguards

Anand Madhav, Vice President of Analytics & AI at Gramener, explains why human-in-the-loop efforts are failing agentic AI, and what teams can do to develop institution-level protection. This article originally appeared in Insight Jam, an enterprise IT community that enables human conversation on AI.

AI accountability is doing the rounds, but I constantly see firms missing one crucial reality in the discussion: a human in the loop is now a hollow reassurance. Presence does not mean oversight.

Agentic AI is about to widen that gap. Research from Gartner shows that agentic AI will make at least 15% of daily work decisions autonomously by 2028, up from 0% in 2024. These aren’t passive recommendations pending human sign-off but self-executing actions, such as releasing a payment, messaging a customer, updating a line of code, or changing a configuration. A human reviewer who can process one recommendation in an hour simply can’t match the pace of an AI agent executing hundreds of decisions a minute. Instead, the institution in the loop, not just a human, must become the safety net.

The safety net is approving what it cannot see

Beyond the speed paradigm of the agentic AI challenge is another problem to contend with: how people behave and think around a confident algorithm. We’re seeing that manifest as cognitive surrender, a term recently coined by researchers at the Wharton School after running experiments with 1,372 participants across thousands of trials. Cognitive surrender is the point at which a person stops evaluating AI-generated outputs and simply adopts them.

In a survey of a thousand U.S. workers who use AI, only 17% said workplace AI was reliable on its own, and 70% said reliability came from a human safety net, either light review or dedicated oversight. The trust model and safety net sit on the person, but we know that humans are prone to routinely endorsing an output, especially under time and volume pressures.

When the share of agent outputs a human approves creeps toward 100%, it’s tempting to presume the system has improved. But usually it means reviewers have stopped reviewing, or cognitive surrender is affecting your operations.

It’s a very real threat. Think verification steps that gradually become less consistent, an invisible change to anyone checking individual outputs, and really only detectable by comparing behavior across runs against an earlier baseline.

Building the institution into the loop

I worked with a company where a simple question from leadership was a mystery: how much had their biggest supplier been paid that year? Suppliers existed in their systems under multiple, slightly different names, the result of years of company acquisitions, each with its own unmerged finance software. As a result, supplier payment processing was slow and painful.

The solution was to have three agents work on every invoice in parallel. The first agent resolved identity, confirming name variations through cross-referencing core data such as tax numbers, addresses, and bank details. The second dealt with contracts. Supplier deals span years, with an overlapping master agreement coupled with layered amendments, and this agent assembles the full picture to ascertain which part of which contract an invoice belongs to. The third checked that the work was actually done, comparing delivery records and timesheets with what the contract says should be paid. When all three agree, the payment goes to the ERP.

Inserting a human at every checkpoint wasn’t feasible for a firm processing tens of thousands of invoices a quarter. It would have rebuilt the old backlog, just with more sophisticated software. Here’s where risk-tiered review comes in. This spans three levels: invoices that match their purchase order within tolerance pay automatically, and anything unusual routes to a reviewer within 24 hours, with the agent’s reasoning attached. A handful of actions, such as adding a new payee, changing bank details, or making a payment above the threshold, always require a named human to approve. The typical split is 85% of invoices processed automatically, 10-12% reviewed, and 2-3% held for approval. The tier attaches to the action, never to the agent, so the same agent doing two different things carries two different risks.

Making it traceable

Provenance is the other side of the institution in the loop, and it’s becoming less of a competitive advantage and more of an expectation. Every consequential agent action should carry a record written at the moment it’s produced—the model and version, the triggering prompt, the data touched, the risk tier, the accountable owner, the escalation path. That enshrines a contemporaneous record as evidence. I’ve seen firms do this well, including one that wrote its AI governance as an actual shared code library that every team plugs into, so tracking and review hooks attach to the platform regardless of who builds the application on top of it.

The infrastructure for this isn’t exotic. It’s a gateway—a single checkpoint that every agent action passes through—paired with a policy engine, such as OPA or Cedar, that approves or blocks each action based on defined rules.

A vital principle to note is that enforcement lives at the gateway, not the prompt. This is not a feature provided by open-source agent frameworks that handle orchestration logic; that’s why this tends to be purpose-built middleware sitting alongside them. Approvals should land in tools reviewers already use, such as ServiceNow and Jira, since asking busy people to learn a new interface hastens abandonment.

Circuit breakers close the loop. An agent that crosses an error-rate or correction-rate threshold loses the right to act automatically. The last resort, revoking an agent’s credentials outright, has to be rehearsed like a fire drill, and in my experience, those rehearsals reveal credentials nobody remembers issuing.

And note that regulations are also working to close the gap. The EU AI Act’s Article 14 requires that high-risk systems be designed so humans can meaningfully monitor, understand, intervene, and halt them. Article 11 requires providers to maintain technical documentation covering governance, monitoring, and risk management, available to regulators on request. The NIST’s AI Risk Management Framework in the US also recommends this kind of governable AI-human collaboration.

Who moves first, and where to start

Adoption doesn’t track regulation as closely as you’d expect. Outside of a few explicit mandates, most companies are building these safeguards on their own calculated risk. The better predictor of who moves first is how connected their data already is.

Banks, payment companies, and online retailers hit the volume at which casual review breaks down first, so they usually build governance first. For everyone else, the pressure is likely to come through contracts before it shows up in the law, similar to how SOC 2 spread. Ultimately, buyers increasingly want any agent touching their systems, including a vendor’s, to produce these decision records as a condition of doing business.

Start with an audit, not a framework. Pick your most consequential agent and determine: can you trace it to its origin? Name who was accountable? Reconstruct the reasoning? Show where it should have escalated? Most firms can’t, but there’s a workable plan.

Capture a census of every agent, including those hidden inside off-the-shelf software. Then build decision records and a single gateway in front of the top two agents. After that, run a live tiering workshop on one high-value workflow. The goal is a single workflow that answers all four questions on demand and is presentable to an audit committee tomorrow. The longer version spreads the same machinery everywhere and adds monitoring, tracking of approval rate trends, escalation frequency, override rate, and time-to-traceability for a randomly sampled decision.

None of this removes the human, but it does change their purpose. The procurement reviewers I’ve seen don’t fail for lack of judgment. The volume forces them to skim. The work now is building the institution that catches what no individual, however capable, can be expected to catch alone.