When a company puts an AI agent to work, leadership’s first reaction is usually: “Fine, but everything goes past a person.” It is an understandable reaction. The research shows, however, that putting a person on top of an AI is not, in itself, a guarantee of a better result.
What the largest review on the subject says
In 2024, MIT researchers published in Nature Human Behaviour a systematic review and meta-analysis of experiments that compared three set-ups: people alone, AI alone, and people working with AI (Vaccaro, Almaatouq and Malone, 2024). The findings run against intuition:
- On average, the person and AI combination performed worse than the better of the two working alone (Hedges’ g = -0.23, a standard measure of effect size).
- The loss was concentrated in decision tasks. In creative tasks, the combination tended to be positive, although not statistically significant on its own.
- When AI outperformed people, adding a person made the result worse (g = -0.54). When people outperformed AI, adding AI improved it (g = 0.46).
Human oversight adds value where the person knows something the AI does not. Where they don’t, it tends to add noise.
Why people stop paying attention
The psychology of automation has studied this problem for decades. In a landmark review, Parasuraman and Manzey (2010) describe two phenomena: complacency, which arises mainly when a person is juggling several tasks at once, and automation bias, which leads people both to accept wrong proposals and to miss what the system has let slip. Both affect novices and experts alike.
A field experiment with 181 professional recruiters showed the same effect in practice. Recruiters supported by a better but imperfect AI did worse than those supported by a weaker AI. Those with the weaker AI spent more time on each CV and examined it more carefully (Dell’Acqua, n.d.).
How to design approvals that work
Concrete rules follow from this:
| Level | Examples | Rule |
|---|---|---|
| Low | Classifying documents, preparing drafts | The agent acts, and it is logged |
| Medium | Email to a supplier, work order | The agent acts within limits, with spot-check review |
| High | Payments, contracts, sensitive communications | The agent prepares, a person approves |
And for approval to add value:
- Only approve what the person can judge better than the system. For example, commercial context, the customer relationship or known exceptions.
- Show why and where from. Every proposal should come with the data used and its source. A person who sees the reasoning approves with judgement; one who only sees the result approves out of habit.
- Few approvals, but real ones. A queue of a hundred approvals a day breeds complacency. A queue of ten breeds attention.
- Measure the quality of approval. How many proposals are changed, and in what kind of case. If nobody changes anything, either the system is excellent or the approval has stopped being real.
Widen autonomy based on the track record
Every approval is a data point. After a few weeks you can see which proposals are approved without changes and which are corrected. The first can move down a level; the second show where the agent needs better rules. In this way the agent’s autonomy widens one type of case at a time, always based on the track record.
Sources
- Vaccaro, M., Almaatouq, A., Malone, T. (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour, 8(12), 2293-2303.
- Parasuraman, R., Manzey, D. H. (2010). Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors, 52(3).
- Dell'Acqua, F. (n.d.). Falling Asleep at the Wheel: Human/AI Collaboration in a Field Experiment on HR Recruiters. Laboratory for Innovation Science, Harvard Business School (working paper).