Publication note20267 min read

What the evidence says about human–AI collaboration in healthcare

A scoping review of 140 empirical studies—and why team performance depends on far more than an algorithm's accuracy.

The evidence is about teams, not tools

Calling a system “human in the loop” says very little about how people and AI actually share a task. The AI may act as a second reader, prioritise cases, draft an output for verification, or influence a treatment decision. Each arrangement changes what success means.

We therefore reviewed empirical human–AI collaboration studies across healthcare rather than asking whether AI was accurate in isolation. The review covers work published from January 2015 to October 2025 and maps where collaboration was tested, how it was evaluated, and which technical, human, and organisational factors shaped the result.

17,463records identified

140studies included

88diagnostic interpretation studies

57studies measuring trust

Where the evidence is concentrated

Alluvial diagram linking clinical task context, evaluation design, and reported outcome direction across 140 studies — How the 140 included studies move from clinical task to evaluation design and reported outcome direction. Task categories are not mutually exclusive. From Figure ii of the paper (CC BY 4.0).

Diagnostic interpretation dominates the literature. Eighty-six studies made a direct human-only versus human-plus-AI comparison; the rest used qualitative, survey, implementation, or other non-comparative designs. Positive-direction findings were common, but that should not be read as a pooled estimate of effectiveness: tasks, designs, and outcomes were heterogeneous, and many studies were controlled reader experiments.

What makes collaboration work?

Benefits depended on alignment between the technology, the person using it, and the organisation around them. Four recurring determinants cut across the literature.

Task

Complement the judgement

Bounded subtasks—prioritisation, highlighting, drafting, or pre-segmentation—were the clearest fit.

Interface

Communicate uncertainty

Explanations sometimes helped, but could also add cognitive load or invite unwarranted certainty.

People

Train for appropriate use

Onboarding and continuing support matter, especially when users must detect when the AI is wrong.

Organisation

Fit the real workflow

Integration, leadership, escalation routes, and shared success criteria shape whether a tool is usable.

Trust is not the same as reliance

Bar charts showing how trust was measured and which trust-related effects were reported in 57 healthcare human-AI studies — Trust was usually measured by self-report; only six studies used calibration metrics. Increased acceptance was much more often reported than calibrated or appropriate trust. From Figure iii of the paper (CC BY 4.0).

The goal should not be to maximise trust. It should be to support appropriately calibrated trust: reliance when the AI is likely to help, and scrutiny or override when it is not. A stronger evaluation triangulates subjective ratings, observed behaviour such as advice-taking and escalation, and calibration against objective correctness.

What better evidence would look like

Decision

Accuracy, time, workload, cognitive burden, and reliance at the point of use.

Workflow

Adoption, escalation, override, equity, and how work is redistributed across a service.

Patient & system

Clinical outcomes, resource use, safety events, and whether benefits persist over time.

Governance

Traceability, contestability, accountability, and explicit control over how AI shapes decisions.

Only 11 of the 140 studies used implementation-focused designs. Accountability and safety were often discussed but rarely tested. The evidence base now needs to move beyond short-term task comparisons toward longitudinal, in-situ studies of the complete clinical system.

Conclusion

Human oversight is not a safety property by itself. Collaboration works when the task is well matched, the interface supports appropriate reliance, the workflow makes intervention possible, and governance gives people meaningful control. The unit of evaluation must therefore be the human–AI system in context—not the model alone.