Publication note20268 min read

Coherent hierarchical learning to defer

How clinical label hierarchies turn deferral into a structured decision—and how we prevent contradictory handovers.

The handover is part of the prediction

A chest X-ray can contain several related findings at once. A model may predict a broad category such as lung opacity, more specific children such as consolidation or oedema, and decide independently for each finding whether to answer or defer to a radiologist.

That last step sounds like ordinary multi-label learning to defer, but the labels form a taxonomy. If consolidation is present, then lung opacity—and, above it, abnormality—must also be present. A set of individually sensible actions can therefore describe an impossible or confusing handover when combined.

Three ways independent deferral can break

Suppose a parent and child can each be marked absent ( $0$ ), present ( $1$ ), or deferred ( $\perp$ ). Treating those two nodes independently creates three distinct failure modes.

Taxonomic contradiction

The model says a parent is absent while asserting that one of its children is present.

Delegation violation

The parent is handed to the reader, but a child prediction already implies the parent's value.

Deductive defect

The model defers a child even though its own parent assertion has already forced the answer.

These are not just quirks of imperfect training. We show that a Bayes-consistent node-wise model can still produce delegation violations and deductive defects because its independence structure never reasons about the handover as a whole.

Defining a coherent action

We formalise a selective-exclusion handoff contract. Let $\mathcal{A}=\{0,1,\perp\}$ be the actions at each node. The parent action determines which child actions are allowed:

\Gamma(0)=\{0\},\qquad \Gamma(1)=\mathcal{A},\qquad \Gamma(\perp)=\{0,\perp\}.

An absent parent forces an absent child; a present parent leaves all options open; and a deferred parent permits an absent or deferred child, but not a present child that would answer the delegated question indirectly. The coherent action set is therefore

\mathcal{C}=\left\{\mathbf a\in\mathcal A^{|\mathcal V|}:\;a_c\in\Gamma(a_p)\;\text{for every edge }p\to c\right\}.

This turns coherence into a precise contract rather than a vague wish that neighbouring predictions should agree.

Two ways to enforce the contract

Both methods begin with the same primitive outputs: for each node, a learning-to-defer head estimates scores for absent, present, and defer. They differ in where the hierarchy enters the system.

MethodProjectionTBP + RPO

IdeaRepair local outputsLearn through the hierarchy

Coherence entersAt inferenceForward pass and training

GuaranteeExactly coherentNear-coherent fast decoding; exact with MAP

Best suited toA safe retrofitA learned, higher-utility deployment

1. Exact coherent projection

Projection chooses the highest-scoring joint action from $\mathcal C$ . A brute-force search would examine $3^{|\mathcal V|}$ action vectors, which becomes impossible even for a modest taxonomy. Because the labels form a tree, dynamic programming can solve the same optimisation exactly in $\mathcal O(|\mathcal V||\mathcal A|^2)$ time— effectively linear in the number of findings.

This is attractive when an existing node-wise model needs a reliable safety layer: no retraining is required, and every final action vector is coherent.

2. Taxonomic Belief Propagation with Recursive Policy Optimisation

Projection repairs a policy after it has been learned. Taxonomic Belief Propagation (TBP) instead pushes probability mass down the tree according to the allowed transitions. Recursive Policy Optimisation (RPO) then trains through that recursion, so losses at descendants also teach ancestors which actions preserve useful downstream options.

The result is a hierarchy-aware policy rather than a collection of independent heads. Its fast decoder is near-coherent; pairing the learned marginals with exact MAP decoding restores a hard coherence guarantee.

What happens empirically

We evaluate on four medical-imaging settings spanning real readers and controlled experts. The two plots below show the central trade-off on VinDr-CXR: system utility as more finding-level decisions are deferred, and the rate at which any incoherence appears.

System label F1 against deferral fraction on VinDr-CXR. Recursive Policy Optimisation and coherent projection retain strong utility across deferral budgets. — System F1 as the fraction of delegated finding decisions increases. The learned TBP + RPO policy is labelled BR-RPO in the figure.

Any incoherence rate against deferral fraction on VinDr-CXR. Independent binary relevance methods are incoherent, exact projection remains at zero, and Recursive Policy Optimisation stays close to zero. — Any-incoherence rate. Exact projection remains at zero by construction; TBP + RPO drives incoherence close to zero while the independent baselines fail much more often.

Across VinDr-CXR, CheXpert, PadChest, and ADPv2, exact projection removes incoherence by construction. TBP + RPO is almost coherent under fast decoding and often provides the strongest utility. That distinction is useful in practice: projection is a dependable guardrail, while joint training can learn a better division of work.

What I take from the paper

The main lesson is not limited to chest X-rays. A deferral system is defining a boundary of responsibility. Whenever decisions have structure—taxonomies, prerequisites, temporal dependencies, or shared constraints—that boundary should be represented explicitly.

Maximising the number of correct AI predictions is only part of the objective. A clinically useful handover must also be understandable: the reader should know what the model has decided, what remains open, and why their input is still needed. Coherence is one concrete step toward that more human-centred notion of utility.