Publication note20267 min read

Deferring to experts the model has never met

Why expert identity can become a shortcut, and how an identity-free architecture adapts from only a small context set.

The expert has changed

A learning-to-defer system may be trained around one group of experts, then deployed with people it has never seen: staff rotate, trainees progress, and specialities differ between sites. A few examples of a new expert's past decisions can help—but only if the model learns the right thing from them.

Population-based methods typically encode class-indexed context into an expert embedding. That exposes absolute label identities to the router. It can learn a shortcut such as “defer when class 3 is strong” rather than the transferable rule “defer on whichever class this expert is best at.” A harmless relabelling can then break the policy.

Fixed coordinates

Identity-conditioned

The router can attach behaviour to a class index. Rename the classes and the shortcut no longer means the same thing.

Structural roles

Identity-free

The router receives only transferable roles and symmetric summaries. Class names never enter the decision.

A competence profile, not an identity

Identity-Free Deferral (IFD) builds a Bayesian competence profile from a small context set of the expert's previous predictions. For each class, a Beta posterior represents both estimated competence and how uncertain that estimate remains:

\theta_k^E \mid \mathcal D_C^E \sim \operatorname{Beta}(\alpha_k^E,\beta_k^E)

The rejector does not see the vector in its original class order. It reads values at roles such as the model's top class and the expert's estimated best class, alongside symmetric aggregates. Training uses a lower-confidence signal—roughly $L_k^E=[\mu_k^E-\alpha\sigma_k^E]_+$ —so poorly observed competence profiles are naturally downweighted.

1ObserveA few labelled past decisions from the new expert.

2ProfileEstimate per-class competence and posterior uncertainty.

3Re-indexConvert class coordinates into structural roles.

4RouteCompare model confidence with the expert's relevant skill.

The shortcut, made visible

Toy experiment comparing L2D-Pop and Identity-Free Deferral when expert proficiency shifts from class 0 to class 2 — Large markers are deferred examples. L2D-Pop's pattern remains tied to the original class identity (top row); IFD moves its deferrals with the expert's speciality (bottom row). From Figure 2 of the paper.

This controlled experiment holds the data and classifier fixed, then changes only which class the expert handles well. The baseline keeps deferring to the old identity. IFD's handovers move to the new speciality—the permutation-invariant behaviour the problem requires.

More context should mean a better handover

Six line charts showing Identity-Free Deferral improving as the number of expert context examples increases — IFD (green) converts additional test-time context into higher-quality routing for both in-distribution and out-of-distribution experts. The population baselines remain comparatively flat. From Figure 3 of the paper.

Across dermatology, blood-cell microscopy, liver imaging, and ImageNet-16H with real human annotations, IFD matched or exceeded the alternatives. The largest gains occurred for variable specialists and out-of-distribution experts—precisely where an identity shortcut is most damaging. On the liver-tumour benchmark, the improvement in area under system accuracy reached 0.10 over the next-best method.

Conclusion

Generalising to a new expert is not only a data problem; it is an invariance problem. If class identity should not determine a handover, the architecture should make that dependence impossible. IFD does this by replacing names with roles—and gains a policy that follows expertise when the expert changes.