Skip to yearly menu bar Skip to main content


Bayesian Online Learning for Consensus Prediction

Sam Showalter · Alex Boyd · Padhraic Smyth · Mark Steyvers

MR1 & MR2 - Number 123
[ ]
Thu 2 May 8 a.m. PDT — 8:30 a.m. PDT


Given a pre-trained classifier and multiple human experts, we investigate the task of online classification where model predictions are provided for free but querying humans incurs a cost. In this practical but under-explored setting, oracle ground truth is not available. Instead, the prediction target is defined as the consensus vote of all experts. Given that querying full consensus can be costly, we propose a general framework for online Bayesian consensus estimation, leveraging properties of the multivariate hypergeometric distribution. Based on this framework, we propose a family of methods that dynamically estimate expert consensus from partial feedback by producing a posterior over expert and model beliefs. Analyzing this posterior induces an interpretable trade-off between querying cost and classification performance. We demonstrate the efficacy of our framework against a variety of baselines on CIFAR-10H and ImageNet-16H, two large-scale crowdsourced datasets.

Live content is unavailable. Log in and register to view live content