Skip to yearly menu bar Skip to main content


Poster

Anytime-Valid A/B Testing of Counting Processes

Danqi Liao · Brian Cho


Abstract: Motivated by monitoring the arrival of incoming adverse events such as customer support calls or crash events from users exposed to an experimental product change, we consider sequential hypothesis testing of continuous-time counting processes. Specifically, we provide a multivariate confidence process on the cumulative rates (ΛAt,ΛBt)(ΛAt,ΛBt) giving an anytime-valid coverage guarantee P[(ΛAt,ΛBt)Cαtt>0]1αP[(ΛAt,ΛBt)Cαtt>0]1α. This provides simultaneous confidence process on ΛAtΛAt, ΛBtΛBt and their difference ΛBtΛAtΛBtΛAt, allowing each arm of the experiment and the difference between them to be safely monitored throughout the experiment. We extend our results by constructing a closed-form ee-process for testing the equality of rates with a time-uniform Type-I error guarantee at a nominal αα. We characterize the asymptotic growth rate of the proposed ee-process under the alternative and show that it has power 1 when the average rates of the two process differ in the limit.

Live content is unavailable. Log in and register to view live content