Trustworthy assessment of heterogeneous treatment effect estimator via analysis of relative error

Accurate heterogeneous treatment effect (HTE) estimation is essential for personalized recommendations, making it important to evaluate and compare HTE estimators. Due to missing counterfactuals, traditional evaluation methods are inapplicable. Current HTE assessment methods rely on additional estimation or matching on test data, whose uncertainty is often ignored, leading to incorrect HTE estimator selection. We propose incorporating uncertainty quantification into HTE estimator comparisons. In addition, we suggest shifting the focus to the estimation and inference of the relative error between methods rather than the absolute errors. Methodology-wise, we develop a relative error estimator based on the semi-parametric efficient influence function and establish the estimator's asymptotic distribution for inference. Compared to absolute error-based methods, the relative error estimator (1) is less sensitive to the errors in nuisance function estimation, satisfies a "global double robustness" property, and (2) its confidence intervals are often narrower, making it more powerful for determining the more accurate HTE estimator. Through extensive empirical study of simulated data and ACIC benchmark datasets, we show that the relative error-based method more effectively identifies the better HTE estimator with statistical confidence, even with limited test data or inaccurate nuisance estimators.