Feature Importance via Sets of Locally Performant Linear Models

Understanding the contribution of individual features to a model’s prediction is critical in applications such as medicine. While feature importance methods aim to quantify how much a feature contributes to a model’s accuracy, they often overlook heterogeneous patterns in the data and suffer from limited robustness. We propose $\ell\text{-MCR}$, a local feature importance method that identifies meaningful neighborhoods around a point of interest, regions where the model or data behavior is locally stable and interpretable. Within these neighborhoods, we estimate feature importance using Model Class Reliance (MCR), which offers robustness by considering the full set of near-optimal models. We also provide a consistency proof for reliably detecting such neighborhoods. Experiments on both synthetic and real-world datasets demonstrate that $\ell\text{-MCR}$ captures localized feature importance patterns that global approaches fail to detect.