Sample Average Approximation for Alpha-Divergence Minimization with Exponential Convergence Guarantees

We study the problem of approximating an unnormalized target distribution using probability densities from an exponential family. Specifically, we establish convergence guarantees for a monotonic alpha-divergence minimization algorithm, which decreases the alpha-divergence at each iteration. To illustrate our theoretical results, we propose an implementable Sample Average Approximation algorithm that solves a discrete approximation of the original problem. Through a detailed analysis of the loss landscape and the algorithm's dynamics, we provide practical design guidelines which suffice to ensure its convergence.