Adversarial Training in High-Dimensional Regression: Generated Data and Neural Networks

In recent years, studies such as \cite{carmon2019unlabeled,gowal2021improving} demonstrate that incorporating additional real or generated data with pseudo-labels can enhance adversarial training through a two-stage training framework. In this paper, we perform a theoretical analysis of the asymptotic behavior of this method in high-dimensional regression problem when using two-layer neural networks. We first derive the asymptotics of the two-stage training framework using linear regression as a preliminary. Then, we analyze the convergence of two-layer neural networks in the two-stage framework. The analysis considers two different regimes: in the first stage of the framework, it is a high-dimensional regime, and in the second stage, the sample size is much larger than the data dimension. To analyze adversarial training, we track the change of the adversarial attack, and reveal that training with two-layer neural networks gives a prediction performance similar to training a linear model with some particular $\mathcal{L}_2$ regularization corresponding to different regimes. To highlight our technical contribution, we are the first to investigate adversarial training in two-layer neural networks under moderate attack strength, which is different from most existing literature in vanishing attack strength.