Deep Gaussian Processes

In this paper we introduce deep Gaussian process (GP) models. Deep GPs are a deep belief net- work based on Gaussian process mappings. The data is modeled as the output of a multivariate GP. The inputs to that Gaussian process are then governed by another GP. A single layer model is equivalent to a standard GP or the GP latent vari- able model (GP-LVM). We perform inference in the model by approximate variational marginal- ization. This results in a strict lower bound on the marginal likelihood of the model which we use for model selection (number of layers and nodes per layer). Deep belief networks are typically ap- plied to relatively large data sets using stochas- tic gradient descent for optimization. Our fully Bayesian treatment allows for the application of deep models even when data is scarce. Model se- lection by our variational bound shows that a five layer hierarchy is justified even when modelling a digit data set containing only 150 examples.