Abstract: We study pixel-wise regression problems with sparsely annotated images. Traditional regression methods based on mean squared error emphasize pixels with labels, leading to distorted predictions in unlabeled areas. To address this limitation, we introduce Neural Point Processes, a novel approach that combines 2D Gaussian Processes with neural networks to leverage spatial correlations between sparse labels on images. This approach offers two key advantages: it imposes smoothness constraints on the model output and enables conditional predictions when sparse labels are available at inference time. Empirical results on synthetic and real-world datasets demonstrate a substantial improvement in mean-squared error and $R^2$ scores, outperforming standard regression techniques. On the real-world dataset COWC, we achieve an $R^2$ of $0.769$ with $81$ out of $40,000$ ($0.2$%) points labeled, while standard regression loss (MSE) results in an $R^2$ of $0.060$.