TESLA: Taylor Expansion of Sinusoidal Learnable Activations

The parity problem—deciding whether the number of ones in a binary vector is odd or even—remains challenging for standard neural networks due to linear inseparability and the need for global interactions. We propose TESLA, an activation defined as a learnable combination of sine and cosine terms whose Taylor coefficients are trained directly, enabling explicit control over polynomial degree and selective amplification of high-order components. Theoretically, we show that constraining TESLA’s coefficients yields Lipschitz/Rademacher complexity bounds and shapes the training dynamics to emphasize higher-frequency structure. Empirically, on parity with input length $n=32$, TESLA achieve perfect generalization using ~100K training sample ($\approx 0.002\%$ of the $2^{32}$ input space). Notably, TESLA maintains strong generalization under heavy corruption, retaining high accuracy with up to 30\% label noise in the parity signals. Beyond synthetic structure, TESLA delivers comparable performance on ImageNet-100, indicating that activation-level degree control transfers to more general vision workloads. These findings suggest that TESLA is an effective mechanism to improve expressivity and sample efficiency on tasks requiring global structure.