Despite many valuable advances in the domainof online convex optimization over thelast decade, many machine learning and networkingproblems of interest do not fit intothat framework due to their nonconvex objectivesand the presence of constraints. Thismotivates us in this paper to go beyond convexityand study the problem of stochasticcontinuum-armed bandit with long-termconstraints. For noiseless observations ofconstraint functions, we propose a genericmethod using a Bayesian approach based ona class of penalty functions, and prove thatit can achieve a sublinear regret with respectto the global optimum and a sublinear constraintviolation (CV), which can match thebest results of previous methods. Additionally,we propose another method to deal withthe case where constraint functions are observedwith noise, which can achieve a sublinearregret and a sublinear CV with more assumptions.Finally, we use two experimentsto compare our methods with two benchmarkmethods in online optimization and Bayesianoptimization, which demonstrates the advantagesof our algorithms.