Inverse optimization aims to recover the unknown state in forward optimization after observing a state-outcome pair. This is relevant when we want to identify the underlying state of a system or to design a system with desirable outcomes. Whereas inverse optimization has been investigated in the algorithmic perspective during past two decades, its formulation intimately tied with the principal's subjective choice of a desirable state---indeed, this is crucial to make the inverse problem well-posed. We go beyond the conventional inverse optimization by building upon prediction market, where multiple agents submit their beliefs until converging to market equilibria. The market equilibria express the crowd consensus on a desirable state, effectively eschewing the subjective design. To this end, we derive a proper scoring rule for prediction market design in the context of inverse optimization.