The physical sciences are replete with high-fidelity simulators: computational manifestations of causal, mechanistic models. Ironically, while these simulators provide our highest-fidelity physical models, they are not well suited for inferring properties of the model from data. I will formulate the emerging area of simulation-based inference and describe how machine learning and probabilistic programming techniques are being brought to bear on these challenging problems. Finally, I will provide examples of how these techniques can impact particle physics at the Large Hadron Collider, astrophysics, neuroscience, and public health.