Sparse-Input Neural Network Augmentations for Differentiable Simulators