Stochastic Pruning for Neural Networks

Pruning algorithms compress deep neural networks while incurring in minimal degradation of accuracy. We propose Stochastic Pruning: an algorithm in which dense solutions are perturbed with Gaussian noise prior to pruning. We use Stochastic Pruning to explore the basin of attraction around one solution, obtained with a single minimiser. Using ResNet18, ResNet50, and VGG19 on CIFAR-10 and CIFA R-100, we show that SP consistently improves one-shot accuracy after extreme pruning and mitigates feature variance explosion. Our analysis adds understanding by linking one-shot accuracy and local gradients.We show that fine-tuned SP accuracy can outperform deterministically pruned solutions in most cases. Finally, we leverage multi-objective evolutionary optimisation to compare single and two-objective results, e.g. optimised for both accuracy and sparsity. The good performance of SP provides a promising approach to extreme pruning, achieving viable high accuracy, and high efficiency neural networks in resource-constrained scenarios.