What can a neural network lose before it forgets how to see?
- Context
- Bachelor’s thesis · 2023
- My contribution
- Hybrid pruning method, implementation, experiments and feature-map analysis
- Tools & setting
- PyTorch · LeNet-1 · MNIST
- Explore
- Source on GitHub
Less network.
More deliberate choices.
Explore the method, one filter at a time.
filters remaining
Approximate importance
Estimate Shapley values with Monte Carlo sampling. Increase the sampling budget as filters are removed.
The question
Pruning removes parts of a trained network to make it smaller. The difficult part is deciding what to remove, especially when there is not enough data to retrain the network afterwards.
I explored Shapley values as a way to assess a filter’s contribution in combination with other filters. Instead of asking whether a filter looks useful on its own, the method considers what it adds to different groups.
Approximate first.
Calculate exactly later.
Exact Shapley calculations become expensive as the number of filters grows. My hybrid approach starts with Monte Carlo sampling, increases the sample budget as pruning progresses, then switches to exact calculation once the layer is small enough.
I tested this on the 12 filters in LeNet-1’s second convolutional layer using MNIST handwritten digits. Filters were removed one at a time, without fine-tuning between steps. The comparisons were random pruning and a fixed-budget Monte Carlo method.
A clear baseline difference.
A small hybrid advantage.
Area under the accuracy-versus-pruning curve. Higher means accuracy was retained better across pruning steps.
Exact values and experiment notes
| Method | Accuracy AUC ↑ | Loss AUC ↓ |
|---|---|---|
| Random | 0.65 ± 0.03 | 0.83 ± 0.02 |
| Monte Carlo | 0.78 ± 0.00 | 0.39 ± 0.00 |
| Hybrid | 0.79 | 0.37 |
The thesis reports three trials and standard deviations where applicable; it does not list a standard deviation for the hybrid column. These are areas under pruning curves, not final classification accuracy or ROC-AUC.
What the result does, and doesn’t, say
Both Shapley-based methods retained performance better than random pruning in these experiments. The hybrid method’s improvement over Monte Carlo alone was small: their curves were almost identical.
This was a study of a small network, not evidence that exact Shapley pruning scales to modern architectures. I also inspected feature maps to explore why filters received different scores, but treated those visual interpretations as hypotheses rather than proof.
Research, in full
The project connects implementation with a question I still find interesting: how do we decide which parts of a model are actually doing useful work?
The repository contains the pruning and graphing notebooks. The thesis was supervised by Dr. C. P. Lawrence.