Enhancing Ensemble Diversity in Neural Ensemble Search for Uncertainty Quantification

This project falls within the EPSRC `Artificial intelligence technologies' research area.
It is an investigation into how to enhance uncertainty quantification through ensembles of deep neural networks.
Especially for safety-critical applications such as autonomous driving or medical diagnosis, uncertainty quantification is relevant, however, neural networks are often badly calibrated and display either overly high or low confidence in their predictions. A popular non-Bayesian approach to improve calibration in neural networks is deep ensembles which average predictions of neural networks that have been trained from different random initialisations thereby achieving competitive predictive accuracy and calibration.
To further enhance performance, methods proposed in the literature for automatically constructing ensembles of neural networks with varying architectures. They show that architectural variation leads to higher ensemble diversity resulting in ensembles with higher uncertainty calibration and robustness compared to the previous state-of-the-art deep ensembles.
To automatically choose base learner architectures they suggest random search in combination with a greedy selection algorithm (NES-RS), an approach that is easily parallelisable, and a more sophisticated evolutionary algorithm based on regularized evolution (NES-RE), which exhibits better performance. Concurrent to this, Wenzel et al. proposed hyper-deep ensembles which aggregate ensembles over different random initialisations stratified over multiple random hyperparameters. However, they keep the architecture of their base learners fixed. The authors show that this approach for inducing diversity into ensembles also leads to increased predictive accuracy and calibration compared to deep ensembles.
Therefore, the question arises whether a more directed search for hyperparameters can benefit the construction of more accurate, well-calibrated, and robust ensembles of neural networks, and, moreover,...