Monitoring of AI Systems

The main aim of this project is to develop new methods for monitoring of neural networks with a particular focus on improving the scalability and accuracy compared to current state-of-the-art methods. Moreover, we intend to explore real world applications for such monitoring algorithms and investigate how we can ensure these algorithms are explainable to the end users.

The key research objective is to create scalable and accurate monitoring methods for neural networks. Neural networks that are deployed in safety-critical applications such as in autonomous vehicles or aviation might not be able to extrapolate from the training data to unseen examples encountered at deployment time. Thus, a monitoring algorithm that checks for spurious outputs at deployment can help determine data points that the neural network is likely to give inaccurate results for. We can use abstractions to determine of a new input to the neural network is out of distribution (OOD) from the training set or if the neural network is behaving in an unexpected way. The goal is to create a monitor that is lightweight and accurate that can work alongside a machine learning model.

Furthermore, another research goal is to determine how monitoring algorithms would be used in real world settings and to consider the challenges in these situations. Current literature in this field focuses on autonomous vehicles but other safety-critical applications in medicine, finance, and aviation also require scalable solutions to OOD detection. It is likely that there is not a one-size-fits-all approach.

Finally, another aim is to improve the explainability of these monitors. Explainability of monitors will help engineers determine when to retrain or repair networks. It will also aid other users of machine learning models, such as regulators, to understand how safety can be guaranteed. This will lead to greater trust in the use of these models in real world domains.

To achieve these objectives, I will extend my...