Towards interpretable and controllable deep language modeling

Large language models (LLMs) such as ChatGPT have rapidly become ubiquitous in the technology sector and are being applied in a wide variety of increasingly important domains, including customer service, automated source code generation, and augmenting the efficiency of the UK civil service. However, these models are "black boxes" - no one, not even the researchers who created them, understand how these AIs "think". This makes it effectively impossible to make any guarantees about their behaviour, including their safety, resilience to malicious inputs, and lack of dangerous capabilities. At best, organizations such as the UK government's AI Safety Institute can perform empirical evaluations of these properties, but the results of these evaluations may not generalize beyond the distribution of inputs on which the model was tested.
The PhD project will develop novel methods for overseeing, steering, and controlling these increasingly powerful and influential systems which will enable us to design real-time automated oversight solutions based on the model's internal states. It will begin by examining the application of sparse autoencoders (SAEs) to the analysis of the latent spaces in the model. SAEs have become broadly popular in the field of mechanistic interpretability due to their ability to locate certain "concepts" within the model, but it remains unclear whether they have sufficient range and flexibility to discover the full range of causally relevant concepts.

For instance, if we detect that a model's internal state suggests that it is helping a user engage in illegal activity, we can flag the interaction for manual review. The hypothesis is that there are types of concepts which SAEs are not well-suited for, and therefore any conclusions about the safety properties of a model based on SAE-powered analysis may be misleading. Concrete case studies where SAEs fail to accurately represent important concepts will be found and then...