Image: A visual interpretation of AI training. The training sequence is x0, x1, x2. The yellow and red are high error (mountains/hills), and the blue (valleys) is a low error. The sequence moves in the direction of steepest descent.
So far in this series of posts, I haven’t really discussed much of my own research from a technical perspective. In some ways, this is because it’s quite mathematical and is hard to motivate unless you have familiarity with training dynamics in machine learning. In this post, I will provide an informal and intuitive picture of what my PhD Thesis attempts to understand, and I will mostly be using a geometric picture to help you visualize what questions I am asking.
The behaviour of an AI System is determined by what is learned through a procedure called “training” or optimization. While much of the work in AI Safety seeks to understand and control AI behaviour after training is done, my thesis posits that to have safe systems we should at least understand how unsafe behaviour emerges and if we can stop learning unsafe behaviour to begin with, then we can control how these systems behave.
It turns out that this training procedure has a nice geometric or visual interpretation. We will say that the training procedure has a “trajectory” or “path” which is just the sequence of updates made to the models used in the AI system. Picture a landscape with lots of hills and valleys. The height of the landscape is the error or how bad the model is doing at mimicking behaviour in a dataset or getting reward from a sequence of actions. So, the goal of training is to find the lowest possible valley in this landscape.
The safety problem is this: some of these paths yield dangerous and unsafe behaviour. That is some valleys are not ones we’d like to end up in. How can this be prevented?
First, we ask, how does the training path evolve? In toy models, there is a simple answer, the path of steepest descent is taken. However, this is an unanswered question for modern deep learning systems due to the complexity of the systems themselves and the complexity of the tasks they are trying to model. In some very meaningful sense, we are not even sure if an answer is possible or what a satisfying answer would be. For example, the landscape is not 2 dimensional but usually many billions of dimensions for modern networks, so any kind of exact characterization is computational intractable. Even counting the number of valleys (Kac-Rice Theorem) is intractable let alone locating them. Further, the landscape is what is known as a Random surface, because we don’t have access to all the data during training or the data changes over time, the landscape evolves and not always in smooth ways. If that all wasn’t hard enough, the landscape has singularities like black holes and worm holes. Yes, machine learning has black hole like phenomena! (Though it’s hardly as dramatic, it usually just means some direction the landscape just disappears suddenly or gets folded in onto another direction as weird as that sounds).
Back to the safety problem, how can we guarantee what paths are taken? In my research I developed a result that allows us to control (or determine) how steep the mountains and hills must be with mathematical certainty. Why is this useful? Remember when I said in toy models the path taken at each step basically consists of taking a step in the steepest direction? Well, we need to determine how big of a step to take in that direction. The curvature (or steepness) of the mountains and hills is what limits our maximum step size. This is more intuitive than it sounds. If we are on a really steep hill and we take too big of a step, we fall! In machine learning we don’t fall but we do completely overshoot any valley we were trying to get to. As a result, the steeper the mountains and hills, the smaller steps we must take. We call this control “convergence rate control” which essentially means by making the landscape very steep, we can increase the number of steps needed to be taken.
What does this mean for safety? Our results show that we can arbitrarily increase the number of steps necessary for a training path to reach a valley. We can apply this to highly capable AI systems and effectively prevent them from being able to be trained. So far, our research only tackles making all mountains and hills steeper, so the method prevents any training whatsoever (while still allowing the model to be used). We are currently working on methods that would only make the mountains and hills around unsafe regions very steep which would allow us to make training models to be unsafe significantly harder.
