
Is it possible to steal an AI? It turns out it can be done quite easily! Last week I spoke about the differences between AI Safety and AI Security and today I’d like to talk about a topic that is very much in the category of AI Security known as Model Exfiltration. I will also talk about one of my own going lines of research: can we prevent stealing AI models?
Model Exfiltration is a collection of different techniques for stealing (parts) of an AI Model. Before we dive into a popular method known as “distillation,” let’s look at the anatomy of a modern AI system. A system like ChatGPT consists of (1) an architecture: this is like the skeleton, and it determines how the input data like a user prompt or image is passed through a sequence of transformations (known as layers or blocks) before finally transforming the input into an output. (2) the weights or parameters: this is like the muscles, nerves, and organs since these are the actual numbers that determine the actual transformations that will happen (3) the user interface: this is like the skin because it doesn’t matter so much to how the model behaves under the hood. There are more complicated things going on nowadays when ChatGPT says “Thinking about response” and so on but we will ignore that for now.
The “threat model” of model exfiltration is typically that the attacker wants to steal knowledge of the architecture and/or the weights to mimic the behaviour of the that model. We assume the attacker has enough knowledge and budget to train a simple AI system but not enough budget or knowledge to train a large scale ChatGPT. We also assume the model being stolen is “closed” meaning that we can only access that model through the web application.
To describe the attack, I am going to tell you some AI history. OpenAI only recently became a commercial enterprise focused on serving large language models. Before that they were a research organization, and they produced a model called GPT-3 to determine whether neural networks trained on internet text could accomplish other tasks without explicitly being trained for them (this is known as zero-shot learning). Now you must understand a little bit about language models work given a half-finished sentence, they predict the next word by giving a probability to tens of thousands of possible words, the highest probability word is then selected for generation. So, when GPT-3 was first exposed to the public through a web application the important part of this story is that OpenAI gave researchers access to all ten thousand probability values.
Geoffrey Hinton and his students at the University of Toronto showed in 2014 in the paper “Distilling the Knowledge in a Neural Network” that if you have access to these probability values then you can easily train a small “student” model that mimics the original “teacher” (the one you have probability values from). This is known as “distillation.”
Because GPT-3 was highly capable but both closed-access and very large, researchers started to use these probability values to “distill” the model to open-access and small models. During their transition to a commercial company, OpenAI stopped providing these probability values. Why? because this was Model Exfiltration; their models were being stolen. To this day, closed-access model providers no longer expose these probabilities.
Largely my PhD thesis focuses on “open weight security” so we are currently asking the question becomes whether it is possible to prevent distillation from open models. Attackers already have access to open weights, but other parts of my research are developing techniques for making it possible for those open weights to be “locked” so they can only be used for authorized purposes. In this world, distillation becomes a major issue because attackers can distill the protected open weights into an unprotected model. Early results are showing that you can “scramble” the ranking of these probabilities as an effective means to create “undistillable” models, but you will have to stay tuned until later this fall when we have completed enough research to publish the work.
Image generated by Imagen4 with prompt: “16-bit SNES style image of a hacker downloading code from a robot”