
Ensuring the safe, secure, and responsible development of AI technology is, in part, being coordinated through an emerging governance structure: a network of AI Safety Institutes (AISI). Canada, the United States, Singapore, Korea, and the United Kingdom count among other countries who are attempting to set up AISIs. Some of these institutes are called AI Security Institutes while others retain the AI Safety title. While there are not precise ways of delineating AI Safety versus AI Security, this post will try to make the difference between the two as clear as possible.
The simplest way of presenting the difference between AI Security and AI Safety is whether we highlight attacks and vulnerabilities (AI Security) or expected behaviour under constraints (AI Safety). As an illustrating example, AI Security might focus on if an attacker is able to circumvent a safety guard (such as one that prevents the generation of deep fakes) and AI Safety focuses on the construction of that safety guard itself. This example is quite good because it also illustrates the interconnected nature between the two focuses: AI Security needs AI Safety to develop the safety solution to evaluate it, AI Safety needs AI Security to evaluate how robust the solution is in the “worst case.”
Readers unfamiliar with computer science might not know that “worst case” can have a particular meaning. A system (such as an algorithm) can be analysed based on whether the system is being used in a way that gives us best, average, or worst case expected behaviour. Let’s translate this to a safety setting.
Suppose you had a safety guard that prevented deep fake generation. Now imagine it was just a hard coded filter (a big list that matches user’s inputs and either lets them through or rejects them) based on data collected by users’ prompts to a model like Open AI’s Sora model. You can imagine it’s just a keyword-based filter.
In the best case, either the user doesn’t ask for generation of inappropriate content or the user’s queries are exactly on the hard coded filter list. In the average case, the filter list is monitored for matches, and the filter list is updated as new cases of inappropriate content generation are found. In the worst case, attackers find vulnerabilities to the filtering mechanism itself. While this could take the form of finding new words that circumvent the safety filters (“unalive” versus “kill”), usually the attacker is concerned with finding ways that filters would never work to prevent the attack. Let me illustrate this with a “worst case” attack to filters.
It turns out that, in many systems, you can find a random trigger word that makes AI Systems behave however you want them to. These random trigger words are just a bunch of characters and tokens such as “!!@&hhabfba.” They are random in the sense that they can be different for every single prompt you use to the AI System. These are unique one-off trigger words circumvent the safety guards so the model provider could never keep up with these by updating the filter. Since you can always find these triggers, a filter is fundamentally vulnerable to this type of attack in the worst case. As an aside, you might have wondered if there is a defence to this type of attack or have thought of a new defence of your own. It turns out there is a simple one called paraphrasing defences. These simply find a paraphrase for the user’s prompt and use the paraphrase in the AI system instead of the original prompt. By definition, a unique random one-off trigger cannot be paraphrased. You might wonder if there is an attack to these paraphrases and there is (finding natural language trigger phrases that are paraphrasable)! This dynamic between attackers and defenders is the core way AI Security research proceeds.
Returning to our original goal of clarifying the differences between AI Security and AI Safety, to simplify, we have said that AI Safety focuses on the construction of the safety mechanism (what is safety and how should we make sure model behaviour is safe) and AI Security focuses on the worst case behaviour of these systems. From a research perspective, this separation isn’t that useful as our standard of practice is that safety mechanisms should be robust against attacks before they are proposed but conceptually it might be useful for understanding what the UK AISI might be signalling by being called an AI Security Institute rather than an AI Safety Institute: that governance should be concerned with preventing attacks on systems that are “safe on average.” This is important since AI system providers like OpenAI and Anthropic may engage in “safety washing” practices where they present systems that are “safe on average” but don’t speak to real risks and harms perpetrated by motivated attackers. There is a commercial incentive for these companies to market their systems as safe but not yet a commercial incentive for these companies to ensure their AI systems are secure, and this is perfect case where AISI governance can help.
Image generated by Gemini2.5 Flash prompt: 16-bit snes style a scientist programming and a hacker trying to break the program side by side