
I study Artificial Intelligence (AI) Safety so this is, of course, the first thing people ask me: “Should we be scared of artificial intelligence?” While the short answer is “yes,” I want to convince you that we should really be asking: “should we be scared of humans using AI”.
Current risks
I encourage the reader to review some of the incidents on the AI Incident Database, incidentdatabase.ai, which contains real harms from AI systems of current systems. Some harms of current systems listed in that database are: (1) generate deep fakes: fake images, video, speech, and text (2) perform cyberattacks including automated vulnerability finding and exploitation and (3) engage in discrimination due to algorithmic biases (4) cause fatal incidents involving autonomous systems due to system vulnerabilities. Generally, there are accidental harms and misuse harms.
For criminal misuse, these aren’t new activities; AI simply makes crime much easier and more cost effective. For accidental harms, AI systems’ decision making on the safety of critical tasks is no less prone to errors and biases than humans are, and systems might even make those biases and errors worse. That isn’t something likely to change in the future.
I would argue that both criminal and accidental harms are an outcome of misusing the technology. Nuclear capabilities are a familiar example of a concept used in AI Safety called “dual-use risk.” Despite the cheap power generation capabilities of Nuclear Power, what makes nuclear capabilities scary is what a bad actor might choose to do with the technology. Emerging misuses of AI like deep fakes for blackmail are made possible because it’s increasingly easy to generate realistic images.
Accidents and bias in AI are a little more subtle than criminal misuse, while they may seem to happen “unintentionally,” there is intentionality and responsibility here. For technologies like self-driving vehicles and CV screening tools, people choose to use them, companies choose to market them, and governments choose to allow them. As an AI researcher it is my responsibility to make it clear that most AI technology is vulnerable to errors and therefore the choice to use AI in high-risk settings may be a choice to introduce harm.
Next generation systems
There is a lot more that could be said about current systems, but part of what AI safety tries to do is anticipate future risks. In addition to mitigating current misuse risks, I work on preventing risks related to autonomous decision-making systems that do not exist . . . yet. At least not as commercial technologies. These so-called “agentic” systems, which use a paradigm called Reinforcement Learning, were made famous when DeepMind’s AlphaGo defeated Lee Sedol in the game Go. This type of AI has been in the news lately as it was one of the pieces that made the ChatGPT competitor known as “DeepSeek” so much more effective.
This technology suffers from several unique failure modes that make it much riskier to deploy and use. I will only focus on one since it will illustrate the point. Agentic systems achieve a high-level goal that requires many subtasks to be completed and achieve that goal. “Instrumental convergence” is a property of these systems that might take actions that help achieve the goal more effectively but are harmful. An example is that if you ask your agent to purchase something online for as cheap as possible, a pair of shoes say. If the shoes can be purchased with a senior discount the agent may lie to save you money and state that you are a senior even though you are not.
Other risks of agentic systems are due to how hard it is to specify a goal to a system designed to maximize achieving that goal. Despite these unique risks, agentic systems are only risky if we give our decision-making power over to them.
Regardless of what these risks are and whether they are current systems or next generation systems, it is humans who will decide to use these systems, companies that chose to market them, and governments that chose (or chose not to) regulate them. That said, there are many technical solutions that can help make these technologies much less risky, including preventing dual use and instrumental harms. These are the focus areas of my research that I am excited to share in future posts.
Picture generated by Dall-E on February 27th, 2025, using the following prompt: “16-bit SNES style image of evil villain behind robot.”