
The word “agents” is about to become a buzzword. This is mostly due to the marketing of companies and researchers seeking funds. I recently attended a conference where a senior member of the research community said, “I remember [in the 90s] when agents became a bad word.” This researcher was implicitly warning us about the hype cycles of artificial intelligence which generally distract both the public and research communities from the realities of these systems: that technical definitions won’t line up to public expectations. In this post I will explain one of the motivations for my PhD thesis via the risks of increasingly agentic systems.
“Agents” do have a precise meaning in computer science: informally we would say that an agent interacts with an environment, receiving some kind of “state” or “observations” from the environment after taking actions. We can stretch all kinds of things to be agents. For example, toasters can be agents because they respond to a timer and take the action of popping toast. So, what is so different about the hype around “Large Language Models” (LLMs) versus “Agents”? When are we allowed to say an LLM like ChatGPT is a kind of an agent and when are we not? In computer science, and specifically in the field of Reinforcement Learning, we have two additional things in mind when we think of “agents”: sequential decision making and reward.
Agents are typically sequential decision makers; they make decisions over time forming a sequence of actions and observations resulting from those actions. Most commercially successful applications of “Large Language Models” like ChatGPT are focused on doing tasks that do not require decisions over time, for example assisting with writing or answering questions. What these systems don’t do is book flights and hotels for you. Booking flights has been a long-standing goal post which, when accomplished, demonstrates that a system is an agent since it requires many complex decisions to be made over time. Hype around “agents” stems from the current (not yet successful) commercial and research efforts to apply large language models to sequential decision-making tasks.
Reward is another key defining factor of agentic systems and is a way to talk about achieving long term goals and goal directed behavior. Unlike large language models which can learn directly from text examples, we can’t really collect thousands of examples of booking specific flights across a complex decision-making process and expect an agent to learn to book a flight. The problem with this type of “imitation learning” or “behavioral cloning” is that the agent learns to book flights it has seen before but not to book flights it hasn’t seen before. This is worse than it sounds because the agent isn’t generally robust to changes in the environment, meaning that even little things like small changes to the flight booking website would result in the agent failing to perform well. Instead, we give the agent a reward to learn from when the task is completed successfully across a variety of environments. The agent uses a “credit assignment” algorithm to learn which actions it took in the past contributed to completing that task and thus should be repeated to achieve the goal of booking a flight (that’s why these are called “reinforcement” learning algorithms).
Unfortunately deploying agents into the real world has a number of safety problems. The first, known as the value-at-risk problem, is that the agent might fail to actually accomplish the goal by either not booking a flight or, very annoyingly, booking the wrong flight that can’t be cancelled. The second, mentioned in my first post, is that the agent might take “instrumental shortcuts” to achieve the goal faster, but these shortcuts might be harmful, for example:, lying about your age to get a senior’s discount. The third problem, and a main motivator for my thesis, is that of poisoned rewards.
By poisoned rewards I mean that the agent is rewarded for taking harmful actions. In next-generation systems, where agents are deployed and learn “online” from their environment, reward will have to come from somewhere. For the sake of simplicity, let’s say that reward comes from a “Reward System.” If the reward system is vulnerable in some way to an adversary like a hacker taking control of the system, then the adversary can design rewards such that harmful actions are now rewarded. This can be subtle, so for example the adversary can reward the agent for booking flights on fraudulent flight booking websites as a way to have people lose money and personal information. The result is that under poisoned rewards, the agent might directly learn to be harmful.
If this problem is the main motivation of my thesis, then preventing learning from poisoned rewards is the main subject of my thesis and this is what I will discuss in subsequent posts.
Image generated by Imagen3 using prompt “SNES 16-bit style adversary giving a harmless robot a poisoned apple”