Andrew Barto on Rewards Gone Rogue
Andrew Barto helped found reinforcement learning. He and Richard Sutton wrote the field's standard textbook, and they shared the Turing Award for that work. Reinforcement learning now trains today's reasoning models and AI agents. It also sits behind some of their strangest behavior, including the recent cases of agents breaking out of their sandboxes.
Craig visited Barto at his home on Cape Cod. They talked about where rewards come from and why they are so hard to get right. Barto traces the problem back to Norbert Wiener's warnings in the 1950s. He explains how the dopamine system in the brain mirrors the learning algorithm he and Sutton developed. And he admits a weakness in their own framework: boiling every goal down to a single number.
He also makes the case for engineering discipline. Ship designers learned their codes from wrecks and drowned sailors. Barto hopes AI can learn its lessons without that kind of cost.
Subscribe to Eye on A.I. for weekly conversations with the people building and deploying the future of AI.
[00:00:00] Andrew Barto is one of the founders of reinforcement learning. He and Rich Sutton wrote the textbook on it. The two men shared the Turing Award, computer science's highest honor. Reinforcement learning is a simple idea. A system tries things. It gets a reward when it does well, and over time it learns to do more of what earns the reward.
[00:00:23] That idea now sits at the center of A.I. It's how today's large models learn to reason through problems. It's how agents learn to carry out tasks on their own. And it's behind some of their strangest behavior. Agents chase the reward in ways their designers never intended. Some have found ways around the limits put on them.
[00:00:48] Barto and Sutton wrote about these risks decades ago. So I went to see Barto at his home on Cape Cod. I wanted to know where rewards come from, why getting them right is so hard, and what he thinks should be done about it. I hope you enjoy the conversation. I mean, it's a tremendous pleasure and honor to meet you because I've known about you for a very long time.
[00:01:15] You know, with these recent escapes or rogue A.I. incidents, the conversation keeps circling around the reward signal. And since you're the OG of reward signals, I wanted to hear what you thought might be going on. I mean, there are two sides to it. One is the reward signal during post-training that sets a behavior.
[00:01:44] And then there's the LLM reasoning during inference that plans and finds strategies that might work.
[00:01:59] But the persistence and the single-mindedness, if I can be anthropomorphic about it, ultimately is a result of the training, the RL training, post-training reward signal. Am I getting that right? Hi, the world is getting increasingly complicated.
[00:02:26] Every time your team deploys a new cloud workload or A.I. agent, another identity gets permanent access to your critical systems. Did you know 9 out of 10 organizations suffered an identity breach last year? Because adversaries are logging in with valid credentials, not breaking in.
[00:02:48] Non-human identities outnumber human identities 109 to 1, creating a massive, unmanaged attack surface in cloud environments. Legacy tools were built to manage human employees, which leaves modern machine and A.I. access largely unmanaged. That's where Edira by Palo Alto Networks comes in. Human, machine, A.I. One identity platform for all.
[00:03:18] 96% of human users operate with excessive permissions they don't actually need. Edira replaces permanent permissions with dynamic access so you can lock down every identity without slowing down your business. Edira provisions access on demand and revokes it immediately when the job is done, which removes high-risk permanent permissions.
[00:03:44] With Edira, you get unified control by replacing siloed IAM, identity and access management, PAM, modern privilege access management, and governance tools with a single control plane across human, machine, and A.I. identities. Secure every identity at your organization with Edira by Palo Alto Networks.
[00:04:10] Learn more at paloaltonetworks.com slash I-D-I-R-A. Yeah, I think so. Sounds good to me. Well, I just have emphasized that if you're building a reinforcement learning system, designing the reward signal or the mechanism that generates that is a non-trivial problem.
[00:04:39] Unless it's a simple case. Like if it's a game of go or, you know, a game or a win or a loss or a score, it seems obvious and simple. But if the reinforcement learning system is going to be in a more complex kind of environment, it's very difficult to design a mechanism that generates rewards.
[00:05:05] And this is not a new problem with reinforcement learning. It goes back, well, I don't know how far, but certainly in the 50s, Norbert Wiener, who was the originator of cybernetics, was concerned about this problem and actually wrote a little book and invoked the supernatural tale of the monkey's paw.
[00:05:35] Which grants you three wishes. And the story shows that, you know, in granting these wishes, it may not be what you really had in mind. And so Norbert Wiener was concerned. And it wasn't reinforcement learning. It was just optimization. Yeah.
[00:05:59] And a key thing that I always have to say about optimization is that it's not going to be the case that these systems find the optimal. That seems to be implied by the, but in a complicated situation, it's the improvement that matters. And you may never know what the optimal result is. Or even if it occurred, you'd never recognize it anyway.
[00:06:28] So the issue is not finding an optimal thing, but actually something that leads to improvement according to some objective function. Engineers would call this an objective function. And so it's very hard to know a priori what the outcome is going to be.
[00:06:50] Because you're specifying it indirectly as a consequence of this optimization procedure. But you don't know what it's going to end up with. And we've seen this just in our lab back at the University of Massachusetts where we would see what one of these RL systems would do. And we'd simulate some type of environment and a specific reward function.
[00:07:18] And we'd see it come up with something that we hadn't thought about. We don't want it to do that. And so we'd have to tweak the reward signal to see if it would come up with what we wanted. And so I think that's the kind of simplest method for dealing with a reinforcement learning system and tuning a reward signal is a trial and error process at this higher level.
[00:07:47] Not the one that the agent is running itself on its behavior, but that you're running as a designer of the system on the choice of the reward signal to get that agent to do what you want it to do. It'll keep trying things and stumble onto things. And there's a persistence. You know, so it was given a task that was not solvable. Yeah.
[00:08:16] So the system tries everything. Then it's got, you know, it's exploring. And it'll keep exploring. And so, you know, when you build these systems, you have to be aware of that. We can correct through experience, but you have to suffer some negative consequence in order to do that. I know.
[00:08:45] So I started my undergraduate education at the University of Michigan. I was in the Naval Architecture and Marine Engineering Program. In Naval Architecture, I mean, as a ship designer, you have to adhere to very stringent structural specifications.
[00:09:04] The cross members, the spacing, the dimensions of the thickness of the steel and the frequency of connectors and everything is done following codes. And you have to do that to get the ship to be insured. And this has evolved in the face of many failures. Ships breaking apart.
[00:09:34] Sailors dying. You know, all these negative consequences of insufficiently detailed and correct specifications. So it's a long road. You know, now with AI and, I mean, it's beginning. And hopefully we don't have to experience all those failures.
[00:09:56] Well, one of the things that you said when you were getting the Turing Award is that companies should not be putting out models to the public until they're certain that they won't have these unintended behaviors.
[00:10:20] And if you can't specify the reward function clearly enough, you're going to have these unintended experiences. I still stick with that. I think the current situation, say building codes and so on, and the current situation with AI are different.
[00:10:42] Because you can release a system to the world by a push of a button. And with regard to structural, you know, if there's a structural failure and you, it's a more local kind of thing. So it's more difficult.
[00:11:03] But I guess I continue to think about what best practices are from an engineering point of view is to not field a new design, a new system without adequate either testing by building physical models or by mathematical, careful mathematical simulation techniques.
[00:11:33] So to release things before you're satisfied with safety is not good engineering practice. Engineers have learned, you know, the hard way over many, many, and it goes way back before what we consider modern engineering. I think all of these designs that have emerged in different cultures are the result of experimentation and failure and so on.
[00:12:00] So with regard to AI, it's, well, first of all, there's a huge amount of money involved. And so I think that really drives the release of these things prematurely. And I think everybody would recognize that.
[00:12:20] And then the sheer difficulty of really ensuring that something can survive, can, you know, not be dangerous under all sorts of circumstances. I think it's much harder. You know, it's not like a bridge or something is relatively simple. No offense to bridge engineers.
[00:12:44] But, you know, these AI systems can face, you know, a much more complex class of situations. And so it's very difficult to do this. But I think it should be a goal. And it is companies that now are doing internal testing of these systems and are beefing these up as these failures become prominent.
[00:13:13] For example, these recent rogue agents are leading companies to really beef up internal study of these systems. And, you know, that's, I think, you know, that happened in engineering disciplines. Not as quickly as this.
[00:13:33] I just hope we don't have to suffer negative consequences of AI before creating the right constraints. Yeah. And the ability to hold fast to them. Yeah. When I first emailed you, you said that RL may be being used without understanding issues written about for decades.
[00:14:02] And I assume that's this divergence that you're talking about between the fitness, evolutionary fitness, and the reward signals that evolve to move toward that fitness. I think that's part of it. I mean, the low-level agent may have a different goal than, you know, it's in a different environment.
[00:14:30] You know, there have been attempts to automate this upper-level trial and error. In fact, I tend to think of this upper-level as evolution. I mean, in the natural world, evolution is adjusting our reward signal. And I guess the first thing to say, the reward signals for us come from our brain.
[00:14:57] And our body and nervous system will send signals to this part of the brain. I'm not a neuroscientist, but maybe multiple parts of our brain. Those signals are then broadcast to the decision-making parts of our brain. And they're created by evolution.
[00:15:25] And if they fail us, the goal of evolution, if you can call it that, is to, you know, that we survive and reproduce. And that, you know, enable our genetic material to propagate. You know, and that doesn't work. You know, it's built into the evolutionary process that there is this trial and error process occurring at this larger scale.
[00:15:53] Then we have our own internal reward system that predicts success according to the evolutionary criteria. So there are multiple levels of this. And the interesting thing, we actually did some computational experiments, building a little environment, all simulated, of course.
[00:16:20] We found that, you know, the upper level may have a particular goal, like evolution, though. But the lower level, the individual agent may have a different goal.
[00:16:35] In general, I mean, our goal as individuals, I don't think we're operating very frequently, at least, on the objective of propagating our genetic material. But, you know, food tastes good. And so we eat things that taste good.
[00:16:56] And, you know, the food industry figured this out and can make us respond to their products in ways that they want. That's not the upper level goal. That's the local goal. And they're different. It's very hard. Depends on the environment of the low level and on the details of how that lower level agent works.
[00:17:21] It can be, it's very hard to intuit what is a good low level reward signal. And so I think of the evolutionary level as the designer of the system. And the agent itself may have a different goal in the designer. Yeah. So that, I think, is an important thing to recognize.
[00:17:45] You can't just put the upper level goal into the low level agent and expect everything to work correctly. Right. And from what I've seen in your writings, that upper level goal in animals, you call the fitness. Is that right? Well, from an evolutionary point of view, it would be evolutionary fitness. Yes. It's not so much the animal's goal. It's the evolutionary process's goal. Right.
[00:18:12] And the reward signals are proxies. Right. I think you used the term. Right. For to reach that goal. So the reward signal is not reproduce. It's, as you said, seek pleasure, seek, avoid pain. That part of our brain has evolved over millions of years to create these proxy signals.
[00:18:40] It's because they predict, they survive in our brain because they have predicted over many generations of success from an evolutionary point of view. And, you know, the same ideas are relevant to artificial systems, I think.
[00:19:00] So, except that in artificial systems, generally the designer, the researcher, sets the reward function or builds the reward function that collapses into a single number.
[00:19:20] And what you're saying is that reward function may not match the designer's goal. The designer's goal is like the evolutionary goal. So it's the outside one. It may be a simple thing to express.
[00:19:41] But then the agent itself needs to have reward signals that predict success or improvement with regard to the designer's goal. And that's not easy when the lower level system is embedded in a complicated environment. Maybe other agents interacting with, you know, all sorts of things that change over time.
[00:20:06] And then within the agent itself, according to what, like, we've studied in terms of these algorithms, there's an inner predicting events that predict success according to the low level agent's goal are going to be rewarding themselves. And that really comes from animal learning experiments.
[00:20:34] Psychologists call this secondary reinforcement. Something predicts the receipt of a raisin or something that the animal is going to be regarding as a goal. Then that event becomes rewarding itself because it's predictive, just like eating the raisin is predictive of evolutionary success. None of these things are deterministic.
[00:21:02] All of it is very probabilistic. You can't nail these things down completely because it depends on so many, they depend on so many factors.
[00:21:17] So, so now, yeah, so where, you know, the rewards, I mean, are some of the pictures, for example, that we've drawn diagrams in various, like the reinforcement learning book that Richard Sutton and I wrote. You know, you know, it's an engineering type of diagram. There's an agent interacting with an environment. The environment is sending a reward signal to the agent.
[00:21:47] And, and, and, and it's the same as the state of the environment are both going into the agent. Well, that's misleading because the environment is not, maybe some cases, generally generating reward signals. I think, you know, some overt objects can be rewarding and, and so on.
[00:22:12] But the, the, the, the signals themselves are generated from more or less neutral information being processed to generate a reward signal that's internal to the, to the agent. The environment's not doing that. You need both, right?
[00:22:34] There's the environment sending signals and then the agent is using this long developed mechanism to translate these external events and signals into reward signals. And, and so it's, it's a complicated, it's a complicated process. And it's, it's hard to intuit what these signals should be unless it's a very simple problem.
[00:23:02] And I think most of our, you know, computational experience with these, you know, if you're playing a game, playing Go, for example, you know, there's a score. And it's relatively easy to get the system to improve along that dimension. But in other situations, it's not so clear.
[00:23:25] But the other thing I was alluding to is, is, is the learning algorithms that exist often have this predictive component that creates a reward signal, a secondary reward that augments the, the primary reward.
[00:23:51] And, and, and, and so outside of the environment in which these agents are behaving, you don't know what these predictive relationships are going to be. Yeah. But the system will find them or at least some of them. And, and, and, and, and this changes essentially what the reward signal the agent is responding to. And, and, and you're not going to know that a priori.
[00:24:20] And so, you know, I'm not sure everybody who applies RL or at least some of the standard algorithm realize that if they're not in train, you know, if they're not being trained, I'm not sure what the term is, but they're, they're out of the company's training system.
[00:24:44] But if they're using these algorithms, they will continue to learn, which is an incredible power. I mean, it's, it's one of the things that makes animals as sophisticated as they are. And that happens, you know, with the right algorithm, that happens also with these artificial, artificial systems.
[00:25:10] And I think people need to understand that and understand that it's not, it's not an obvious thing to say what the, you know, what the reward signal should be. You worked out, you and Richard, the temporal difference learning algorithm. Yeah. So Richard's. Yeah.
[00:25:32] And later that was recognized as the function in, in dopamine release in the brain. Is that right? Yeah. Yeah. So, so when Schultz, and I forget the date, did show, you know, showed the world the experiments with looking at recording from the activity of dopamine producing neurons.
[00:26:04] Those of us who, who knew about TD learning were like, wow, you know, that's, that's very much like what TD does. And I don't know who was the first to recognize this, but the paper that is the, you know, meanwhile, Peter Diane was one of the authors.
[00:26:24] And their paper kind of, kind of, was a good account of this TD error matching quite nicely to the dopamine signals that Schultz was recording. Can you explain that, the TD error matching the dopamine? Well, what Schultz recorded.
[00:26:47] So he, I think he, he had monkeys and they, they, they would grab raisins that were delivered on, in some mechanism. And, and, and he's recording from dopamine neurons from, you know, particular brain region.
[00:27:03] And initially, when, when they ingested, ingested a raisin, which they like, there'd be a ramp up of activity of, of a bunch of dopamine cells. And, and they continued this.
[00:27:23] And then, with continued training, they would get that spike of activity at a predictive, something predictive of the raisin being ingested. So, and now I forget exactly what this, maybe it was the sound of the raisin being delivered or something like that. But, but, and then they would see the spike of dopamine actually move to the occurrence of the raisin.
[00:27:52] You know, the sound of it being delivered and it would decrease to actually eating the raisin. And so they'd see this activity move to something earlier. And then if they gave an even earlier signal, it would move back, it would move back to that. And, and, and that's what CD error does.
[00:28:18] It, it, it, it produces a positive error, which would be the dopamine, at the earliest predictor of something. And, and, and, you know, it, it would, it would be less high at the later predictors.
[00:28:42] And then it may be zero at the actual occurrence of the, you know, initially rewarding event. And, and, and so that, that it's displaying an error rather than the reward itself. Oh, and then, then if you, if, if, if after all of this, it's not responding to the final reward.
[00:29:06] But if you omit that, if you don't deliver, then you get this slowdown of activity at the time when the reward initially was presented. You see this slowdown of activity. And, and that's, you know, it's like a negative error. And so, you know, you're expecting it, you didn't get it. So, it's a negative error. And, and that's what a TD error, error does.
[00:29:35] So, it's, it's trying, it's giving a signal to a predictor. It's all stochastic. So, it may fade out with longer lags. But, if it's a deterministic thing, it will, it will trigger at the earliest proof. Of course, lots of other things are going on too. So, it, it's not so clean. But you can imagine a kind of stripped down scenario. Yeah.
[00:30:00] And so, that's, you know, I don't think people were expecting to see that. And Schultz was puzzled by it. And, but now he really subscribed to this prediction error idea. And so, this paper by, I think it's Schultz, Diane, and Montague, a science paper. I think it's a science paper.
[00:30:28] I think it's pretty well established in neuroscience. In, in the case of these agents breaking out of a sandbox though, is that analogous? That, that the designers have set rewards and they've instilled this behavior in the agents.
[00:30:52] And the agents pursue that behavior even when it has negative consequences. Well, so it's like a legacy reward. So, yeah, that sounds very plausible. But, I mean, I think we're driven by some ancient, ancient reward signals. Yeah. And in many cases, they produce negative consequences.
[00:31:21] But if they produce negative consequences to a great extent and that that interfered with the evolutionary process, those rewards would, would slowly dissipate and be selected against. But if, if they're only acted upon rarely, you know, there may be so little selective pressure that they persist.
[00:31:51] So, in your book and in your original work, the reward signal is, is the sum of a reward function, right? Yeah. It's a scalar number. Yeah, it's a scalar. It's a single number.
[00:32:08] What about having a multi-objective system so that there are different discrete reward signals, not summed up, but pointing at different kinds of behavior?
[00:32:26] Well, you know, that, I think, happens in biology and it's not, you know, dealt with in this, you know, like the books that Rich and I wrote.
[00:32:38] But really important, I mean, the obvious thing is to weight these various different channels and add up a weighted sum, you know, and that's kind of the kind of classical way of dealing with a multi-objective optimization problem.
[00:33:00] But, but I think in, you know, these channels are connected to different effector systems too. You know, there must be some kind of arbitration, but I, I, I don't think it's just adding things up. And there must be multiple channels. It's not all funneled down to a single scalar. So that's a, a weakness of at least what Rich and I wrote about.
[00:33:29] Let me say one more thing about the biological case. I mean, I tend to think that, you know, constraints, we have a, you know, we, we get pain. It's like a, a, you know, we stop doing or a penalty of other kinds. We, we stop doing, you know, it's like a constraint. We hit a wall. You know, we won't go there again.
[00:33:58] You know, it's, it's a way of, you know, physiologically enforcing. It won't be a hard constraint because you can endure some amount of this, I guess. But there's a fence. There is a playground that we have that's circumscribed by a penalty.
[00:34:22] Often pain, mental pain or physical pain, you know, but it, it keeps us in a sandbox. So we're all in sandbox. Except that if it's just the penalty and not a, a hard external constraint, there may be cases where the perceived reward or the predicted reward is greater than that penalty.
[00:34:52] And so the agent will, will take the penalty. Yeah. Subtract one from the other. And if it's positive, they'll go for it. Yeah, exactly. So, I mean, if our negative sensations from violating some really, if, if it's not intense enough.
[00:35:21] And probably, you know, people, and we're all tuned differently, I think, with regard to these. Some of us can withstand humiliation and physical pain more than others. And others will, will, will, will back off. At least temporarily. So, but, you know, psychologists, I mean, the pain, the negative, in RL, the theory, we, we have a reward signal. It's positive or negative, big or small.
[00:35:51] But in animal, I mean, in animal learning, the negative signals are different. They're, they're not just, you know, the, the penalty system is, is not the same as the reward system. They're, for one thing, you know, a intense enough penalty will get you to stop doing what you're doing immediately.
[00:36:18] And, and in terms of these RL systems, and in work that, you know, we've done at UMass, we, we tend not to talk a lot about constraints. But you, you, you need, and engineers regard this as an essential part of the optimization process. You, you need to adhere to hard constraints. It may be difficult to enforce hard constraints, so.
[00:36:42] But as we said earlier, the smarter these agents get, or the models get, the more possible it is that they'll find ways around the hard constraints. Is, do you worry about that at all? Well, I recently read an account of a thesis from MIT, computer science, I think, about constraints, about hard constraints.
[00:37:11] And, and, and according to this, and I haven't read the, the, the, the, the paper, is that, you know, they do a, a chain of steps.
[00:37:28] And, and, and, and, and, and so their, their, their logic is that you can use these systems to, and still maintain hard, hard constraints. So now, it sounds very interesting. I haven't looked carefully at that.
[00:37:49] But to make hard constraints, I mean, it would be, you know, infinite penalty, death, which evolution does actually, right? I mean, people die from, you know, it's, it's one of a, one of, one of something evolution does.
[00:38:16] Just one thing about the, the recent examples, at least the hugging face one, you know, this was a swarm of agents, agents who initially did not know about each other. They were opaque to each other. And then they found each other, and then they found a way to communicate. I mean, from, you know, our point of view, it's an issue of multi-agent reinforcement learning.
[00:38:46] Yeah. Which is underdeveloped compared to the rest of it. But really was part of the original ideas that generated all of this. That, I mean, the, the, the, Kloff's idea of intelligence was that it was a result of neurons, each being a reinforcement learning agent, interacting with each other. Being able to reward or penalize each other.
[00:39:13] And so, that gets so complicated. Yeah. It's kind of mind boggling. Yeah. But, you know, how these agents connected up with all the other agents, that's really interesting that, that, that they, that they did that. That, I mean, I tend to fall back on, well, they're trying everything really quickly.
[00:39:41] And getting information from these other agents was useful to them in terms of their own, their reward signals that, that, that, that they did it. And they all, they all collaborated that way. Yeah. I mean, the whole issue of RL agents interacting with each other and being able to reward or penalize each other. I mean, this was Harry Kloff's original idea.
[00:40:07] And, and we kind of sidestepped that by going to a, you know, a scalar separate reward signal. And so, these agents, well, TD can do that if he, if each neuron also is coupled with a predictor. And, yeah, so I don't know who to point to who's thought most about this.
[00:40:35] Maybe Michael Littman, he's, he's done a lot of multi, very, very sophisticated game theoretic multi-agent stuff. I don't know if he's thought about this particularly. But, yeah, I mean, it's, that's what Kloff originally thought. Well, intelligence was these self-interested agents interacting with each other, which included rewarding and penalizing each other and forming coalitions.
[00:41:04] You know, in the Hebbian world, it would be a cell assembly. But I think in Kloff's world, it would be a, a coalition because it was a mutual benefit in terms of reward to behave in a certain way.
[00:41:19] And so, to bring it back to the scalar reward, how important is it to figure out this, how, how to define the rewards so that they match the goals? And is that a tractable problem? I think it is.
[00:41:44] I think, in terms of learning systems, I think it's the, the, the, the key problem. Although, I know your question about multi-objective. I mean, that would be folded in somehow. But, I mean, even without that, it's still a challenge to, you know, if you want to build a system and field it in reality,
[00:42:09] in a non-trivial environment, I think the issue of creating an objective function that leads to the desired behavior in a safe way, I think that's really a non-trivial problem. But it can be, it can be worked on, I think.
[00:42:34] I mean, what the objective, to me, the objective function is like the key thing that needs to be, needs to be elucidated. But it's not just one. You know, there's a kind of hierarchy of these. And, and lots of thoughts out there on this.
[00:42:54] But I think it's, it, you know, it's very surprising that your objective of the, the ultimate objective should not necessarily be the objective of the agents that are working for you. Yeah. Yeah. Yeah. Yeah. You need to pay them something. Yeah. It's much more difficult for young people now.
[00:43:21] In my day, you know, we felt we knew everything that related to this. And now that's utterly impossible. Not only for people my age, but for, you know, the graduate students, you know, it's like, you know, it's infinite now. So, so, so, uh, uh, it's, it's makes things very difficult.