See all Lex Fridman transcripts on Youtube

Speaker 1

00:00

A lot of people were saying like, oh, this whole idea of game theory, it's just nonsense. And if you really want to make money, you got to like look into the other person's eyes and read their soul and figure out what cards they have. But what happened was where we played our bot against 4 top heads up, no limit, hold them poker players. And the bot wasn't trying to adapt to them.

Speaker 1

00:19

It wasn't trying to exploit them. It wasn't trying to do these mind games. It was just trying to approximate the Nash equilibrium and it crushed them.

Speaker 2

00:28

The following is a conversation with Noah Brown, research scientist at FAIR, Facebook AI research group at Meta AI. He co-created the first AI system that achieved superhuman level performance in no limit Texas Hold'em, both heads up and multiplayer. And now recently, he co-created an AI system that can strategically out-negotiate humans using natural language in a popular board game called Diplomacy, which is a war game that emphasizes negotiation.

Speaker 2

00:59

This is the Lex Friedman Podcast. To support it, please check out our sponsors in the description. And now, dear friends, here's Noam Brown. You've been a lead on 3 amazing AI projects.

Speaker 2

01:12

So we got Libratus that solved, or at least achieved human level performance on No Limit Texas Hold'em poker with 2 players, heads up. You got Pluribus that solved No Limit Texas Hold'em poker with 6 players. And just now you have Cicero, these are all names of systems, that solved or achieved human level performance on the game of diplomacy, which for people who don't know, is a popular strategy board game. It was loved by JFK, John F.

Speaker 2

01:44

Kennedy, and Henry Kissinger, and many other big famous people in the decades since. So let's talk about poker and diplomacy today. First, poker. What is the game of No Limit Texas Hold'em?

Speaker 2

01:59

And how is it different from chess?

Speaker 1

02:00

Well, no limit Texas Hold'em poker is the most popular variant of poker in the world. So, you go to a casino, you sit down at the poker table, the game that you're playing is no limit Texas Hold'em. If you watch movies about poker, like Casino Royale or Rounders, the game that they're playing is no limit Texas hold'em poker.

Speaker 1

02:17

Now it's very different from limit hold'em in that you can bet any amount of chips that you want and so the stakes escalate really quickly. You start out with like 1 or 2 dollars in the pot and then by the end of the hand, you've got like a thousand dollars in there maybe.

Speaker 2

02:32

So the option to increase the number very aggressively and very quickly is always there.

Speaker 1

02:36

Right, the no limit aspect is there's no limits to how much you can bet. In limit hold'em, there's like $2 in the pot, you can only bet like $2. But if you got $10,000 in front of you, you're always welcome to put $10,000 into the pot.

Speaker 2

02:50

So I've got a chance to hang out with Phil Helmuth who plays all these different variants of poker and correct me if I'm wrong, but it seems like no limit rewards crazy versus the other ones rewards more kind of calculated strategy or no, because you're sort of looking from an analytic perspective, is strategy also rewarded in no limit Texas hold'em?

Speaker 1

03:15

I think both variants reward strategy, but I think what's different about no limit hold'em is it's much easier to get jumpy. You go in there thinking you're going to play for like $100 or something and suddenly there's like $1,000 in the pot. A lot of people can't handle that.

Speaker 2

03:32

Can you define jumpy?

Speaker 1

03:33

When you're playing poker, you always want to choose the action that's going to maximize your expected value. It's kind of like with investing, right? If you're ever in a situation where the amount of money that's at stake is going to have a material impact on your life, then you're going to play in a more risk-averse style.

Speaker 1

03:51

If somebody makes a huge bet, you're going to, if you're playing no limit hold'em and somebody makes a huge bet, there might come a point where you're like, this is too much money for me to handle. Like I can't risk this amount. And that's what throws a lot of people off. So that's the big difference, I think, between no limit and limit.

Speaker 2

04:09

What about on the action side, when you're actually making that big bet? That's what I mean by crazy. I was trying to refer to the technical term of crazy, meaning use the big jump in the bet to completely throw off the other person in terms of their ability to reason optimally.

Speaker 1

04:30

I think that's right. I think 1 of the key strategies in poker is to put the other person into an uncomfortable position. And if you're doing that, then you're playing poker well.

Speaker 1

04:40

And there's a lot of opportunities to do that in no limit hold'em. You can have like $50 in there, you throw in a $1,000 bet. And that's sometimes, if you do it right, it puts the other person in a really tough spot. Now, it's also possible that you make huge mistakes that way.

Speaker 1

04:56

And so it's really easy to lose a lot of money in no limit hold'em if you don't know what you're doing. But there's a lot of upside potential too.

Speaker 2

05:02

So when you build systems, AI systems that play these games, we'll talk about poker, we'll talk about diplomacy, are you drawn in in part by the beauty of the game itself, AI aside, or is it to you primarily a fascinating problem set for the AI to solve?

Speaker 1

05:19

I'm drawn in by the beauty of the game. When I started playing poker when I was in high school and the idea to me that there is a correct, an objectively correct way of playing poker. And if you could figure out what that is, then you're making unlimited money basically.

Speaker 1

05:38

That's like a really fascinating concept to me. And so I was fascinated by the strategy of poker, even when I was like 16 years old. It wasn't until much later that I actually worked on poker AIs.

Speaker 2

05:48

So there was a sense that you can solve poker, like in the way you can solve chess, for example, or checkers. I believe checkers got solved, right?

Speaker 1

05:57

Yeah, checkers got completely solved.

Speaker 2

05:59

Optimal strategy.

Speaker 1

06:00

Optimal strategy. It's impossible to beat the AI.

Speaker 2

06:01

Yeah, and so in that same way, you could technically solve chess.

Speaker 1

06:05

You could solve chess, you could solve poker.

Speaker 2

06:07

You could solve poker.

Speaker 1

06:09

So this gets into the concept of a Nash equilibrium. So it is a Nash equilibrium. Okay, So in any finite two-player zero-sum game, there is an optimal strategy that if you play it, you are guaranteed to not lose an expectation no matter what your opponent does.

Speaker 1

06:26

And this is kind of a radical concept to a lot of people, but it's true in chess. It's true in poker. It's true in any finite two-player zero-sum game. And to give some intuition for this, you can think of rock, paper, scissors.

Speaker 1

06:38

In rock, paper, scissors, if you randomly choose between throwing rock, paper, and scissors with equal probability, then no matter what your opponent does, you are not going to lose an expectation. You're not going to lose an expectation in the long run. Now, the same is true for poker. There exists some strategy, some really complicated strategy, that if you play that, you are guaranteed to not lose money in the long run.

Speaker 1

07:00

And I should say, this is for 2 player poker. 6 player poker is a different story.

Speaker 2

07:03

Yeah, it's a beautiful giant mess. When you say in expectation, you're guaranteed not to lose in expectation. What does in expectation mean?

Speaker 1

07:12

Poker is a very high variance game. So you're gonna have hands where you win, you're gonna have hands with your lose. Even if you're playing the perfect strategy, you can't guarantee that you're gonna win every single hand.

Speaker 1

07:20

But if you play for long enough, then you are guaranteed to at least break even and in practice probably win.

Speaker 2

07:27

So that's an expectation, the size of your stack, generally speaking. Now that doesn't include anything about the fact that you can go broke. It doesn't include any of those kinds of normal real world limitations.

Speaker 2

07:39

You're talking in a theoretical world. What about the 0 sum aspect? How big of a constraint is that? How big of a constraint

Speaker 1

07:46

is finite? So finite's not a huge constraint. So I mean, most games that you play are finite in size.

Speaker 1

07:54

It's also true, actually, that there exists this perfect strategy in many infinite games as well. Technically, the game has to be compact. There are like some edge cases where you don't have a Nash equilibrium in a two-player zero-sum game. So you can think of a game where you're like, you know, if we're playing a game where whoever names the bigger number is the winner, there's no Nash equilibrium to that game.

Speaker 1

08:13

17. 18.

Speaker 2

08:16

You win again. You're good at this.

Speaker 1

08:18

I played a lot of games.

Speaker 2

08:21

Okay, so that's, and then the 0 sum aspect.

Speaker 1

08:24

The 0 sum. 0 sum aspect. So there exists a Nash equilibrium in non 2 player 0 sum games as well.

Speaker 1

08:30

And by the way, just to clarify what I mean by 2 players, 0 sum, I mean there's 2 players, and whatever 1 player wins, the other player loses. So if we're playing poker and I win $50, that means that you're losing $50. Now, outside of two-player zero-sum games, there still exists Nash equilibria, but they're not as meaningful. Because you can think of a game like Risk.

Speaker 1

08:51

If everybody else on the board decides to team up against you and take you out, there's no perfect strategy you can play that's going to guarantee that you win there. There's just nothing you can do. So outside of two-player zero-sum games, there's no guarantee that you're going to win by playing a Nash equilibrium.

Speaker 2

09:07

Have you ever tried to model in the other aspects of the game, which is like the pleasure you draw from playing the game? And then if you're a professional poker player, if you're exciting, even if you lose, you know, the money you would get from the attention you get to the sponsors and all that kind of stuff, is that, that'd be a fun thing to model in. Or is that make it sort of super complex to include the human factor in this full complexity?

Speaker 1

09:36

I think you bring up a couple good points there. So I think a lot of professional poker players, I mean, they get a huge amount of money, not from actually playing poker, but from the sponsorships and having a personality that people want to tune in and watch. That's a big way to make a name for yourself in poker.

Speaker 2

09:53

I just wonder from an AI perspective, if you create, and we'll talk about this more, maybe AI system that also talks trash and all that kind of stuff, that that becomes part of the function to maximize. So it's not just optimal poker play. Maybe sometimes you want to be chaotic.

Speaker 2

10:10

Maybe sometimes you want to be suboptimal and you lose the chaos. And maybe sometimes you want to be overly aggressive because the audience loves that. That'd be fascinating.

Speaker 1

10:22

I think what you're getting at here is that there's a difference between making an AI that wins a game and an AI that's fun to play

Speaker 2

10:27

with.

Speaker 1

10:28

Yeah. Yeah.

Speaker 2

10:29

Or fun to watch. So those are all different things. Fun to play with and fun to watch.

Speaker 1

10:33

Yeah, and I think I've heard talks from game designers. And they say people that work on AI for actual recreational games that people play. And they say, yeah, there's a big difference between trying to make an AI that actually wins.

Speaker 1

10:46

And you look at a game like Civilization, the way that the AIs play is not optimal for trying to win. They're playing a different game. They're trying to have personalities. They're trying to be fun and engaging and that makes for a better game.

Speaker 2

11:00

Yeah. And we also talk about NPCs. I just talked to Todd Howard, who is the creator of Fallout and the Elder Scrolls series and Starfield, the new game coming out. And the creator of what I think is the greatest game of all time, which is Skyrim and the NPCs there.

Speaker 2

11:15

The AI that governs that whole game is very interesting, but the NPCs also are super interesting. And considering what language models might do to NPCs in an open world RPG role-playing game, it's super exciting.

Speaker 1

11:30

Yeah, honestly, I think this is 1 of the first applications where we're going to see real consumer interaction with large language models. I guess Elder Scrolls 6 is in development now. They're probably pretty close to finishing it.

Speaker 1

11:44

But I would not be surprised at all if Elder Scrolls VII was using large language models for their NPCs.

Speaker 2

11:49

No, they're not. I mean, I'm not saying anything. I'm not saying anything.

Speaker 2

11:53

Okay, this

Speaker 1

11:54

is me speculating, not you.

Speaker 2

11:55

No, but they're just releasing the Starfield game. They do 1 game at a time. And so whatever it is, whenever the date is, I don't know what the date is, calm down.

Speaker 2

12:06

But it would be, I don't know, like 2024, 25, 26. So it's actually very possible that would include language models.

Speaker 1

12:14

I was listening to this talk by a gaming executive when I was in grad school, and 1 of the questions that a person in the audience asked is, why are all these games so focused on fighting and killing? And the person responded that it's just so much harder to make an AI that can talk with you and cooperate with you than it is to make an AI that can fight you. And I think once this technology develops further and you can have a, you can reach a point where like not every single line of dialogue has to be scripted, it unlocks a lot of potential for new kinds of games, like much more like positive interactions that are not so focused on fighting.

Speaker 1

12:50

And I'm really looking forward to that.

Speaker 2

12:51

It might not be positive. It might be just drama. So you'll be in like a call of duty game.

Speaker 2

12:56

Instead of doing the shooting, you'll just be hanging out and like arguing with an AI about like passive aggressive, and then you won't be able to sleep that night, you have to return and continue the argument that you were emotionally hurt. I mean, yeah, I think that's actually an exciting world. Whatever is the drama, the chaos that we love, the push and pull of human connection, I think it's possible to do that in the video game world. And I think you could be messier and make more mistakes in the video game world, which is why it would be a nice place.

Speaker 2

13:28

And also it doesn't have as deep of a real psychological impact because inside video games it's kind of understood that you're in a not a real world. So whatever crazy stuff AI does, we have some flexibility to play. Just like with a game of diplomacy, it's a game. This is not real geopolitics, not real war.

Speaker 2

13:49

It's a game. So you can have a little bit of fun, a little bit of chaos. Okay, back to Nash equilibrium. How do we find the Nash equilibrium?

Speaker 1

13:59

All right, so there's different ways to find a Nash equilibrium. So the way that we do it is with this process called self-play. Basically we have this algorithm that starts by playing totally randomly and it learns how to play the game by playing against itself.

Speaker 1

14:15

So it will start playing the game totally randomly, and then if it's playing poker, it'll eventually get to the end of the game and make $50. And then it will review all the decisions that it made along the way and say, what would have happened if I had chosen this other action instead? You know, if I had raised here instead of called, what would the other player have done? And because it's playing against a copy of itself, it's able to do that counterfactual reasoning.

Speaker 1

14:42

So I can say, okay, well, if I took this action and the other person takes this action, and then I take this action. And eventually, I make $150 instead of $50. And so it updates the regret value for that action. Regret is basically like, how much does it regret having not played that action in the past?

Speaker 1

15:00

And when it encounters that same situation again, it's going to pick actions that have higher regret with higher probability. Now, it'll just keep simulating the games this way. It'll keep, you know, accumulating regrets for different situations. And in the long run, if you pick actions that have high regret with higher probability in the correct way, it's proven to converge to a Nash equilibrium.

Speaker 2

15:24

Even for super complex games? Even for imperfect information games?

Speaker 1

15:28

It's true for all games. It's true for chess, it's true for poker, it's particularly useful for poker.

Speaker 2

15:33

So this is the method of counterfactual regret minimization?

Speaker 1

15:36

This is counterfactual regret minimization.

Speaker 2

15:37

That doesn't have to do with self-play, it has to do with just any, if you follow this kind of process, self-play or not, you will be able to arrive at an optimal set of actions.

Speaker 1

15:48

So this counterfactual regret minimization is a kind of self-play. It's a principled kind of self-play that's proven to converge to Nash equilibria, even in imperfect information games. Now you can have other forms of self-play and people use other forms of self-play for perfect information games, where you have more flexibility, the algorithm doesn't have to be as theoretically sound in order to converge to that class of games because it's a simpler setting.

Speaker 2

16:11

Sure, so I kind of, in my brain, the word self-play has mapped to neural networks, but we're speaking something bigger than just neural networks. It could be anything. The self-play mechanism is just the mechanism of a system playing itself.

Speaker 1

16:26

Exactly, yeah. Self-play is not tied specifically to neural nets. It's a kind of reinforcement learning, basically.

Speaker 1

16:31

And I would also say this process of like trying to reason, oh, what would the value have been if I had taken this other action instead? This is very similar to how humans learn to play a game like poker, right? Like you probably played poker before and with your friends you probably ask like, oh, what do you have called me if I raise there? You know, and that's a person trying to do the same kind of like learning from a counterfactual that the AI is doing.

Speaker 2

16:54

Okay, and if you do that at scale, you're gonna be able to learn an optimal policy.

Speaker 1

16:59

Yeah, Now where the neural nets come in, I said like, okay, if it's in that situation again, then it will choose the action that has high regret. Now the problem is that poker is such a huge game. I think no limit Texas Hold'em, the version that we were playing has 10 to the 161 different decision points, which is more than the number of atoms in the universe squared.

Speaker 2

17:18

That's heads up?

Speaker 1

17:18

That's heads up.

Speaker 2

17:19

Yeah. 10 to the 161, you said?

Speaker 1

17:21

Yeah. I mean, it depends on the number of chips that you have, the stacks and everything, but like the version that we were playing was 10 to the 161.

Speaker 2

17:27

Which I assume would be a somewhat simplified version anyway. Because I bet there's some like step function you had for like bets.

Speaker 1

17:36

Oh, no, no, no. That's I'm saying like we played the full game. You can bet whatever amount you want.

Speaker 1

17:39

And the bot maybe was constrained in like what it considered for bet sizes, but the person on the other side could bet whatever they wanted.

Speaker 2

17:45

Yeah, I mean, 161 plus or minus 10 doesn't matter. Yeah.

Speaker 1

17:51

And so the way neural nets help out here is, you don't have to run into the same exact situation because that's never gonna happen again. The odds of you running into the same exact situation are pretty slim. But if you run into a similar situation, then you can generalize from other states that you've been in that kind of look like that 1.

Speaker 1

18:06

And you can say like, well, these other situations, I had high regret for this action. And so maybe I should play that action here as well.

Speaker 2

18:12

Which is the more complex game, chess or poker or go or poker? Do you know?

Speaker 1

18:19

That is a controversial question.

Speaker 2

18:21

Okay.

Speaker 1

18:21

I'm gonna-

Speaker 2

18:22

It's like somebody's screaming on Reddit right now. It depends on which subreddit you're on. Is it chess or is it poker?

Speaker 1

18:27

I'm sure like David Silver is gonna get really angry at me. Yeah. I'll say, I'm gonna say poker actually.

Speaker 1

18:31

And I think for a couple of reasons.

Speaker 2

18:33

They're not here to defend themselves.

Speaker 1

18:36

So first of all, you have the imperfect information aspect. And so it's, we can go into that, but like once you introduce imperfect information, things get much more complicated.

Speaker 2

18:48

So we should say, maybe you can describe what is seen to the players, what is not seen in the game of Texas Hold'em.

Speaker 1

18:57

Yeah, so Texas Hold'em, you get 2 cards facedown that only you see. And so that's the hidden information of the game. The other players also all get 2 cards face down that only they see.

Speaker 1

19:08

And so you have to kind of, as you're playing, reason about like, okay, what do they think I have? What do they have? What do they think I think they have? That kind of stuff.

Speaker 1

19:16

And that's kind of where bluffing comes into play, right? Because the fact that you can bluff, the fact that you can bet with a bad hand and still win is because they don't know what your cards are. And that's the key difference between a perfect information game like chess and go and imperfect information games like poker.

Speaker 2

19:36

This is what trash talk looks like. The implied statement is the game I solved is much tougher. But Yeah, so when you're playing, I'm just gonna do random questions here.

Speaker 2

19:49

So when you're playing your opponent under imperfect information, is there some degree to which you're trying to estimate the range of hands that they have? Or is that not part of the algorithm? So what are the different approaches to the imperfect information game?

Speaker 1

20:06

So the key thing to understand about why imperfect information makes things difficult is that you have to worry not just about which actions to play, but the probability that you're going to play those actions. So you think about Rock-Paper-Scissors, for example. Rock-Paper-Scissors is an imperfect information game because you don't know what I'm about to throw.

Speaker 2

20:26

I do, but yeah, usually not. Yeah.

Speaker 1

20:29

And so you can't just say like, I'm just going to throw a rock every single time because the other person's going to figure that out and notice a pattern and then suddenly you're going to start losing. And so you don't just have to figure out like which action to play, you have to figure out the probability that you play it. And really importantly, the value of an action depends on the probability that you're going to play it.

Speaker 1

20:47

So if you're playing rock every single time, that value is really low. But if you're never playing rock, you play rock like 1% of the time, then suddenly the other person is probably going to be throwing scissors. And when you throw rock, the value of that action is going to be really high. Now, you take that to poker, what that means is the value of bluffing, for example, if you're the kind of person that never bluffs and you have this reputation as somebody that never bluffs, and suddenly you bluff, there's a really good chance that that bluff is going to work, and you're going to make a lot of money.

Speaker 1

21:18

On the other hand, if you've got a reputation, like if they've seen you play for a long time, and they see, oh, you're the kind of person that's bluffing all the time, when you bluff, they're not going to buy it. And they're going to call you down. You're going to lose a lot of money. And Finding that balance of how often you should be bluffing is the key challenge of a game of poker.

Speaker 1

21:37

And you contrast that with a game like chess. It doesn't matter if you're opening with the queen's gambit 10% of the time or 100% of the time. The value, the expected value is the same. So, that's why we need these algorithms that understand not just we have to figure out what actions are good, but the probabilities.

Speaker 1

21:57

We need to get the exact probabilities correct. And that's actually when we created the bot Libratus, Libratus means balanced because the algorithm that we designed was designed to find that right balance of how often it should play each action.

Speaker 2

22:10

The balance of how often in the key sort of branching is the bluff or not to bluff. Is that a good crude simplification of the major decision in poker?

Speaker 1

22:20

It's a good simplification. I think that's like the main tension, but it's not just how often to bluff or not to bluff. It's like, how often should you bet in general?

Speaker 1

22:29

How often should you, what kind of bet should you make? Should you bet big or should you bet small? And with which hands? And so this is where the idea of a range comes from.

Speaker 1

22:40

Because when you're bluffing with a particular hand in a particular spot, you don't want there to be a pattern for the other person to pick up on. You don't want them to figure out, oh, whenever this person is in this spot, they're always bluffing. And so you have to reason about, okay, would I also bet with a good hand in this spot? You wanna be unpredictable.

Speaker 1

22:59

So you have to think about, what would I do if I had this different set of cards.

Speaker 2

23:04

Is there explicit estimation of like a theory of mind that the other person has about you, or is that just a emergent thing that happens?

Speaker 1

23:14

The way that the bots handle it that are really successful, they have an explicit theory of mind. So they're explicitly reasoning about what's the common knowledge belief? What do you think I have?

Speaker 1

23:27

What do I think you have? What do you think I think you have? It's explicitly reasoning about that.

Speaker 2

23:32

Is there multiple use there? So maybe that's jumping ahead to 6 players, but is there a stickiness to the person? So it's an iterative game, you're playing the same person.

Speaker 2

23:45

There's a stickiness to that, right? You're gathering information as you play. It's not every hand is a new hand. Is there a continuation in terms of estimating what kind of player I'm facing here?

Speaker 1

23:59

That's a good question. So you could approach the game that way. The way that the bots do it, they don't, and the way that humans approach it also, expert human players, the way they approach it is to basically assume that you know my strategy.

Speaker 1

24:13

So I'm going to try to pick a strategy where even if I were to play it for 10,000 hands and you could figure out exactly what it was, you still wouldn't be able to beat it. Basically, what that means is I'm trying to approximate the Nash equilibrium. I'm trying to be perfectly balanced. Because if I'm playing the Nash equilibrium, even if you know what my strategy is, like I said, I'm still unbeatable in expectation.

Speaker 1

24:33

So that's what the bot aims for. And that's actually what a lot of expert poker players aim for as well, to start by playing the Nash equilibrium. And then maybe if they spot weaknesses in the way you're playing, then they can deviate a little bit to take advantage of that.

Speaker 2

24:47

They aim to be unbeatable in expectation. Okay, so who's the greatest poker player of all time and why is it Phil Hellmuth? So this is for Phil.

Speaker 2

24:56

So he's known, at least in part, for maybe playing suboptimally and he still wins a lot. It's a bit chaotic. So maybe can you speak from an AI perspective about the genius of his madness or the madness of his genius? So playing suboptimally, playing chaotically as a way to make you hard to pin down about what your strategy is.

Speaker 1

25:23

So, okay. The thing that I should explain first of all with like Nash equilibrium, it doesn't mean that it's predictable. The whole point of it is that you're trying to be unpredictable.

Speaker 1

25:32

Now I think when somebody like Phil Hellmuth might be really successful is not in being unpredictable, but in being able to take advantage of the other player and figure out where they're being predictable or guiding the other player into thinking that you have certain weaknesses and then understanding how they're going to change their behavior. They're gonna deviate from a Nash equilibrium style of play to try to take advantage of those perceived weaknesses and then counter exploit them. So you kind of get into the mind games there.

Speaker 2

26:02

So you think about these heads up poker as a dance between 2 agents. I guess, are you playing the cards or are you playing the player?

Speaker 1

26:10

So this gets down to a big argument in the poker community and the academic community. For a long time, there was this debate of what's called GTO, Game Theory Optimal Poker, or exploitative play. And up until about 2017 when we did the Labradors match, I think actually exploitative play had the advantage.

Speaker 1

26:29

A lot of people were saying, like, oh, this whole idea of game theory, it's just nonsense. And if you really want to make money, you got to look into the other person's eyes and read their soul and figure out what cards they have. But what happened was people started adopting the game theory optimal strategy, And they were making good money. And they weren't trying to adapt so much to the other player.

Speaker 1

26:50

They were just trying to play the Nash equilibrium. And then what really solidified it, I think, was the Labradors match, where we played our bot against 4 top heads up, no limit hold'em poker players. And the bot wasn't trying to adapt to them. It wasn't trying to exploit them.

Speaker 1

27:05

It wasn't trying to do these mind games. It was just trying to approximate the Nash equilibrium. And it crushed them. I think, you know, we were playing for $50, $100 blinds.

Speaker 1

27:17

And over the course of about 120,000 hands, it made close to $2 million.

Speaker 2

27:21

120,000 hands. 120,000 hands. Against humans.

Speaker 1

27:24

Yeah, and this was fake money to be clear. So there was real money at stake. There was $200,000.

Speaker 2

27:28

First of all, all money is fake, but that's a different conversation. We give it meaning. It's a phenomena that gets meaning from our complex psychology as a human civilization.

Speaker 2

27:43

It's emerging from the collective intelligence of the human species, but that's not what you mean. You mean like there's literally, you can't buy stuff with it. Okay, can you actually step back and take me through that competition?

Speaker 1

27:55

Yeah, okay, so when I was in grad school, there was this thing called the Annual Computer Poker Competition, where every year, all the different research labs that were working on AI for poker would get together, they would make a bot, they would play them against each other. And we made a bot that actually won the 2014 competition, the 2016 competition. And so we decided we're going to take this bot, build on it, and play against real top professional heads-up no-limit Texas Hold'em poker players.

Speaker 1

28:25

So we invited 4 of the world's best players in this specialty, And we challenged them to 120,000 hands of poker over the course of 20 days. And we had $200,000 in prize money at stake where it would basically be divided among them depending on how well they did relative to each other. So we wanted to have some incentive for them to play their best.

Speaker 2

28:47

Did you have a confidence, 2014, 16, that this is even possible? How much doubt was there?

Speaker 1

28:53

So we did a competition actually in 2015 where we also played against professional poker players and the bot lost by a pretty sizable margin actually. Now there were some big improvements from 2015 to 2017. And so-

Speaker 2

29:06

Can you speak to the improvements? Is it computational in nature? Is it the algorithm, the methods?

Speaker 1

29:11

It was really an algorithmic approach. That was the difference. So 2015, it was much more focused on trying to come up with a strategy upfront, like trying to solve the entire game of poker, like, and then just have a lookup table where you're saying like, oh, I'm in this situation, what's the strategy?

Speaker 1

29:29

The approach that we took in 2017 was much more search-based. It was trying to say, okay, well, let me in real time try to compute a much better strategy than what I had pre-computed by playing against myself during self-play.

Speaker 2

29:42

What is the search space for poker? What are you searching over? What's that look like?

Speaker 2

29:50

There's different actions like raising, calling. Yeah, what are the actions? Is it just a search over actions?

Speaker 1

29:59

So In a game like chess, the search is like, okay, I'm in this chess position and I can like, you know, move these different pieces and see where things end up. In poker, what you're searching over is the actions that you can take for your hand, the probabilities that you take those actions, and then also the probabilities that you take other actions with other hands that you might have. And that's kind of like hard to wrap your head around.

Speaker 1

30:23

Why are you searching over these other hands that you might have and trying to figure out what you would do with those hands? And the idea is, again, you wanna always be balanced and unpredictable. And so if you're a search algorithm is saying like, oh, I want to raise with this hand. Well, in order to know whether that's a good action, like let's say it's a bluff, let's say you have a bad hand and you're saying like, oh, I think I should be betting here with this really bad hand and bluffing.

Speaker 1

30:50

Well, that's only a good action if you're also betting with a strong hand, otherwise it's an obvious bluff.

Speaker 2

30:56

So if your action in some sense maximizes your unpredictability, so that action could be mapped by your opponent to a lot of different hands, then that's a good action.

Speaker 1

31:07

Basically what you want to do is put your opponent into a tough spot. So you want them to always have some doubt, like should I call here, should I fold here? And if you are raising in the appropriate balance between bluffs and good hands, then you're putting them into that tough spot.

Speaker 1

31:21

And so that's what we're trying to do. We're always trying to search for a strategy that would put the opponent into a difficult position.

Speaker 2

31:26

Can you give a metric that you're trying to maximize or minimize? Does this have to do with the regret thing that we're talking about in terms of putting your opponent in a maximally tough spot?

Speaker 1

31:37

Yeah, ultimately what you're trying to maximize is your expected winnings, like your expected value, the amount of money that you're gonna walk away from, assuming that your opponent was playing optimally in response. So you're going to assume that your opponent is also playing as well as possible a Nash equilibrium approach, because if they're not, then you're just going to make more money. Right?

Speaker 1

31:58

Like anything that deviates... Like by definition, the Nash equilibrium is the strategy that does the best in expectation. And so if you're deviating from that, then you're just, they're gonna lose money. And since it's a 2 player 0 sum game, that means you're gonna make money.

Speaker 2

32:12

So there's not an explicit like objective function that maximizes the toughness of the spot they're put in. You're always, this is from like a self play reinforcement learning perspective. You're just trying to maximize winnings and the rest is implicit.

Speaker 1

32:27

That's right, yeah. So what we're actually trying to maximize is the expected value, given that the opponent is playing optimally in response to us. Now in practice, what that ends up looking like is it's putting the opponent into difficult situations where there's no obvious decision to be made.

Speaker 2

32:41

So the system doesn't know anything about the difficulty of the situation?

Speaker 1

32:46

Not at all, doesn't care.

Speaker 2

32:47

Okay, in my head it was getting excited whenever I was making the other, the opponent sweat. Okay, so you're in 2015, you didn't do as well. So what's the journey from that to a system that in your mind could have a chance?

Speaker 1

33:00

So 2015, we got, we beat pretty badly And we actually learned a lot from that competition. And in particular, what became clear to me is that the way the humans were approaching the game was very different from how the bot was approaching the game. The bot would not be doing search.

Speaker 1

33:17

It would just be trying to compute. It would do like months of self-play. It would just be playing against itself for months, but then when it's actually playing the game, it would just act instantly. And the humans, when they're in a tough spot, they would sit there and think for sometimes even like 5 minutes about whether they're gonna call or fold a hand.

Speaker 1

33:36

And it became clear to me that that's, there's a good chance that that's what's missing from our bot. So I actually did some initial experiments to try to figure out how much of a difference does this actually make? And the difference was huge.

Speaker 2

33:48

As a signal to the human player, how long you took to think?

Speaker 1

33:52

No, no, no, I'm not saying that there were any timing tells. I was saying when the human, like the bot would always act instantly. It wouldn't try to come up with a better strategy in real time over what it had pre-computed during training.

Speaker 1

34:04

Whereas the human, like they have all this intuition about how to play, but they're also in real time leveraging their ability to think, to search, to plan, and coming up with an even better strategy than what their intuition would say. So

Speaker 2

34:17

you're saying that there is, you're doing, that's what you mean by you're doing search also.

Speaker 1

34:21

Yeah.

Speaker 2

34:22

You have an intuition and search on top of that looking for a better solution.

Speaker 1

34:28

Yeah, that's what I mean by search. Instead of acting instantly, a neural net usually gives you a response in like 100 milliseconds or something. It depends on the size of the net.