Heads-Up Bots Will Be Stronger Than Humans; No-Limit Hold’em May Never Be Solved
A recent PokerNews article that caught the attention of our readers was The Perfect Poker Bot Is Almost Here. The article described a bot known as Baby Tartanian 8, which had won the most recent Annual Computer Poker Competition and how researchers plan to create a perfect poker bot within two to three years.
The bot's creator, Noam Brown, was kind enough to take some time from his busy schedule to chat about his award-winning bot, including how it came to fruition, and even how it would perform in the online poker world; the answer to the latter is quite worrying.
Brown explained that the Annual Computer Poker Competition has been running since 2006, with no-limit hold'em featured from around 2008, and during that time, as you'd imagine, the bots have grown more complex.
"I think back in 2008-2009 we were just starting on no-limit Texas hold'em, and the bots would only consider calling, folding, making a pot-sized bet, or going all in," Brown said. "When you look at the card abstractions, which is what we call them, when you consider the possible hands, you have to group them together and treat a lot of hands identically to keep the game a reasonable size. Back then, all of the possible river hands were grouped into maybe 500 buckets and this time we did around 1 million buckets."
Writing the code for the bot took approximately one month to complete, then the bot was run for about a month on the San Diego Super Computer, which has a staggering 3,700 CPUs 18 terabytes assigned to it. However, you don't need access to a super computer to create such a powerful bot, as is evident by two other teams who finished in the top three of the competition Brown won.
"Actually, this year for example, we were one of the top three bots, but the other two were not part of a university or research group, they were out there on their own," Brown said. "One of them is a professional poker player who is also a software engineer, and one of them is an amateur poker player who is a former employee of Google, and both of them made great bots on budgets of under $10,000.
"The cost of computing has come down dramatically, particularly in the past year, so they were able to leverage Amazon's Cloud computing services to make competitive bots, so it's not out of reach of the average person to make a bot similar to ours, assuming they know how to make a bot like this, which is a tricky thing to do."
If you frequent poker forums then you will no doubt have heard claims that certain poker variants are "solved," that is it's possible to play absolutely perfectly, but Brown claims that no-limit hold'em will not be solved in our lifetimes, if ever.
"Limit hold'em and no-limit hold'em are very different from a competition standpoint," he said. "In limit hold'em, you have three options: call, fold, or bet. In no-limit hold'em you have essentially an infinite number of options because you can bet any amount, and each of those different bets you can make is treated by the computer as a separate action. If you wanted to solve no-limit Texas hold'em you would have to consider, if you were talking of chip stacks of 20,000 chips as we use in the annual poker computer competition, that you're talking about a game that is something like 10^163 (10 followed by 163 zeroes), whereas limit Texas hold'em is probably around 10^15, so the scale is unimaginable. There is no chance that no-limit Texas hold'em is going to be solved within our lifetime, if ever. That said, there are ways to get good approximate solutions and I think that certainly in the next few years we will see a bot that can take down the very top pros, but there is a big difference between that and solved."
Brown went on to explain that many people make false assumptions about the best poker bots in that they believe the bot has to be predictable, but this isn't the case, they even bluff.
"It's not that surprising [that bots bluff], because what the computer is doing is simulating trillions and trillions of hands of poker, and it doesn't really think about bluffing the way we do," he said. "Really what it does is it runs into a situation repeatedly and it notices that even if it doesn't have a very strong hand that it can still make more money by betting."
The bots do have some weaknesses, though. One is that the bots on this high level are limited to heads-up poker and do not perform well in, say, a six-handed environment. This could be set to change, because Brown revealed there was a lot of interest at this year's Annual Computer Poker Competition for a six-player contest and he believes that this will happen at the 2017 edition of the competition, although those entering it will need to put in a lot of time and effort to have a competitive bot.
Another weakness the bot shows is its ability to exploit weaknesses in its opponents with one reason being it needs to play trillions of hands to create a Game Theory Optimal strategy, so it wouldn't be able to play enough hands against a human opponent. When we asked Brown if bots can exploit opponents' weaknesses, here's what he had to say:
"That's a great question," Brown said. "When we're doing these trillions of hands we're playing against our own bots and it's proven that if the bot plays itself then it will eventually converge to a Nash Equilibrium, that is Game Theory Optimal (GTO) strategy that is impossible to beat. Now there's another component to poker which is opponent exploitation. Just because you have a GTO strategy does not mean you are going to make as much money as possible from your opponents. This is something that human pros are very good at, they find a weakness and take full advantage of it. Obviously you could try to exploit your opponent and their weaknesses, but you're opening yourself up to exploitation and therefore not playing a GTO strategy, but it is still in many cases worthwhile.
"There has been a lot of research into how to do this with a bot, and unfortunately it hasn't been very successful so far. The techniques that we use, where we play trillions of hands and it learns what works against an opponent and what doesn't work, would work against human pros also but the problem is you would have to play trillions of hands against your opponent to create a strategy against them. The techniques we use now cannot arrive at a good strategy to respond an opponent's weaknesses in a small number of hands. It's an area of active research, how to exploit an opponent over a limited number of samples. We don't know why human players are very good at this, and it is probably the key area of weakness in poker bots today."
Current limitations aside, Brown is confident that Baby Tartanian 8 would dominate the heads-up no-limit hold'em scene if it was deployed online. But would Brown's bot clean up at online poker?
"Definitely," he said. "If we're talking about two player, yeah, it would definitely win. Our previous bot played against the top four heads-up players in the world, and although it didn't come out ahead, it was very competitive at that level and this agent is far stronger, so I think there are very few people who would be able to beat it."
Although Brown is extremely confident that bots such as the Baby Tartanian 8 would dominate heads-up games if they were deployed in the online poker world, he does not believe that bots as strong as his will be widespread online, as some people believe.
"I certainly think two-player bots will be stronger than humans, but I don't necessarily think that they will be widespread online," he said. "I think the online poker sites are very effective at detecting and removing bots."
You can discover more about Noam Brown and his research via his website www.noambrown.com.
Get all the latest PokerNews updates on your social media outlets. Follow us on Twitter and find us on both Facebook and Google+!
