Artificial Intelligence in Poker Infographic: History and Implications
The Terminator, I, Robot, and A.I. Artificial Intelligence are among the many films in the past four decades with the theme of artificial intelligence evolving beyond humans' reach and leading to the end of humankind. While such a scenario remains unlikely in the near future, many believe that at least the first part – AI surpassing the human brain and mind to reach a level of artificial superintelligence (ASI) – will most likely happen some time in the next 30 years, according to Jeff Nesbit in U.S. News & World Report.
Whether ASI leads to the end of humanity remains to be seen, but AI has certainly made a mark on the poker world recently.
Poker has become relevant in the field of AI with a history that dates back to 1984, the same year viewers first saw Arnold Schwarzenegger playing the cyborg "Terminator" on screen and probably thought the theme was truly sci-fi — not a possibility in the next century. That year, Mike Caro created a basic Poker AI software called Orac to compete in the World Series of Poker.
Poker AI has advanced quite drastically since that time, and a recent infographic provides an in-depth visual and statistical overview of the evolution of AI in poker along with insights from the research to date.
Poker & AI Infographic
A couple of months back, Josh Wardini, editorial contributor and community manager for various online poker websites, published the infographic titled “Poker and AI: The Rise of Machines Against Humans” on the PokerSites blog. Wardini provides an organized, visual presentation of the relationship between poker and artificial intelligence over the past 30 years.
While AI research has been done in a variety of fields, applying it to strategy games has been one of the most prevalent. In the last two decades, AIs have defeated humans in a variety of games including chess, Go, and Jeopardy. In the past decade, much of that research has shifted to creating poker-playing AIs.
One question you may ask is why poker?
According to Wardini, AI departments at big universities do research on poker because of the game’s complexity. Like many real-life situations, “poker is dealing with loads of imperfect information” and the problems that AIs aim to solve are thus reflected in poker.
Noam Brown, Ph.D. student and AI researcher at Carnegie Mellon University (CMU), said there are 10^163 (10 followed by 163 zeroes) different game situations in no-limit hold’em assuming starting stacks of 20,000 chips, making it impossible for even the strongest computers to calculate all possible situations.
Poker AI Research Milestones
According to the infographic, the leading research in Poker AI has come out of University of Alberta, University of Auckland and the aforementioned CMU, with the latter perhaps the most famous in the poker world for putting out two AIs, Claudico and Libratus, that recently faced off against some of poker’s top heads-up players.
In 2015, Claudico was defeated by a team of four players. Doug Polk, Bjorn Li, Dong Kim and Jason Les prevailed, but the bot had some victories along the way.
Polk recently told PokerNews that Claudico "used a very complex strategy, consisting of many small and large bets" and the AI had some leaks.
The new and improved iteration, Libratus, faced off against four top pros in January 2017. Kim and Les returned for the second battle, joined by Jimmy Chou and Daniel McAulay. After 120,000 hands of heads-up no-limit hold’em played out bot vs. human, Libratus had won $1,766,250 in chips from the four pros, a very different result from the first edition of "Brains vs AI."
Dong Kim spoke with PokerNews’ Sarah Herring after the matches.
“It was really difficult for us to play," he said. "We would bring a strategy and it would be good for the day of, and then the next day it would bring something new to the table. And we were not ready for that, so it was overall really, really tough.”
The reason for that? Libratus was self-correcting. The AI analyzed its own play each night, correcting mistakes and using different strategies the following day. But that’s not all.
Unlike its predecessor bots, Libratus relies on an algorithm that computes the strategy based on millions of core hours of computation made possible by the Pittsburgh Supercomputing Center (PSC) “Bridges” source rather than having fixed built-in strategies. It also randomizes its actions, making it hard for opponents to decipher whether it is bluffing or not. Most amazingly, Libratus is not just a poker AI, but can be applied to any imperfect information system, with countless potential applications.
Kim acknowledged the impressive abilities of his fleshless opponent.
“The AI played really well and I think it’s world-class.”
When asked which of Libratus’ strategies he can employ in his own game, Kim explained that the bot’s mixed strategy made it difficult to play against.
“There are a lot of good things that I would like to incorporate into my game, but the execution’s really tough," Kim said. "It has these really, really great strategies but it’s also a mixed strategy so the moment you do one thing a little bit too much, then you’re already too exploitable, so it’s going to be pretty hard to mimic or emulate in general so we’ll see. I’ll take the simple ones.”
Libratus frequently employed overbets in its crushing performance.
“It overbets more than anyone," Kim said. "If it was a human player, we would have thought it was a complete fish to be honest because it just does it so much.
“Actually when it overbets it actually bluffs a lot. Yeah, it just bluffs a lot.”
Brown, the CMU researcher, explained to PokerNews back in March 2016:
"It's not that surprising [that bots bluff], because what the computer is doing is simulating trillions and trillions of hands of poker, and it doesn't really think about bluffing the way we do. Really what it does is it runs into a situation repeatedly and it notices that even if it doesn't have a very strong hand that it can still make more money by betting."
Implications for AI
Libratus' co-creator Tuomas Sandholm assessed the implications of the bot's victory in a release from Carnegie Mellon.
“The best AI's ability to do strategic reasoning with imperfect information has now surpassed that of the best humans," he said.
Polk echoed that sentiment when he told PokerNews, the leaks he noticed in Claudico were fixed in Libratus.
"It became clear to me at this point humans had fallen behind the curve in the game of heads-up no limit, and I imagine this will continue to happen in other formats."
As far as real-world implications, which are the ultimate goal of research on AI in poker, the success of Libratus — in particular its ability to bluff — is momentous.
Frank Pfenning, head of the Computer Science department at CMU, said incorporating bluffing was a key concept, as it couldn't win without the ability.
“Developing an AI that can do that successfully is a tremendous step forward scientifically and has numerous applications," he said. "Imagine that your smartphone will someday be able to negotiate the best price on a new car for you. That's just the beginning.”
Comparing Humans and Poker Bots
While most poker players will not be going head-to-head with a poker AI anytime soon, it could be fruitful to examine how the AIs’ poker skills stack up against human players. What makes AIs like Libratus so good?
While having some emotion while playing poker is not inherently bad — and may even be beneficial in some cases — one major characteristic that separates successful professionals from break-even or losing amateurs in the game is the ability to imitate robots, in a sense.
While most pros and poker coaches wouldn’t put it in those terms, they tell aspiring professionals that some keys to success in the game are not thinking about the money when making decisions in the game, not letting financial results affect you, staying emotionally level and not going on tilt.
In a PokerNews article by Andrew Robl in 2016, he notes six characteristics that make for a successful player, five of which can be ascribed to a bot like Libratus: experience, intelligence, desire and willingness to learn (minus the desire part), ability to control emotions and having a "sick" amount of gamble.
The sixth one is all humans, though — social and networking skills. The bots are lousy at that — for now.
The AIs have a clear advantage over humans in terms of emotion, including tilt and fatigue factors. When a bot like Libratus goes on a losing streak, it’s not going to get angry or frustrated, berate its opponent or start playing hands less optimally. It’s not going to get mentally exhausted after a long day of grinding and start making mistakes as a result. And it is a better player because of it.
One comment from Kim after facing off against Libratus highlighted this key difference between bots and humans:
“We didn’t expect to be losing so much, and that was the tough part… It wasn’t necessarily dealing with AI itself. It was more the constant losing. Team morale was pretty down.”
Meanwhile, Libratus kept chugging along, adapting its strategy based on experience, consistently making optimal plays.
AIs don’t feel the value of money, which allows them to be unaffected by risk and always make the “correct” game theory optimal (GTO) play, even when the risk-to-reward ratio is steep. This leads to a fair amount of bluffing.
Poker coaches talk about the necessity of making positive expected value decisions and not being results-oriented, which the poker AIs have down due to their very nature. The bots are all about making the best decision at each turn no matter the risk, and it helps that they do not have bankroll concerns or bills to pay.
Hope for Humans
It seems we have a lot to learn from these poker-playing bots, but don't get tilted — there are some poker skills that we humans are better at than the machines.
While bots are very adept at arriving at a GTO strategy by simulating trillions of hands, they are less capable of employing exploitative strategies against individual opponents because of the limited sample size of hands they can play against a given opponent.
“It's an area of active research, how to exploit an opponent over a limited number of samples," Brown said. "We don't know why human players are very good at this, and it is probably the key area of weakness in poker bots today.”
And while poker players might quake in fear at the thought of emotionless robots raking in all of their money, the development of these programs can actually have a positive effect on their bottom line. Wardini mentions existence of "AI simulations and training software to help humans," including tools to help players work on developing a “balanced, non-exploitative approach” and employing “better decision making” to improve their game.
For now, we can take these strategy tips and try to improve, and be thankful we don't have to go up against super-computing robots when we hit the tables.
