Simulating The November Nine Part 1: The Process and Results
In 2012 when Russell Thomas made the final table of the World Series of Poker Main Event, he hired poker pro Jason Somerville to coach him.
Somerville has spoken out about this training regimen in several interviews, part of which included a number of final table simulations. It was shown in an excellent video series created and produced by Somerville and shot and edited by Jake Gabbay and Justin Tyrrell, dubbed The Final Table.
Somerville brought in pro players and friends to emulate the styles of the different players in the November Nine and they put Thomas through the paces. They recreated the conditions he would encounter at the final table, analyzed it all and developed strategies around the different situations he would run in to, making sure he was prepared for anything and everything.
I can remember thinking these simulations were likely to ensure Thomas, who ultimately made fourth at the final table, would come into the final nine among the most prepared.
This year, when the final nine was set in mid July, hordes of poker media descended upon the Amazon Room stage seeking quotes from this year's November Nine. In the melee, I joined a scrum surrounding Gordon Vayo and asked what his plan was to prepare for the final table.
"I'm going to do a lot of playing and run simulations," he said. "I have some friends that are really good poker players and hopefully they can emulate some of the styles of these guys and we can run some good simulations to prepare."
For the media, that night was about capturing the excitement of the players making the November Nine and getting some honest initial reaction. As has become custom, the next day the media was invited to sit down with the players in a more casual setting at the Palazzo Suites next to the Rio All-Suite Hotel & Casino and the Convention Center where the WSOP is held. PokerNews conducted relatively in-depth audio and video interviews with the players there. These formed the basis for a series of November Nine bios that have run on the site for the past nine weeks.
Planning Our Own Simulation
As we ran through the process of creating these bios, outlining who these players were and how they managed to make this year's Main Event final table, it dawned on me how valuable the information would be to anyone running a Main Event simulation. The facts contained in each could provide a solid blueprint for emulating each of the player's styles in a simulation. Then I started thinking perhaps PokerNews should run one.
Using the information we had gathered, we could devise a basic playing style for each of the players, simulate the final table a number of times and provide readers a reasonably researched prediction of how things would shake out come November. By no means did I ever think we could predict exactly who would win the Main Event this year with any real accuracy, there are just too many variables for that, but it certainly seemed like a fun way to create some interesting content and give us all a better idea of who might come out on top.
Of course, whether we used PokerNews staffers, who live everywhere from Jomtien Beach, Thailand to San Diego, California, or pro players, who have relationships with the site and are almost always on the road, putting together a live simulation of the November Nine looked like a logistical nightmare. We might have been able to figure out how to run the game on Skype, but at some point, what we would be doing would become so far removed from actually playing poker, the simulation would lose all semblance of accuracy and credibility.
Creating November Nine Bots
This is where Advanced Poker Training came into the picture. The organization, which was founded by a pair of Floridian academics, software developers and brothers named Steve Blay and Allen Blay, is a unique website that offers poker training, not in the usual video tutorial format, but by allowing players to play in simulated games against artificial intelligence bots. Coincidentally, APT actually sponsored PokerNews' series of November Nine bios this year. A quick look on their website revealed they've designed thousands of unique AI bots with different playing styles, weak and strong, for players to test their mettle against.
It wasn't long before I started thinking that the Blay brothers, using information from the bios on PokerNews, might be able to create bots for each of the November Nine and set them up to play in a simulation against themselves, using all the data from the Main Event and its structure. Within the first minute of our initial meeting on Skype to discuss the idea, the Blays confirmed it was not only possible, but a fun idea they'd love to be involved with.
Steve Blay, who is actually the software developer behind the APT product, explained that they designed bots based on a diverse set of criteria that included tendencies, frequencies and adjustments real players might have or might make. First and foremost, however, the AI bots were designed to actually play like real humans, warts and all. This meant the information gathered from the November Nine bios on PokerNews, along with select hand histories from the WSOP's own event coverage, could allow them to create bots that emulate the style of play of each of the 2016 November Nine.
No one was pretending these bots would be perfect clones of this year's final table participants, as there are just too many personal variables to consider; and the hand histories available represent such a small sample size, it would be impossible to accurately represent each player's unique and different approach to the game. However, a decision was made to move forward with creating the bots and running the simulations because the overall feeling was it would provide something that, at the very least, was as accurate as the live training simulations others had done before.
The following basic bios were provided to Steve Blay and he attempted to map the information about each player, their personality, playing style and experience to the 42 configurable personality traits available for APT's AI bots, adjust them based on an assessment of how comfortable each player would be in the spotlight and ultimately create bots that played similarly to each of the November Nine:
| Cliff Josephy |
| Cliff Josephy is the most experienced and most accomplished player in the group. He is also the oldest, all of which suggests he will make the least amount of mistakes. He has previously won two WSOP Bracelets, has been ranked as the number one online poker player in the world and is one of the originators of the Poker X Factor training site, suggesting he is in tune with today's modern aggressive playing style. He is also known for running a longtime online and live poker backing operation, suggesting he knows how to evaluate poker talent and therefore adjust well to opponent's different tactics and playing styles. All indications are that Josephy will use his chip lead and experience to put pressure on his opponents, suggesting he will play a wider range of hands more aggressively and adjust quickly to a more tight-aggressive style should this initially fail. |
| Qui Nguyen |
| Qui Nguyen is the least experienced player in the group and has admitted to being more of a gambler than a poker player. This, and the fact he has the second largest chip stack, suggests he will play a relatively loose-aggressive style: Boom or bust. Unlike Josephy, Nguyen is more likely to continue to gamble if things do not go well from the start. His lack of experience at this level also suggests he is more likely to make a mistake. |
| Gordon Vayo |
| Gordon Vayo is one of the more experienced online players from the group, indicating he could straddle the line between the tight-aggressive and loose-aggressive playing styles that are known to be successful online. In fact, he has been backed by Cliff Josephy in the past, suggesting he plays a similar style. Considering his stack size, Vayo is likely to play a tighter range of hands than Josephy. Vayo has claimed he will be running simulations of the final table himself, potentially making him one of the most comfortable players (in many scenarios) coming in and among the least likely to make a mistake. |
| Kenny Hallaert |
| Kenny Hallaert has experience both playing and organizing poker tournaments, suggesting he understands how much play there is left in this structure and how to use it to his advantage. While he may want to play the more loose-aggressive style popular in Europe, his stack size and understanding of the structure may tighten his range up a little and he will likely play a more tight-aggressive style. Hallaert's industry experience also indicates that he may be among the least likely to make a mistake. |
| Michael Ruane |
| Michael Ruane is an experienced online poker player, suggesting he is comfortable with the math side of poker and tends to straddle the line between the tight-aggressive and loose-aggressive playing styles that are known to be successful online. Considering his stack size, it is likely he begins the final table playing a tighter range of hands and loosens up based on table dynamic and stack size adjustments. |
| Vojtech Ruzicka |
| Vojtech Ruzicka is an experienced poker player, online and live. He appears comfortable with the math side of poker, can play both a tight-aggressive and loose-aggressive playing style and adjusts well to his opponents. However, all indications show he is a little uncomfortable with the spotlight and his command of the English language. All this indicates that he may be more likely to make a mistake under the added pressure of the stage. Considering his stack size, it is likely he begins the final table playing a tighter range. |
| Griffin Benger |
| Next to Cliff Josephy, Griffin Benger is the most experienced and most accomplished player in the group. He would also have to be considered the most comfortable in the spotlight, considering his background as a professional gamer, all of which suggests he will make among the least amount of mistakes. He has some previous experience running relatively deep in the Main Event and has also been ranked as the No. 1 online poker player in the world, suggesting he is also very well practiced with today's modern aggressive playing styles. Benger is somewhat handcuffed by his current stack size, suggesting he will play a more tight-aggressive style to start out, but would have to be considered the most likely to adjust to the playing styles of the rest of the table and act accordingly and optimally. |
| Jerry Wong |
| Jerry Wong has a lot of live poker experience, suggesting he can play the short stack optimally. He likely understands shoving ranges, knowing what to wait for and how long he can. Wong will likely be looking to double up early and see if he can make deeper run. |
| Fernando Pons |
| Fernando Pons is an online qualifier who will likely be looking to gamble early to see if he can double up to a spot where laddering up becomes a realistic consideration. He is the most likely to consider it a win to have made the final nine in the first place and will likely play with little fear of finishing ninth. Should he ladder up, his inexperience may make him more likely to make a mistake the deeper he goes. |
Running 100 Sims
Additional strategic elements were added to the bots' playing styles, including that big stacks will generally be more aggressive against small stacks and that smaller stacks will be looking for opportunities to get it all in.
Two predictions were made, including the number of wins based on each stack size as a percentage of the total chips in play and the average number of dollars won, using the Independent Chip Model (ICM). A total of 100 simulations were run with consideration made for the speed of play, blinds beginning at 250,000/500,000 with an ante of 75,000, 32:50 left in Level 35 of the existing structure and the chip counts where they sit currently according to the WSOP. The outcomes were as follows:
2016 November Nine Simulation Results (100 Sessions):
| Player | Actual Wins | Predicted Wins (Per 100) | Total $ Won | Actual Average $ Won | Predicted Average $ Won | Percent Difference |
|---|---|---|---|---|---|---|
| Cliff Josephy | 17 | 22 | $381,472,246 | $3,814,722 | $4,089,787 | -7% |
| Michael Ruane | 9 | 9 | $259,000,392 | $2,590,004 | $2,729,587 | -5% |
| Gordon Vayo | 15 | 15 | $355,544,422 | $3,555,444 | $3,375,798 | 5% |
| Kenny Hallaert | 13 | 13 | $326,954,083 | $3,269,541 | $3,172,393 | 3% |
| Jerry Wong | 1 | 3 | $158,120,494 | $1,581,205 | $1,677,554 | -6% |
| Griffin Benger | 6 | 8 | $219,990,335 | $2,199,903 | $2,498,299 | -12% |
| Vojtech Ruzicka | 8 | 8 | $244,699,268 | $2,446,993 | $2,547,844 | -4% |
| Fernando Pons | 5 | 2 | $169,517,183 | $1,695,172 | $1,423,125 | 19% |
| Qui Nguyen | 26 | 20 | $427,993,577 | $4,279,936 | $3,918,534 | 9% |
The Quick Takeaways
Since the winner of the WSOP Main Event is considered poker's world champion, our initial statistical analysis focused on who won the simulations. Steve Blay said he was amazed how close the data came to the actual predicted values, as he'd expected the numbers to be all over the place.
However, what most people might find the most surprising is that Qui Nguyen won 26 out of 100 times - The most of any player. As it was designed to do, the Nguyen bot played the riverboat gambler to perfection and it paid off. The fact he was only up by nine percent overall indicates he likely crashed and burned a bunch of times as well, which only serves as further proof this bot was the gambler of the bunch.
It was also somewhat surprising that chip leader and clear favorite Cliff Josephy underperformed, winning only 17 of the 100 simulations. He was expected to win 22, but the fact that his average money won was down only seven percent overall suggested he finished rather high in most of the simulations, as one would expect.
888poker qualifier Fernando Pons managed to win five times, although he was predicted to only win twice. He also won 19 percent more prize money than was expected, leading the group in that category. Blay figured the bot's no-fear attitude contributed greatly to its ability to punch above its weight on average.
The rest of the group won about as many times as predicted, with Griffin Benger's experience edge leading to a couple extra wins. Jerry Wong won only one time, twice less than predicted, suggesting his bot made a consistent effort to double up early as designed, but simply could not get lucky enough times in this sample size to make it count.
Next week, in Part 2 of this series, we will look closer at some of the key hands in specific simulations that led to these outcomes. A week later, in Part 3 of the series, we will provide a strategic analysis of the performance of each of the November Nine bots, outlining their keys to victory and important pitfalls to avoid. Finally, in the week leading up to the live 2016 WSOP Main Event final table, we will recap the entire series.
