NexusGTO

Six-player poker AI

NexusGTO

NexusGTO is an AI for six-player no-limit Texas hold'em. It follows the design of Pluribus, which in 2019 became the first program shown to beat elite professionals at that game.

NexusGTO plays a blueprint strategy trained over 8.05 billion iterations of self-play. It searches only when two or three players are left after the flop.

Heads-up in the blinds, a local best responder built to exploit NexusGTO won about 115 big blinds per 100 hands from the blueprint alone, while a player who folded every hand would have lost 75. But against a version of NexusGTO's search that tracks what its actions reveal about its hand, the best responder's winnings on the same deals were too close to zero to tell apart from luck.

Read the write-upSee the results

BTN Plays 45% of hands after three folds

The blueprint's first-in strategy at each seat: ButtonA grid of the 169 starting hands: pairs on the diagonal from aces down to twos, suited hands above it and offsuit hands below. Each hand is a dot, larger the more often the blueprint plays it, black for the share it raises and grey for the share it only calls. Plays 45% of hands after three folds.
The blueprint's first-in strategy at each seat: ButtonA grid of the 169 starting hands: pairs on the diagonal from aces down to twos, suited hands above it and offsuit hands below. Each hand is a dot, larger the more often the blueprint plays it, black for the share it raises and grey for the share it only calls. Plays 45% of hands after three folds.
Fig. 1. The blueprint's first-in strategy at each seat. Each dot is one of the 169 starting hands, larger the more often it's played.

Pluribus

Noam Brown and Tuomas Sandholm, researchers at Carnegie Mellon and Facebook AI Research, described Pluribus in a 2019 paper in Science . Pluribus played professionals in two formats, and the paper scored both with AIVAT, which subtracts the luck it can calculate. In the first, five professionals drawn each day from a pool of 13 played one copy of Pluribus for 10,000 hands over 12 days. Pluribus won 4.8 ± 2.5 big blinds per 100 hands (bb/100, ± one standard error), and the paper's one-sided test found the win significant. In the second, one professional at a time played five copies of Pluribus. Two professionals played 5,000 hands each, and Pluribus won 3.2 ± 1.5. Earlier poker programs had beaten professionals only one on one (heads-up).

Most strong poker programs learn by counterfactual regret minimization (CFR). They play themselves and shift toward the moves they regret not making. Plain CFR walks the whole game tree on every pass, but no-limit hold'em's tree is far too big for that. And only in heads-up play is any form of CFR guaranteed to converge to an equilibrium. A heads-up equilibrium is a strategy that can't lose on average. With six players Pluribus had no such guarantee, so it aimed to beat strong players instead.

Pluribus used Monte Carlo CFR, which samples the tree instead of walking all of it, to learn one rough strategy for the whole game. Its makers called that strategy the blueprint. During play Pluribus also searched on the flop, turn and river, each time from the start of the betting round. Each search worked like a poker solver, but where a solver takes the ranges from its user, Pluribus built each player's range from the blueprint and the actions so far. NexusGTO works the same way, except that it searches only with two or three players left.

In Brown and Sandholm's view, Pluribus's most important change for six players is a search that can stop before the end of the hand. Earlier heads-up programs either searched to the end of the hand (Libratus) or needed huge numbers of precomputed solves (DeepStack), and both get too costly as players are added. Where a Pluribus search stops short, each remaining player picks one of four versions of the blueprint to play out the hand: unchanged, or shifted toward folding, calling or raising. That way the search never assumes that everyone keeps to the blueprint. NexusGTO's three-player flop searches end in the same four strategies. So do its three-player turn searches, which Pluribus ran to the end of the hand. Its heads-up searches run through the river, as Pluribus's did.

PluribusNexusGTOWhy
Blueprint trainingMonte Carlo CFR, 8 days on one 64-core server (12,400 core-hours)The same Monte Carlo CFR and Pluribus's discounting and pruning schedule, about 10 days and 26,500 core-hours on rented AWS serversPluribus timed its discounting and pruning in minutes of an 8-day run, so NexusGTO sets them at the same fractions of its total iterations
Bet sizes1 to 14 raise sizes in the blueprint, 1 to 6 in searchUp to two raise sizes before the flop. After it, 0.75 pot for a first bet and pot for later raises, in the blueprint and the search alikeAn early tree with more preflop sizes grew too big to train, and changing sizes means retraining from scratch
When it searchesAfter the flop, with any number of players. Before it, only after a raise well off the blueprint's sizes and with four or fewer players leftAfter the flop, with two or three players leftIn later tests, four-way spots came up about once in 254 hands, so searching them changes little; a five-way flop search took 38 to 76 s against a 3 s budget
Time per search1 to 33 s on 28 cores3 s on 8 threadsA later paired test of an early search configuration found 10 s no better than 3 s
How it was tested10,000 hands of five elite professionals against one copy, and 5,000 hands each for two other professionals against five copiesA best responder, and matches against copies of its blueprintNo outside six-player program was available, and a best responder gives a lower bound on how much NexusGTO can be exploited

Where NexusGTO differs from Pluribus, and why.

How it was built

  1. NexusGTO began on March 23 as a two-player solver, with rules of thumb planned for pots of three or more players. The next day it switched to Pluribus's design, a six-player blueprint with search. How it works →
  2. The trainer runs Monte Carlo CFR on the CPU, because in a March 24 test a GPU version with a noisier sampling method took 5 to 10 times as long to make the same progress. The GPU test →
  3. Several bugs turned up in training. The trainer groups similar hands into buckets that share a strategy, but for about 16 days one of them made it store each flop hand separately. Every blueprint trained in that time had the bug. One passed 9 of 12 preflop sanity positions, and a continuation of it passed only 2 and was set aside. The final run started from scratch on May 12, with a check that won't let training begin unless three test flops each land in a bucket. It wrote the 12.2 GB blueprint by May 23. Four training bugs →
  4. On June 1 a benchmark compared search on the GPU and the CPU. The GPU solver's time per iteration grew from 1.89 ms with two players to 2,049 ms with six, because it updates every hand in each range on every pass. The CPU search samples one hand per player instead, so range size doesn't drive its cost. At five players it came closer to equilibrium than the GPU solver did, and in less time. Five-player search moved to the CPU that day, and the rest followed by June 6. CPU over GPU →
  5. The July 1 target was to match Pluribus in everything except bet sizes. Pluribus never searched for more than 33 seconds, so a six-way flop search had to finish in that time. Two speedups on July 29 cut a benchmark six-way flop search (250,000 iterations on 32 vCPUs) from about 22 minutes to 13.4 seconds. The engine ended up differing from Pluribus in more than bet sizes, as the table above shows. Speedups →
  6. Matches replay each deal six times, with the tested player in each seat once. The rotation cancels much of the card luck, and AIVAT removes more. In a large batch of matches that finished on September 13, the search as it stood then didn't beat five copies of its own blueprint. Chips won from copies of the blueprint don't show how much NexusGTO can be exploited, so from September 16 a best responder also played against it. How it was measured →
  7. The tests led to several changes, including these three. Opponents' ranges built from preflop actions alone had kept hands their later actions, checks included, ruled out. A fix tested on September 16 narrows each opponent's range with every action that opponent takes, and the search has used it since September 23. An audit on September 17 found that the flop search rated its own options about half a pot too high, because of the values it assumed where its tree stopped. Heads-up flop searches have run through the river since. On September 23 a version of the search that also tracks its own range joined the tests, because a search that assumes it played the blueprint leaves its real range open to a best responder. The first search →
How NexusGTO's parts fitThe trainer computes the blueprint offline. Before the flop, or with four or more players left, the engine plays the blueprint's strategy. Once the flop is dealt and only two or three players remain, a search on the CPU re-solves the current betting round each time the engine is to act. The search takes the opponents' ranges, and the values where its tree stops early, from the blueprint. An evaluation harness plays the engine against copies of the blueprint and a local best responder.OfflineWhen the engine actsTrainerExternal-sampling MCCFRin C with OpenMPBlueprint8.05B iterations.bps file, 12.2 GBOn the flop, turnor river, with2 or 3 players left?CPU search3 s budget, 8 threadsuses the blueprintBlueprint strategylooked up, no searchActionYesNoEvaluation harness:duplicate deals, AIVAT, local best response
Fig. 2. The trainer writes the blueprint once; the engine either plays it or uses it to set up a search.

How well it plays

The best responder played the same 2,000 deals against each version, heads-up in the blinds with 100-big-blind stacks. It took 114.8 ± 21.1 bb/100 from the blueprint alone, 64.4 ± 17.6 from the untracked search (a version of NexusGTO's search that doesn't track its own range) and 9.2 ± 18.6 from the tracked search, which does. For the tracked search the 95% interval (1.96 standard errors either side) runs from −27.3 to 45.7, which includes zero. These figures value each showdown over every hand the version could hold. On that measure the search and the tracking each brought the best responder's winnings down, and together they cut them by 105.6 bb/100 (95% interval 58.1 to 153.1). In chips actually won, the tracking's part isn't significant.

A stronger exploiter could win more, because the best responder looks only one step ahead and bets only NexusGTO's own sizes.

In six-player matches against five copies of the blueprint, the tracked search won 5.73 bb/100 over 14,250 deals with no rake (95% interval 2.74 to 8.72). The blueprint would score exactly zero against five copies of itself, so the search plays better than the strategy it starts from, at least against these opponents.

The best responder against three versions of NexusGTOBars show the bb/100 a local best responder took heads-up in the blinds at 100 big blinds, with each showdown scored as an average over every hand the version could hold. Each bar has a 95% interval of the value ± 1.96 standard errors. On the same 2,000 deals it took 114.8 from the blueprint alone, 64.4 from the untracked search and 9.2 from the tracked search, which tracks its own range. The tracked search's interval crosses zero. Lower is better for NexusGTO.−50050100150200bb/100 the best responder took, with 95% intervalsBlueprint aloneNo search+114.8 ± 21.1Untracked searchNo own-range tracking+64.4 ± 17.6Tracked searchTracks its own range+9.2 ± 18.6
Fig. 3. What the best responder took from each version, heads-up in the blinds, on the same 2,000 deals, with each showdown scored as an average over every hand the version could hold. Each label is the value ± one standard error, and each line is its 95% interval. Lower is better for NexusGTO.
The numbers
VersionDealsbb/100
Blueprint alone2,000+114.8 ± 21.1
Untracked search2,000+64.4 ± 17.6
Tracked search2,000+9.2 ± 18.6

The best responder against three versions of NexusGTO.

Preflop explorer and drills

The explorer shows the blueprint's preflop strategy for each hand from all six positions, up to eight actions deep. The drills deal a random spot and grade your action by how often the blueprint takes it.