How NexusGTO was built and tested

The game is six-player no-limit hold'em with blinds of 50 and 100 chips, 10,000-chip stacks (100 big blinds) and no rake. The blueprint gives a probability for each action in each spot.

The full game has too many information sets to store one by one, so similar hands share a bucket and get the same strategy. Preflop, hands fall into 169 classes that lose no information, but on each later street they're grouped into 200 buckets. On the flop, a precomputed table assigns each hand on each flop texture to one of the 200 buckets.

Bucketing is one half of the abstraction, the simplified game the blueprint is trained in. The other half is a short menu of bet sizes. Before the flop, the first two raises can be 0.7 or 1.0 times the pot, and the third and last can only be pot-sized. So the first player to act, under the gun (UTG), opens to 2.8 or 3.5 big blinds. After the flop there's no choice of size: the first bet is 0.75 pot and every raise after it is the size of the pot. The menu has no explicit all-in, so a player goes all-in only when a size on the menu is at least what they have left.

The search is a C library with OpenMP, which the Python game engine and evaluation harness call. Inside a search, each two-card combo gets its own strategy instead of sharing a bucket. The engine stays with the blueprint when four or more players are left. Four-way spots come up only about once in 254 hands, so searching them would change little, and a five-way flop search took 38 to 76 s against the 3-second budget. Even after the speedups, one benchmark solve of a six-way flop (250,000 iterations on 32 vCPUs) took 13.4 s.

A three-player flop or turn search is depth-limited. Its tree stops at the next card, and rollouts value each leaf there under four continuation strategies, the blueprint and three versions of it biased by a factor of 5 toward folding, calling or raising. With two players left, the search runs in-tree through the river instead, with no leaves. On a heads-up flop that means sampling the cards to come, 8 turn cards and 6 river cards for each turn card. Three-player flop and turn trees allow at most two raises, and heads-up trees and rivers allow three.

To build an opponent's range, the engine starts with every two-card combo equally likely, as dealt, then weights each combo by how likely the blueprint was to play it the way that opponent has played the hand. For NexusGTO's own range, the untracked search assumes NexusGTO played the blueprint too, while the tracked search uses the strategies NexusGTO's searches actually played earlier in the hand.

Across 1,200 turn and river spots taken from the blueprint's own games for the decision audit, heads-up searches reached medians of 98,000 to 100,000 iterations in their 3 seconds, at or near the ceiling of 100,000. Three-player searches got through fewer in the same time, a median of about 70,000 on the river and only about 13,000 on the turn.

The 3-second deadline stopped only the CFR iterations. Setting up each search happened outside it, so wall times ran past 3 seconds. In the tracked search's 14,250-deal run, the median search took 3.01 s and the 95th-percentile search 3.47 s. The run had 19 jobs, one per seed, and each figure is the median of the 19 jobs' own values. The slowest search took 12.1 s.

The engine fell back to the blueprint in 907 of that run's 38,451 solves (2.36%). In 417 of them every player left in the hand was already all-in, but the engine still had to act, with a check as its only legal action. Another 310 were three-player spots where the game allowed a third raise and the search tree didn't. About 120 of the rest were spots with a real choice where the search returned an exactly uniform strategy, and the Results page lists every cause.

NexusGTO takes Pluribus, Brown and Sandholm's six-player program (Science 2019), as its design target but uses fewer bet sizes on purpose. Pluribus searches with one to six raise sizes, while NexusGTO's search has only the blueprint's one size for each bet or raise.

Pluribus re-solves only at the start of a round or after an opponent's off-menu action, a bet size outside its abstraction. When it re-solves mid-round, it keeps the strategy it has already played for the hand it actually holds. NexusGTO's search re-solves every time it is to act. It holds its own earlier actions in the round fixed only after an opponent's off-menu action, and otherwise a new solve can change the strategy behind actions it has already taken. Table 1 compares the two programs.

The trainer uses counterfactual regret minimization (CFR). It plays the game against itself, and in each information set it keeps a running total of how much better each action would have done than the strategy it played. The running total is the action's regret. MCCFR samples the tree instead of walking all of it. In the external-sampling form the trainer uses, one player, the traverser, tries every action, while the cards and the other players' actions are sampled.

An iteration is one traversal for one player, and the traverser rotates through the seats. So 8.05 billion iterations are about 1.34 billion six-player sweeps. The Pluribus paper gives no iteration count to compare this with.

OpenMP threads update one shared regret table without locks, and each thread has its own random number generator. The table uses open addressing with linear probing. A compare-and-swap on a per-slot flag claims a slot, and lookups take no lock. The table's size had been a product of powers of 2 and 5, so keys collided in long runs and the longest probe chain reached the cap of 4,096 slots. From April 29 the table used prime sizes, which cut the longest chain to about 37.

Linear CFR discounts earlier iterations so that later ones count for more. The trainer applies it in about 40 periodic ticks over the first 3.5% of iterations. Pluribus discounted over almost the same share of its training, the first 400 of 11,520 minutes, or 3.47%. The threading bug that broke Linear CFR in the June solver tests was in how the search solver weighted each iteration, so it didn't affect the trainer, which discounts only at those ticks.

Pruning starts once 1.7% of training is done. From then on, the trainer skips actions with regret below −300 million in 95% of iterations, except on the river and for fold.

The run started from scratch on May 12 and finished on May 23, 249 hours later. It ran on AWS spot instances, and interruptions split it into about ten processes on eight instance types with 96 to 192 vCPUs each. In all, the processes used 195.5 instance-hours, about 34,000 vCPU-hours, at a cost of about $440.

Information sets grew from 289 million at 100 million iterations to 993.8 million at 8 billion. The run stopped at 8.05 billion iterations.

Pluribus's blueprint took 12,400 core-hours. Counted the same way, in physical cores, this run took about 26,500. The two don't compare directly, though, because this run had spot restarts, a memory-bound table and a coarser abstraction, and it counts an iteration differently.

On March 27, the snapshot interval in the training config came to 2.35 billion, too big for a 32-bit integer, and wrapped to −1,941,949,696. Training crashed at about 14 million iterations. The fix capped every config value at the 32-bit limit.

The second bug raised no error. A postflop bucket already accounts for the board, but the information-set key held the board as well, so every board made its own information sets, and the tree grew to billions of them. The 1.34-billion-slot table filled at 80 million iterations, 0.17% of the way into the 46.5-billion-iteration run planned then. A miss counter added during the diagnosis showed that from then on 27% of lookups returned zero. For trash hands the true values were negative, so every zero overstated what a raise was worth. The fix on April 4 took the board out of the key.

The same day, tiered bet sizes before the flop (fewer sizes at each later raise) cut the preflop tree 19-fold, from 43.8 million nodes to 2.28 million.

The third was a race in the lock-free writes. In the 200-million-iteration checkpoint, hands like 22 under the gun had exactly zero regret. A small C scanner streamed the 26.4 GB checkpoint and found 38 duplicate root entries, all for UTG, with regrets split between the copies. It also found 19.3 million all-zero entries among 34.2 million preflop entries. The fix on April 6 made threads wait for a slot another thread was still writing, and it merged duplicate keys when loading old checkpoints.

The fourth was in the flop texture table. On May 10, a diagnostic looked up three known flops in the stored table, and all three missed under the trainer's hash function but hit under a different one. The table had been built with that other function, so every flop lookup in training had missed. For about 16 days, from April 24 to May 10, the trainer keyed flop information sets by exact hand (1,326 ways) instead of by 200 buckets.

That afternoon I first chose to retrain from scratch, then changed my mind within the hour and kept the 5.21-billion-iteration blueprint. It came from the run begun on April 24, as did the blueprint before it, so the flop lookups of both had missed throughout training. That evening the newer blueprint passed only 2 of 12 preflop sanity positions, spots across all six seats where a check compares the strategy with the shape it should have. The blueprint before it had passed 9 of 12, so I dropped the newer one.

The from-scratch run started two days later with a new preflight check. The check looks up three test flops, one rainbow, one paired and one monotone, and exits before training unless each flop lands in one of the 200 buckets.

Under the gun, the blueprint folds 81.0% of the 1,326 two-card combos and limps (calls the big blind without raising) with 9.7%. It raises to 2.8bb with 6.6% and to 3.5bb with 2.7%, and Fig. 2 breaks the strategy down by hand.

Aces limp 35.8% of the time, raise to 2.8bb 42.9% and raise to 3.5bb 21.3%.

Pluribus learned not to open-limp, as its paper reports. Why this blueprint does is an open question. One check would compare AA's regret for limping with its regret for raising, at the root or at an earlier snapshot, but I haven't run it.

The preflop table extracted from the blueprint has 16,306 decision points across all six positions, every line up to eight actions deep, and stores frequencies in steps of 1/255. It holds 2,755,714 cells, one for each of the 169 hand classes at each of those points, and 121,359 of them (4.4%) sit within one step of an even split across the actions. Training rarely sampled those spots, so the explorer marks them.

In early April the preflop chart showed every cell except AA with another hand's strategy, because the extractor that writes the table numbered hands differently from the trainer. AKs, for example, showed A7s's 100% fold. A browser test had shown that the chart rendered, but nothing checked whether its ranges made poker sense. So the April 7 fix added a check of eight hands at the under-the-gun root, four that must never fold and four that must always fold. The extractor refuses to write the file if two or more of them fail.

The explorer opens the table at the UTG root, and its data panel lists the file's format and SHA-256. The panel's download is the whole table as one 10.3 MB JSON file, under a CC BY 4.0 ↗ license.

A test on March 24 ran outcome-sampling MCCFR on an RTX 3060, with six players and ten equity buckets, at 17 million trajectories per second. Per wall-clock hour, though, the GPU run converged about 5 to 10 times more slowly than external sampling on the CPU, because outcome sampling's estimates are noisier. Training stayed on the CPU.

Search went the other way at first. A solver for up to six players, from the same day, handled one street at a time on the GPU, in CUDA. With 200-hand ranges and 200 iterations it took 67 ms on the river and 236 ms on the flop. On May 9 I decided to keep search on the GPU, because the only CPU search solver then was a preflop one, over the 169 hand classes with no card removal. Searching on the CPU would have meant writing a new solver over individual two-card hands.

A benchmark on June 1 led me to reverse that call. It timed the GPU solver against a new CPU solver that uses external sampling, on one desktop with an RTX 3060 and an i7-13700K (24 logical cores). The GPU solver's vector CFR updates every hand in each range on every iteration, and its opponent-reach step grows with the square of the hand count. With 180-hand ranges it took 1.89 ms per iteration at two players and 2,049 ms at six, and from three players on, each added player made it slower. At the GPU solver's budget of 1,600 iterations, those times project to 3.0 seconds at two players and 54.6 minutes at six.

External sampling draws one hand for each player on every pass, so a pass costs the same however many hands the ranges hold. On one thread it ran 708,000 traversals per second at two players and 197,000 at six.

At the time I still planned to search with five and six players, as Pluribus did, so the fair test was how long the CPU took to match the GPU's exploitability at five players. Exploitability here is the sum of what each player's best response could gain against the others. The GPU took 599 seconds for its 1,600 iterations and reached an exploitability of 12.69 chips in a pot of 2,000. For this test the CPU solver ran Linear CFR, as the GPU solver did. On 24 threads it got below the GPU's figure, to 11.20, in 414 seconds, about 1.4 times faster.

The CPU won only after a bug fix. Its first Linear CFR runs did no better than vanilla CFR, which weights every iteration equally and had lost to the GPU at five players. A comparison of one thread with 24 found the cause. At three players and 1 million iterations, Linear CFR reached an exploitability of 11.11 on one thread but 151.72 on 24, against 48.49 for vanilla CFR. Under OpenMP's schedule(static), each thread took a fixed block of iterations. The threads holding late blocks stamped large iteration weights onto the shared strategy sum from the start of the run and broke Linear CFR.

The fix switched to schedule(dynamic), which makes the weight follow the progress of the whole run. The same test then gave 11.11 on one thread and 9.87 on 24. On the same chip, a five-player run of 10 million iterations took 204 seconds on one thread and 11.9 seconds on 24, 17 times faster.

Five-player search could have stayed on the GPU only with an exact opponent-reach kernel for four opponents, and a check on June 2 found that the kernel needed a 549 KB table against a 48 KB shared-memory budget. From June the two- and three-player searches ran on the CPU too.

On July 1 I made Pluribus the design target in everything except bet sizes. Pluribus searched with any number of players, so the speed target became a six-way flop solve inside 33 seconds, the longest a Pluribus search took. A benchmark timed two changes to the solver on a synthetic six-way flop on AhKsTd, run for 250,000 iterations. Each run used all 32 vCPUs of a c7a.8xlarge from a cold start, with two seeds for each version of the code.

Before either change, a July 28 profile of a two-player, 150,000-iteration solve put 86.3% of cycles in five-card hand evaluation. The solver calls it to map turn and river hands to buckets. A six-player profile was lost when perf's output filled the disk. Since the code path is the same for any player count, the two-player profile stood in for it.

Both changes went in the next day. The first caches turn and river bucket IDs for the length of a solve. In that profile, recomputing turn buckets had taken 63.6% of cycles, and the cache cut the benchmark from 1,324.8 seconds (about 22 minutes) to 52.2, 25.4 times faster.

The second change is a table of all 2,598,960 five-card hands, built when the library loads. Its benchmark ran on a second machine, where the cached code took 59.4 seconds against 52.2 on the first, and the table cut that to 13.4 seconds, 4.4 times faster. Across the two machines the benchmark fell from 1,324.8 seconds to 13.4, about 99 times faster, and the two same-machine speedups multiply to about 112. The six-way solve then met the 33-second target, but a search in the evaluation matches gets 3 seconds on 8 threads, so NexusGTO still plays the blueprint in six-way spots (How it works).

The commit that added the table also keeps the bucket cache warm when one engine runs several solves. The benchmark builds a fresh solver for every run, so all its times are cold, but the match harness holds one engine for a whole match, and from the second search in a hand the cache starts warm. In one process, two identical two-player solves of 20,000 iterations on one thread took 46.24 seconds cold and 0.43 warm, with the same output. Those solves ran to the end with no deadline, while a search in the evaluation matches runs on 8 threads and stops at its deadline, so neither time carries over to the matches.

Both changes only remove repeated work, so neither should change the output. I kept each one only after two-player solves with a fixed seed on one thread gave the same SHA-256 of the full strategy before and after the change. An exhaustive test also matched the table against the old evaluator on every five-card hand. No check covered the multithreaded six-player runs.

Two tests measure NexusGTO. Chips won against fixed opponents don't measure exploitability, so the first test asks how much a program built to exploit NexusGTO can win from it. The second asks whether the search beats five copies of the blueprint.

The best responder is a local best responder (LBR), a method from Lisý and Bowling (2017). It tracks the other player's range and picks whichever action scores best one step ahead. The best responder plays only the heads-up slice, where the other four seats fold and the small blind plays the big blind. Each deal is played twice, with the seats swapped.

The best responder's results are lower bounds, because a full best response could only take more. It also bets only the abstraction's sizes, so its results say nothing about bets off that menu. A policy that folds every hand would lose 75 bb/100 in this slice. The 75 comes from arithmetic rather than a run, since a folder gives up its small blind in one seat and, whenever the best responder raises, its big blind in the other. On one PC, a deal takes about 1.7 seconds against the blueprint. Against the search it takes about 30, because the search needs a fresh 3-second re-solve for every move, including the ones the best responder only looks ahead to.

Six-player results are noisy, and the second test cuts the noise with duplicate deals. The harness deals each deck six times, and the tested player takes each seat once against five copies of the blueprint. Because the harness keys its random seeds on the deal and the seat, the six rotations share their luck. The deal is the sample unit, so 14,250 deals make 85,500 hands but count as 14,250 samples.

In the blueprint field, all five opponents are copies of the blueprint. They play exactly what the search assumes, so this field mainly catches bugs. A search can beat it and still lose to opponents who play differently.

Two deviating fields change the blueprint on purpose. In the nit field, a player facing a bet moves half of its call and raise mass to fold. With no bet to face, it moves half of its bet mass to check. In the maniac field, half of the check and call mass moves to raises, and 30% of fold mass moves to call. The amounts are arbitrary stress settings. They aren't models of people.

The score is the tested player's net chips, in bb/100, and the code asserts that it is exactly 0 with the blueprint in every seat. In a deviating field, a score relative to the blueprint is the tested player's score minus what the blueprint scores in its place on the same deals. Each result carries ± one standard error, and a 95% interval is the value ± 1.96 standard errors.

Offline, every strategy and every card is known, so the harness also runs AIVAT variance reduction. AIVAT computes the luck in each sampled action and in each turn and river card, and subtracts it. The flop cards get no correction term. AIVAT still applies in the deviating fields, because each changed mix is known exactly.

To check AIVAT for bias, the harness played 20,000 deals with a blueprint that picks a random action 5% of the time. AIVAT cut the per-deal standard deviation from 2.545 to 1.775 bb per hand, so the same precision needs about half as many deals. The raw estimate was −6.89 ± 1.80 and the AIVAT estimate −7.58 ± 1.26. The corrections alone came to −0.68 ± 1.20, consistent with zero, so the test found no bias from AIVAT.

Before AIVAT, a 3,000-deal test missed the same 5% deviator (+0.46 ± 3.58) but caught a blueprint that played randomly 20% of the time (−41.2 ± 8.2). Two nearly identical configurations scored 2.30 ± 1.37 apart on the same 14,250 deals. The gap is most likely solver noise, so picking the best of many configurations overstates the winner.

The engine's first timed run, playing the blueprint on June 5, crashed 688 times in the middle of hands. Its game state didn't track real chip counts, so on lines where much of a stack was already in the pot it offered actions the blueprint had dropped. The fix made the game state track real chips the way the trainer does, instead of clamping the mismatch, and 40,004 decisions then ran with no crash.

A review of the match runner on July 11 found that its seat-differential metric inflated results by a factor of 1.2: a true 2.0 printed as 2.4. The net-chips score replaced it before the first reported match.

The first large batch of matches finished on September 13. The first search configuration scored −3.74 ± 3.22 bb/100 against the blueprint field over 3,000 deals (95% interval −10.06 to +2.58). This configuration isn't the tracked search, and its inputs had defects that turned up later. It built opponent ranges from preflop actions only and started by treating all 169 hand classes as equally likely. Equal weight per class gave each combo of a pair twice the weight of an offsuit combo, and each suited combo three times.

Giving it 10 seconds instead of 3 didn't help. On the same deals it scored 2.28 ± 2.63 bb/100 lower, a gap smaller than its own standard error. Against the nit field, over 3,000 deals, it scored −7.31 ± 1.78 bb/100 relative to the blueprint (95% interval −10.79 to −3.82).

It almost never bet where its solve stopped at leaf values (Fig. 4). Against the nit field it bet on 4.1% of unopened heads-up flops, where the blueprint bet 46.8% of the time, and on 0.5% of unopened three-player turns, against the blueprint's 37.1%. On heads-up turns, which it searched in-tree through the river with no leaves, it bet 38.5% of the time against 41.6%, but on heads-up rivers, also searched with no leaves, it bet 24.7% against 43.2%.

The low rates on flops and three-player turns pointed at the leaf values. On September 14 a series of tests traced them to the search valuing its leaves against only the one opponent hand the solver had sampled on that pass. Drawing the opponent's leaf hands from its range at the start of the street brought betting back to about the blueprint's rates or higher. With that fix the search scored −4.5 ± 3.9 bb/100 against the blueprint field over 3,000 deals, level with the first search configuration. Runs later that night found that its turn and river searches were where it lost chips. It still bet too little on heads-up rivers, where it used no leaves, so on September 16 I checked the ranges. On 599 heads-up river spots, a re-solve given the opponent's true range beat the blueprint's own river play by 2.74 ± 0.33% of the pot per spot. With the preflop-only range the search actually used, it trailed by 0.6%, which isn't significant. Once the flop and turn actions were applied, that assumed river range kept only 32% of its probability mass. Most of the range was hands the opponent's own actions, usually checks, had ruled out or made unlikely. The fixes that day conditioned each player's range on the flop and turn actions and added a combo prior, which starts each range with every two-card combo equally likely, as dealt.

The same day I asked whether matches against copies of the blueprint were the right way to compare search configurations, and started measuring exploitability directly with the best responder.

On September 17 I asked why blueprint-only play could beat search. To find out, a root value audit over 150 heads-up flop spots compared the search's estimate of each root action's value with the action's true value when the blueprint played out the hand. The search built those estimates from the flop leaf values, and the estimates were about half a pot too high for every hand strength. Across spots, the search's bet-minus-check estimates correlated with the true values at r = 0.06. On those spots, where the search acted first, the version with the September 14 fix bet more often than the blueprint did, and its choices did worse than the blueprint's own.

The range fixes of the day before changed how the search played but not its exploitability, because on the flop the leaf values set the strategy. The best responder's runs had also tried tracking the search's own range, with the same result. So the next version of the search, the in-tree flop search, kept the preflop-only ranges, without the range fixes, and changed the flop instead. On heads-up flops it dropped the leaf values and searched in-tree through the river, over sampled turn and river cards. Three-player flops and turns still stop at leaves, and the audit hadn't covered them.

On September 22 I asked whether my bar, a win over the blueprint at 95% confidence, matched the one Pluribus had cleared. Pluribus's bar had been a significant win over professionals, never a win over its own blueprint. At 3,000 deals the standard error, about 3.2 bb/100, was too wide to see an effect of 4 bb/100, so the next test ran 14,250 deals on the blueprint field and brought back the deviating fields.

That test used flop-only search, which searches the flop and leaves the turn and river to the blueprint. It left them out because the in-tree flop search, still on the preflop-only ranges, had scored −7.1 ± 3.8 bb/100 against the blueprint field over 3,000 deals when it also searched the turn and river. Over the 14,250 deals, flop-only search scored +4.11 ± 1.37 bb/100 against the blueprint field (95% interval 1.42 to 6.80).

I still wanted every street searched, as Pluribus did, so the next check was a decision audit of the turn and river. The audit took 1,200 turn and river spots with two or three players from the blueprint's own games and scored each re-solve against the blueprint's mix at the same spot, with every player's true range. The spots come from blueprint play, not from games the search played, so the result is a first-order estimate.

Given every player's true range as input, turn and river search came out +3.51 ± 2.00 bb/100 against the blueprint's own play with the river strategy averaged over iterations, a result whose 95% interval includes zero. With the last iteration's strategy, which is what the engine plays, it came out +4.05 ± 2.02. The true ranges were worth +4.65 ± 1.48 over the ranges the search had been using, and the combo prior alone +3.55 ± 1.13. Table F on the Results page lists the earlier 3,000-deal runs with their range inputs.

So the 14,250-deal every-street runs used the combo prior from September 16 and the range conditioning with the fixes made on September 23, which together give the ranges How it works describes. Own-range tracking went into the matches on September 23, the day those runs started.

In the heads-up slice, the best responder played the blueprint and both searches on the same 2,000 deals. It took 114.8 ± 21.1 bb/100 from the blueprint alone, 64.4 ± 17.6 from the untracked search and 9.2 ± 18.6 from the tracked search. Paired on deal and seat, the differences are 50.4 ± 24.9 between the blueprint and the untracked search (95% interval 1.6 to 99.2) and 55.2 ± 21.8 between the two searches (95% interval 12.4 to 98.0).

Those three figures value each showdown over the whole range the best responder tracks for its opponent, not the cards the opponent held. Counted in chips actually won, the best responder took 152.7 ± 27.9 from the blueprint, more than folding every hand would lose, and 31.7 ± 24.1 from the tracked search. The best responder never played flop-only search. On September 17 it took 85.9 ± 21.4 from the in-tree flop search, which used the preflop-only ranges and also searched the turn and river.

Without tracking, a re-solve assumes the search's own earlier actions came from the blueprint, and it plays as if the search held the blueprint's range. Copies of the blueprint make the same assumption, so they can't exploit the difference. The best responder can, because it tracks the range the search actually played. Pluribus tracked its own range for this reason (Brown and Sandholm, Science 2019).

The tracked search's +5.73 ± 1.53 bb/100 against five blueprint copies is its AIVAT score, 57 milli-big-blinds (mbb) per hand, and without AIVAT it scored +5.14 ± 2.15. Pluribus won its 48 mbb per hand against professionals (Brown and Sandholm, Science 2019), and the tracked search won its 57 against copies of its own blueprint, so the two figures aren't comparable.

The untracked search scored higher against the same field, +8.61 ± 1.72 (95% interval 5.24 to 11.98), but the best responder took 64.4 from it. Paired on those deals, the tracked search scored 2.88 ± 1.62 less than the untracked search, which isn't significant (95% interval −6.05 to 0.30).

The decision rule for the run dates from a commit at 08:42 ET on September 23, before the run started. It has three clauses:

Every-street [is adopted] if its minus-flop-only lower 95% bound on the blueprint field is above −1.5 bb/100, it beats flop-only at 95% on at least one deviating field, and LBR is not worse.

It [is adopted] outright if it beats flop-only on the blueprint field at 95%.

Otherwise flop-only [is adopted] and [the decision-audit cells] say what to fix next.

The plan around the rule said that the run would pair "the best every-street configuration" with the stored flop-only run, the +4.11 from September 22, without saying which configuration. It did list own-range tracking among the range fixes to build, and at 12:35 ET the project's results log named it in the run's configuration. Its code went in at 12:45, about four hours after the rule, and the run launched at 12:49. At 13:02 a fix to the range conditioning landed, and the run's jobs restarted on it.

The rule compares every-street search with that stored run, which wasn't repeated. Against it the tracked search scored +1.62 ± 1.65 (95% interval −1.61 to 4.86), so its lower 95% limit missed the −1.5 margin by 0.11. A matched flop-only run played the same deals alongside the every-street searches, with the untracked search's range inputs, and scored +2.49 ± 1.37. The tracked search beat it by +3.24 ± 1.64 (0.04 to 6.45).

On the nit field, over 6,000 deals, the tracked search beat the stored flop-only run by +2.54 ± 1.13 (0.32 to 4.76), so it met the deviating-field condition. Its +1.92 ± 1.08 against the matched flop-only run isn't significant. It also beat the blueprint there by 2.53 ± 1.09, and no earlier configuration had done so at 95%. On the maniac field it scored +19.9 ± 5.4 relative to the blueprint and +1.3 ± 3.9 relative to the stored flop-only run.

With no best-responder figure for flop-only search, the tracked search met the "LBR is not worse" condition against the 85.9 from the September 17 search instead. The first clause needs all three conditions, though, so the tracked search failed it on the margin alone, and it failed the second clause too.

The untracked search met the "outright" clause, at +4.50 ± 1.79 against the stored flop-only run (0.99 to 8.01), but it joined the run after launch, while the tracked search ran from the start.

Read as written, the rule adopts the untracked search if a configuration added during the run counts, and flop-only search if it doesn't. It never adopts the tracked search. The project's results log counted the untracked search under the second clause, and the configurations tried after the run built on it. These results still lead with the tracked search, because heads-up in the blinds, the best responder took 55.2 bb/100 less from it than from the untracked search when showdowns are valued over the range. In chips actually won, that gap is 30.8 ± 30.8 and isn't significant.

The first search configuration has stored runs on the first 6,000 of the same deals, its 3,000-deal run of September 13 among them. Paired on those 6,000, the untracked search beat it by +6.70 ± 3.13 (95% interval 0.57 to 12.83). The tracked search scored +5.17 ± 2.77 above it, but its interval, −0.26 to 10.60, includes zero. The Results page gives all seven configurations on the 14,250 deals, with their paired differences.

• The main results come from offline self-play against copies of the blueprint, plus a local best responder, at 100bb with no rake.
• All three opponent fields are copies of the blueprint. The nit and maniac fields add scripted deviations of arbitrary size, and none of the fields adapts.
• The 14,250-deal runs on the blueprint field share seeds 9101 to 9119, so they aren't independent replications.
• The blueprint and the search have one postflop size at each point, 0.75 pot for the first bet and pot for every raise after it. Pluribus searched with one to six sizes (Brown and Sandholm, Science 2019).
• The best responder plays only the heads-up slice, so there is no six-player exploitability figure. Its results are lower bounds on exploitability, and since it bets only the menu's sizes, an exploiter free to choose its own could take more. No offline evaluation includes bets outside the menu.
• Bets off the menu came up in an outside test, GTO Wizard's benchmark leaderboard, where play is heads-up with 200-big-blind stacks. NexusGTO's entry there, listed as "Nexus", scored −100.6 bb/100 over 21,210 hands, below the leaderboard's "Always Fold" baseline. The entry played those hands in August, before the searches in the results existed, and pooled several test configurations of an engine trained for six players at 100bb with one postflop bet size. An analysis of those hands found the heaviest losses where the opponent bet a size off that menu. No test measured how much of the rest came from the differences in stack depth and player count.
• NexusGTO searches only once the flop is dealt and two or three players remain. Three-player flop and turn trees allow at most two raises a street, and other trees allow three. In the tracked search's run, 2.36% of solves fell back to the blueprint, some of them in three-player spots where the game allowed a third raise and the search tree didn't.
• The preflop betting tree has 153,891 lines that reach the flop. A sweep probed two hands, QhJh and AsAd, at the first flop decision of each line. It found 24.43% of those decisions clearly non-uniform for at least one of the two hands, 73.27% near-uniform for both and 2.29% missing from the blueprint. The sweep counts each line once, however rarely play reaches it, and its near-uniform share is an upper bound on undertraining, because a correct strategy can be near-uniform too. A second test is looser, since it passes any stored row that isn't exactly uniform, and it counts every blueprint lookup, preflop through river, as often as play reaches it. In 2,000 deals of blueprint self-play, 99.70% of 106,422 lookups passed it.
• On one reference board, a walk through the postflop trees under those 24.43% of lines found 1.9% of flop nodes missing from the blueprint, 19.1% of turn nodes and 43.4% of river nodes. Raise-war branches make up most of the nodes counted, and the counts don't weight a node by how often play reaches it. In heads-up flop subtrees, 0.00% were missing.
• The blueprint opens by limping with 9.7% of combos under the gun, and no test has looked at why (The under-the-gun limp). Near-uniform values fill 4.4% of preflop cells.
• The blueprint comes from one training run, with no second seed and no convergence measurement, and no record says which version of the code trained it.
• The code's test coverage is unmeasured.

• Noam Brown and Tuomas Sandholm. "Superhuman AI for multiplayer poker." Science 365(6456), 885–890, 2019, and its supplementary materials. Page numbers for Pluribus details refer to the supplement.
• Viliam Lisý and Michael Bowling. AAAI Workshop on Computer Poker and Imperfect Information Games, 2017 (arXiv ↗). The local best response method comes from this paper.
• Neil Burch, Martin Schmid, Matej Moravčík, Dustin Morrill and Michael Bowling. "AIVAT: A new variance reduction technique for agent evaluation in imperfect information games." AAAI 2018.
• Numbers about NexusGTO come from the project's results documents and commit history. The GTO Wizard result comes from GTO Wizard's benchmark leaderboard. Pluribus figures come from the paper and its supplement.
• Times are US Eastern.