The combat system, as a design

This is the design document for everything the agent decides inside a fight, written for an ML engineer meeting the system cold. The short version: the network never plays a card — a fully-C++ search over a perfect reimplementation of the game engine plays the turn, with the network pricing leaf boards and shaping which branches get visits. One model per encounter, one risk-aware selection rule, and a release discipline whose core invariant is that no fight the system ever learned to win may be silently unlearned.

What makes this hard, in ML terms

A combat turn is a sequential decision under imperfect information: your own draw order is hidden, enemy moves roll from a known finite-state machine, and the within-turn action space (card × target × potion × end-turn, over orderings) branches combinatorially. The live budget is 1,200 simulations per decision, a few seconds of wall clock. Three properties shape the design:

• We own a perfect simulator. The engine is reimplemented in C++, deterministic under seed control, fast enough to be the search's transition function. Anything it can compute — legality, modifier-applied damage, lethal availability — is computed and fed to the network as input, never learned.
• Combat dynamics are encounter-specific. A single global net kept a family of flat-value misplays no search-side configuration could fix; per-encounter specialists could, and the champion was retired outright.
• The failure mode that matters is rare and terminal. A misplay costs HP; a mispriced death ends the run — never average away the low tail.

A serving loop and a learning loop, meeting at the model file

3.1 Per-encounter specialists, and why the champion died

Live play resolves one model file per (encounter, character) — 63 deployed Ironclad specialists today — at combat start (also the hot-reload point), and a missing specialist raises, naming the encounter: no silent fallback, no character borrowing another's nets. The champion net that preceded them was retired because its worst defects — over-blocking with no damage incoming, mispricing decisive plays — had no global fix: two compensating search-side gates were built and rejected on measurement, because the death signal saturates to 1.0 on every threatened boss board, leaving a gate nothing to discriminate with; only per-encounter retraining has moved these failures.

3.2 The state is a set of entities, and the engine does the arithmetic

The board is tokenized as entities, not a sequence: player, each monster with its intent and rolled next-move distribution, each card instance per zone (Strike ≠ Strike+; copies are distinct), each relic with its live counter, potions, orbs, statuses as weighted embedding sums. Most load-bearing: playmath — modifier-applied damage, block, lethal availability — is engine-computed and attached to card tokens; a fleet once shipped for weeks with playmath gated off and the result was systematic strength-blindness. Transformers are bad at arithmetic, so they are never asked to do it. One C++ tokenizer serves live search and training alike, parity-tested at max abs diff 0.0 — the net trains on states byte-identical to the ones it sees live.

3.3 The network: a small transformer with a distributional value head

Each specialist is a 4-layer, 4-head transformer at d=128 — deliberately small, because it runs inside a 1,200-forward search budget. The heads:

• Value = a 21-bin categorical distribution over end-of-fight outcome, bin 0 = death; leaf value is its mean, P(death) is bin 0's mass. A per-checkpoint temperature, fit on held-out data at save time, calibrates measured overconfidence (one audit: predicted 0.03 vs realized 0.26 death).
• 16 monotone survivor end-HP quantiles, plus aux heads — incoming damage, block sufficiency, near-term death, 0–3-turn forecasts — supervised by real teacher rollouts.
• A frozen-anchor residual value head: the deployed distribution is anchor + α·correction (α=0.9 live), so fine-tunes train only the correction.
• A trunk-decoupled encounter head pricing P(death | deck, entry HP) for the out-of-combat planner — decoupled precisely so refreshing it can never move combat play.

3.4 The search: determinized ISMCTS with CVaR selection

The searcher explores the within-turn action graph over the real engine. Hidden information is handled by honest determinization: the unknown draw-pile remainder is reshuffled — the search never peeks at the true RNG stream — and end-turn edges open explicit chance nodes, each child a fresh re-determinization whose random-number salt depends only on (world, depth), never the path, so no line can shop for a lucky world. The net value is the leaf: rollout grounding is 5–15× slower and effectively hangs on multi-enemy boards, so it cannot run live — and everything downstream (training, gating, validation) evaluates on that configuration. Root selection is lexicographic CVaR: a death gate — P(death) read from the net's own death mass, because rollout death rates saturate flat at 1.0 exactly where a gate matters — builds a safe set at a 0.20 risk threshold, then a robust max-Q pick among well-visited children. A symbolic turn planner adds macro edges at the root — whole proven lines folded to their first action — and dominance folding prunes actions no deterministic line can justify.

A teacher five times stronger than the student's budget

Training labels come from self-play in which the same searcher runs at 6,000 simulations with 8 determinizations per decision — 5× the live budget, marginalized over worlds so no label ever encodes one seed's future. Fights start from a battery: a frozen set of entry states spanning deck-strength and entry-HP variation, versioned in object storage and pulled onto whichever training box needs it — never regenerated, so every retrain sees the same distribution and results stay comparable across the fleet. One honest caveat: the collected policy labels are one-hot in 100% of rows — no soft visit distribution — which matters for the pathology in §5.

Anchored fine-tunes; pins as decisive-state reweighting

The canonical per-encounter trainer runs collect → train → gate in rounds. The important lesson is when not to run it: a full retrain re-exposes the value function to the whole self-play distribution, and on guard-heavy encounters this measurably drifts fragile, hard-won behaviors off their optimum — proven five times on one boss. The non-perturbing alternative is the surgical pin: one fine-tune from the deployed anchor with the trunk frozen and the value and policy losses zeroed, trained only on synthesized rehearsal pins — ~60 state variants around one decision, contrasting the correct play against the observed misplay. Fragile encounters are flagged in a committed config and always re-encode pin-only.

Why pins work is now measured, and it is not the original theory. Auditing the corpus for a targeting defect: the alternative branch is present, thousands of times — "self-play never explores it" is refuted — but at the decisive board configuration the one-hot labels split ~8:1 against the correct target (675 vs 81 teacher moves), plausibly holding the policy prior wrong and starving the value head of correct-branch outcomes. A pin is therefore reweighting the decisive-state contrast, not injecting a missing branch. Pins push a region, not a point: each fix is checked against neighboring guards.

The ratchet: won fights may never be unlearned

The unit of evaluation is a case: a frozen reproduction of a real board — relics and potions included, they are encoder inputs — with a machine-checkable statement of the correct play. The suite holds ~450 cases: guards (behaviors the deployed lineage has passed and must keep) and xfails (known-open misplays a candidate is invited to fix). Promotion is decided only by the full suite at the live budget of 1,200 simulations, net-value leaf, the exact live selection config — a cheaper 200-sim screen measurably disagrees and once would have shipped a guard regression. The accept rule is the ratchet: pass every committed ratchet case (373 today, 43 quarantined with written reasons) and fix at least one case not yet in it; breaking any ratchet case rejects regardless of net total. A passing candidate is staged, never auto-deployed. On deploy the ratchet is blessed: the fixed xfail becomes a guard, so any future retrain that drifts it is rejected at the gate — which is what makes fixes durable. Subtle search-config changes get their own gate: a repeated per-encounter OFF/ON A/B counting only consistent flips — the suite is measured deterministic, so "that flip was noise" is inadmissible.

The shadow engine, and reading the tree before blaming it

Live play maintains a shadow state: the C++ engine stepped in lockstep with the real game, a divergence checker comparing predicted against observed after every action, with expected divergence classes (a relic that hides enemy intents) auto-filed so the actionable stream stays clean. The same discipline applies to post-mortems: before a live death becomes a case or a pin, the decision log is checked for the search-divergence tell — if the tree already ranked the correct play highly and the selection layer ejected it, the defect is valuation, not the policy prior, and a pin would treat the wrong organ. Observability gaps count as defects too: a missing potion field once silently invalidated two investigations.

What is measured to be wrong, and the boss-fight interlock

• Flat Q on decisive plays. Engine-verified on two encounters: a 40-damage play, a 9-damage play, and doing nothing priced within q=0.021 with end-turn on top; elsewhere the correct target loses by 0.0165 on a line worth 6 real damage. Root-vote dumps settle attribution — the correct action is unpruned and visited; the value ranking itself is the defect.
• The prescribed fix is 0-for-2. Value retrains on both flat-Q encounters produced no separation, and one moved the value backwards (gap 0.0165 → 0.0790) while the aggregate probe read "no change" — aggregate gates can score a specific defect worsening as neutral, which is why the case suite exists. The one-hot / 8:1 label pathology (§5) is the best current explanation for the stagnation; the backwards movement remains unexplained.
• Structural gaps. Permanent max-HP gains, gold, lesson accrual, "this line is an infinite" — properties of sequences or the run, priced below the value net's noise floor by design. The symbolic planner exists because of them, and the remaining open misplays cluster on exactly those axes.

The interlock: the trunk-decoupled encounter heads price fight danger for the path planner, campfires, and card-reward valuation, so the models that play the fights also inform which fights get taken and with what deck. The joint exam is the act boss: deck quality is the out-of-combat system's job, play quality is this one's, and the run is decided where they compound.