The out-of-combat system, as a design

This is the design document for everything the agent decides outside a fight: drafting cards, routing the map, resting or upgrading, shopping, resolving events, managing potions and keys. It is written for an ML engineer meeting the system cold. The short version: a run is a single sequential decision process with a sparse, run-terminal reward, and the architecture is what survived measuring why the obvious approaches fail on it — exact enumeration wherever the game's structure permits exactness, one shared value function trained on causally paired outcomes for everything else, and a release pipeline that refuses any model that wins one axis by breaking another.

What makes this hard, in ML terms

A Slay the Spire run is ~50–100 out-of-combat decisions across roughly fifteen distinct surfaces, interleaved with combats, ending in one binary outcome. Three properties shape every design choice below:

• The reward is a property of the run, not the state. Every state inside a run shares the run's outcome. Naive value regression collapses to an effective sample size of runs, and the only per-state signal it finds is "how far along am I" — measured: a one-scalar lookup on floor beat a trained regressor on held-out data.
• Dynamics variance swamps single decisions. Two futures branched from the same state, differing by one card, correlate only +0.47 in outcome and reach the same final floor 48.6% of the time. Any label built from unpaired outcomes inherits that variance as noise.
• We own a perfect simulator. The engine is reimplemented in C++ with Python bindings, deterministic under seed control, fast enough to play full runs by the thousand. This is the asset the architecture is built around: when you can enumerate or replay the truth, never learn it.

Those three facts sort every decision surface into one of two regimes: surfaces where the engine's structure permits exact computation, and surfaces that need a learned judgment — and they dictate that the learned judgment be trained on within-run counterfactual differences, never absolute outcomes.

Two loops around one value function

The serving loop is deliberately boring: enumerate candidates the engine says are legal, price each one as a counterfactual edit through one value function, take the argmax (or run exact dynamic programming where the surface is a known DAG). All the interesting machinery lives in how that value function is fed, trained, and released — and in the detectors that make its failure modes loud.

3.1 Candidate enumeration is engine truth

Every surface's option set comes from the engine's own legality rules, and an engine parity oracle — the engine's internal getAllActionsInState exposed to Python — plus a committed sweep test pins the binding layer to it. This is not pedantry: the enumerator once silently offered campfire options the player lacked the relics for, and every corpus collected through it described a bot receiving free rewards. Options that resolve ambiguously (event buttons, follow-on card grids) are grounded the same way: clone the state, apply the option against the real engine, read what actually happened. Grid semantics (is this select an obtain or a removal?) come from the engine's own screen type where visible, and from engine-source-verified per-event tables where not — with any unclassified label self-reporting to a divergence log rather than falling through silently.

3.2 One value function, five consumers

A single leaf network prices every out-of-combat state: inputs are the deck as a bag of (card, upgraded) pairs plus static per-card features, relics, potions, HP/max-HP, gold, floor, keys and act-boss identity; output is a calibrated distribution over final floor reached, with the win as its terminal bucket — a scale chosen because every run calibrates it (a win-probability head has no positives to calibrate against at this ascension, and saturates exactly where decisions live), and because any consumer can read the functional it needs from it: P(floor ≥ 17) is the act-boss pass probability, E[floor] ranks candidates, and P(win) is the top bucket. Draft, removal, shop, campfire, event and path decisions all price their candidates as counterfactual state edits through this one function — V(state + edit) − V(state) — which buys two invariants for free: units are commensurable across surfaces (a removal can be compared to a purchase), and reward/removal can never disagree (a card worth taking is definitionally not the first card to cut). Null options — skip, leave, don't-buy — are ordinary candidates, priced by the same function; the only hardcodes permitted are engine-fact dominance sentinels (e.g. a zero-heal rest does strictly nothing), never value judgments.

3.3 Per-encounter danger heads

Combat danger — P(death | this deck, this entry HP, this encounter) — is not the leaf's job. It is learned by small heads attached to each encounter's combat specialist, trained on fight outcomes, and deliberately decoupled from the combat trunk so refreshing a head can never move combat play. Their training batteries sample a grid over entry-HP bands × deck strength — the original low-HP-only design taught heads exactly one rule ("low HP ⇒ death") and left them below chance at ranking real deaths, which is the canonical example of augmentation design mattering more than model capacity. The heads feed the path planner's danger pricing and the campfire's survival weighting.

3.4 Exact planning where structure permits

The act map is a small known DAG: route value is computed by exact dynamic programming over nodes × HP-buckets, integrating (never sampling) hallway-fight damage distributions and elite danger from the heads, with key-collection feasibility applied lexicographically as game logic. Memory-game events are belief-state enumeration. The rule is uniform: if the game's structure makes a computation exact, the learned components supply only the leaf costs, and the planning is arithmetic.

Counterfactual pairs, because everything cheaper measurably fails

Three label designs were run to completion and refuted before this one: absolute regression on run outcome (learns the floor counter), graded milestone regression (same confound, loses to a one-scalar lookup), and observational matched pairs — twins mined from different runs matched on observables ("relabels the confound rather than removing it": held-out sign base rates swung 0.41→0.67 across splits). What remains is the expensive, correct thing:

sequenceDiagram
  participant R as Base run (seed s)
  participant B as Branch point (contested draft)
  participant A as Arm A: take the card
  participant K as Arm B: skip it
  R->>B: clone state at the decision
  B->>A: continue base run (free)
  B->>K: replay the alternative to run end
  A-->>B: outcome_A (floor / milestone / boss passage)
  K-->>B: outcome_B
  B->>B: label = outcome_A − outcome_B
  

Same seed, same map, same deck history, same RNG stream — the only difference between the two futures is the one decision, so the outcome difference is caused by it. Labels come in three grains (raw floor difference, graded milestone coordinate, and act-boss passage — the last optimizing the operator's stated goal directly). Roughly half of all pairs tie (both futures reach the same milestone) and carry zero signal; corpus accounting is always in non-tied pairs. Take-vs-skip pairs share the null arm as a common reference and pool cheaply; option-vs-option pairs are costlier per bit but supply the within-offer ordering constraints a skip-referenced corpus structurally lacks. Both kinds flow through one collector and one schema, and the same collector generalizes to every declinable surface — campfire, shop, removal, events — by decorating that surface's decision seam.

Anchored contrastive fine-tuning

The trainer freezes the leaf's trunk, embeddings and attention — history: trunk movement once doubled the model's blind spots — and trains only the three static-feature projection matrices. The loss has two terms:

• A magnitude-weighted sign hinge on pair deltas: for each pair, the model's marginal V(state+option) − V(state+reference) must agree in direction with the measured causal advantage, weighted by how much the outcome actually swung.
• A distillation anchor: MSE pinning base-state predictions to the frozen warm-start's values. This is load-bearing, not regularization garnish. Measured without it: the pairwise metric improves while the leaf's absolute scale saturates and its card ordering inverts (attacks pricing below curses) — better on one axis, broken as a whole model. The anchor-weight sweep defines the release frontier: a candidate must hold its pairwise gains at an anchor strong enough to pass the absolute-value probe.

The trainer carries its own honesty machinery: warm-start verified bit-identical to the deployed leaf, an assertion that the loss actually reaches every declared-trainable tensor (dead-gradient channels once trained "successfully" for hours), refusal of configurations that collapse to a constant on sister splits, and staging-only output — it is structurally incapable of writing a deployable filename.

The gate ladder, and hardcodes as kill-gates

No candidate deploys on its training metrics. The ladder, in cost order:

• Delta probe (seconds): fixed boards, known-good orderings — premium cards above commons above curses, absolute scale sane, the specific mined blunders flipped.
• Case suite (minutes): frozen live-blunder reproductions replayed through the exact serving stack, with a ratchet — any case that ever passed on a deployed model may never regress, and fixed cases are promoted to permanent guards.
• Paired-seed A/B (the decision): hundreds of full runs, same seeds both arms, scored on the goal metric (act-boss pass rate) with a paired McNemar test — not on averages that can hide a regression behind a rescue.
• Kill-gate baselines: every learned candidate races the dumbest policy that could do its job (e.g. "always take a card early"). If the hardcode ties the model, the less-complex thing ships — hardcoded policies exist here as baselines and kill criteria, never as the product.

Deployment itself is boring by design: staged file, one-copy revert path, blessing into the anchor chain so future retrains fine-tune from what is actually live — a lineage mismatch here once meant every retrain quietly anchored from an inverted model.

Silence must be meaningful

The system's worst historical bugs were silent-wrong, not loud-wrong: an enumerator offering illegal options for weeks, an event grid duplicating a curse because a score annotation was routed as a button label. The design answer is a small set of always-on detectors, each turning a silent failure class into a log line: the bindings↔engine parity test (enumeration can't drift), the grid-label fall-through log (no unclassified event screen is ever resolved without a trace), the combat shadow's predicted-vs-observed divergence records (with expected-divergence classes auto-filed so the actionable stream stays clean), and process-level health checks on the live loop. A quiet log is evidence, not absence of evidence.

End state: a self-improving loop with gates in it

The steady state this design converges to: the deployed policy plays; the collector branches its contested decisions and plays out both futures; the corpus grows exactly where the current policy is uncertain; the anchored trainer produces a candidate; the ladder promotes it or kills it; the improved policy plays, shifting the contested band to harder decisions. Each pass compounds — and every pass is gated, so the loop can only ratchet upward. The same loop then widens: branch points extended into act 2, the danger heads refreshed act by act, potion-economy labels riding the same corpus.

Where the build stands today: the serving loop, enumeration truth, danger heads for all act-1 encounters, the collector, the anchored trainer, and the full gate ladder are built and battle-tested; the shipped draft policy (a learned imitation prior blended over the shared value function, with a floor-graded null anchor) moved the act-1 boss pass rate from ~35% to ~50% observed. The flywheel's first ignition waits on corpus scale — the measured curve says a few thousand non-tied pairs — with every threshold, command and acceptance criterion written down. The combat model is the sibling program: the two meet at the boss fight, where deck quality (this system's job) and play quality (the combat system's job) jointly decide the run.