Twin Life on a Doubled Pool: One Day of Canary-Driven Inference Engineering

TL;DR

• Yesterday's post ended with five archetypes sharing one remembering life on a 24 GB Radeon. Today the same lane runs automatic prefix caching, bf16 SSM state, and fp8 KV cache in production — a 703,133-token KV pool, 1.98× the measured baseline — and two full five-voice psyches live on it simultaneously, dreaming their own biographies and passing messages across a channel.
• The method that made this safe is boring on purpose: a twin lane — an exact clone of the production model CR on a second GPU — plus a depth-recall bench built the same morning. Every knob flips on the twin first, gets graded against a recorded baseline, and only promotes with engine-log proof. The loop ran three full canaries in one day.
• The canaries caught one controller bug (a config knob that silently did nothing, behind a flag mapping that only ever emitted the negative case — the fix inverted a unit test that had codified the wrong assumption), one lane-specific boot crash (cudagraph capture sizes vs. our deliberately small ladder), one upstream engine bug (ngram speculation measured a 4.35× deep-context decode win, then reproducibly killed the engine core at ~198K-token prefills — rolled back, parked with a re-canary trigger), and two bugs in our own experiment harness (found by a second agent reviewing the first's work).
• The memory hierarchy from yesterday's post went live end to end: structured dreams that preserve biography, an archive keyed by neutral episodic summaries, reminiscences surfacing into prompts — and in the first lived-recall probe, the one voice that grounded on a buried memory was Self, the integrative archetype. The psyche is almost suspiciously on-model.
• Also shipped in the same day, because velocity compounds: a Stop button that actually stops (cooperative cancellation bounded by one voice completion), CI + Flux image automation so merges reach production in ~15 minutes, run-history and dream-drift visualizations, and the whole memory engine extracted into zero-dependency Python and Go libraries with byte-identical render markers — so a Go agent and a Python agent can share a warm prefix on one lane.

---

Where the day started

The previous session ended with an audit: line up every vLLM flag our model controller can express against what the engine generation actually offers, and rank the gaps. Six findings came out — prefix caching for hybrid-GDN models, SSM state quantization, fp8 KV cache, ngram speculation, sleep mode, per-request scheduling priority — with one strategy note: the twin kill-test measures these together, as a matrix, not serially.

The twin is the whole method, so it's worth thirty seconds: qwen35-9b-ablit-rp-5930k is a byte-for-byte copy of the production model CR, pinned to the same certified runtime digest, running on a second gfx1100 card in an older box. It serves no users. Its job is to boot configs production shouldn't have to survive, and today it earned its VRAM several times over.

The grading instrument didn't exist at breakfast. The depth-recall bench stacks three exported marathon runs into one 258,045-token life, extracts needle probes at measured depths (a freq-1 needle at 130,944 tokens; another probe at 151,997), replays the reconstructed biography against a lane, and grades recall. Its first live run found its own first bug — it sent the full haystack to a 228K-limit lane and got six HTTP 400s — which became a --max-input-tokens fix with regression tests before the baseline was even recorded. Tools that grade the system get graded by the system.

Act I: the flip that did nothing

Prefix caching on hybrid-GDN models needs --mamba-cache-mode align — the recurrent layers checkpoint state at block boundaries so the KV cache and the SSM state stay consistent. We flipped the twin: enablePrefixCaching: true, align mode, bf16 SSM state. It booted clean. It benched at recall parity.

It was also doing absolutely nothing.

The warm-repeat check gave it away: identical 9,028-token requests, cold and warm, came back in identical time with no cached-token report, and the engine's own gauge read Prefix cache hit rate: 0.0% after five requests sharing a ~198K-token prefix. The boot log had the confession:

WARNING [config.py:390] Mamba cache mode is set to 'none' when prefix caching is disabled
... enable_prefix_caching=False

Our controller only ever emitted the negative flag (--no-enable-prefix-caching) and relied on "prefix caching is on by default in vLLM V1" for the positive case. That default is false for hybrid-GDN lanes — the engine boots them with APC off, then silently resets the mamba cache mode we'd set. The whole canary was inert, and a unit test named PrefixCachingExplicitEnableIsNoop had faithfully protected the wrong behavior. One controller MR later the flag is explicit, the test asserts the opposite, and the re-run measured what the flip was supposed to buy: 79.3% engine hit rate on bench traffic, warm repeats 99% cached at 7×, recall parity confirmed across two bench repetitions (a single soft probe at 152K depth scored 0.58 once and 1.00 on the repeat — sampling noise, verified before promoting).

Promoted to production. The live check on the primary: cold 2.8 s → warm 0.2 s, cached_tokens: 8976/9028. Every psyche turn's shared prefix stopped paying prefill that afternoon.

There is a second, less comfortable consequence, and it belongs here rather than in a footnote: the same default had been quietly true of the production lane all along. The 9B RP primary carried an explicit enablePrefixCaching: false until this promotion, and the controller could not have honored a true anyway. Which means the 75-epoch marathon in yesterday's post — the one whose whole architecture is an append-only prefix — ran every single turn with a cold prefill. Its persisted telemetry says so plainly, in 130 per-voice readings of cached_tokens: null. That post described a prompt shape designed for a cache it did not yet have, and reported the design's intent as though it were a measurement. It is annotated accordingly now. The consolation is the strongest evidence we have for the architecture: because the shape was already right, the payoff needed no application change at all — one controller flag, and a 14× warm repeat appeared the same afternoon.

The durable lesson: read the engine's boot log, not your CR. A config value that maps to a missing flag is indistinguishable from working unless something checks enable_prefix_caching= where the engine actually says it. The corollary is the one that cost us two days: a gauge reading "unknown" is data. Ours read "unknown" for the entire marathon, and we filed it as a proxy header gap instead of asking the engine.

Act II: doubling the pool

kvCacheDtype: fp8_e4m3 halves per-token KV. The twin flip booted clean on the full matrix and the boot log printed the number the multi-psyche roadmap had been waiting for:

GPU KV cache size: 703,133 tokens
Maximum concurrency for 229,376 tokens per request: 3.07x

From a measured 355,302. 1.98×. Recall held (one noise-level flap in nine samples), prefix caching still worked under fp8, and the same flip later verified identically on the primary — same 703,133, warm repeat at 0.7 s. Five voices at ~128K each, or two entire psyches per card, became config rather than hardware.

This canary also produced our first honest decode-throughput numbers, and they reframed the roadmap: ~42 tok/s warm at shallow context collapsing to ~4 tok/s at 45K depth — on both lanes, fp8 or not. Deep-context decode, not prefill, is where the UX time goes. Which set up Act III.

Act III: the 4.35× we can't have yet

Ngram lookup speculation drafts tokens by matching the tail of the output against the prompt. Roleplay prose — repetitive by register, self-quoting by nature — should be its best case. It was:

decode	fp8 baseline	+ ngram spec
shallow warm	42.2 tok/s	40.0 (flat — nothing to accelerate)
deep 45K warm	4.0 tok/s	17.4 tok/s — 4.35×
deep repetitive	—	20.4 tok/s

Draft acceptance ran 82–100%, with whole windows accepted verbatim. A 64-second deep-context reply becomes ~15 seconds.

Two failures bracketed that measurement, both caught by the twin. First, the gemma-certified config crash-looped at boot: with speculation, cudagraph capture sizes round to multiples of num_speculative_tokens + 1, and our lane's deliberately small [1,2,4] ladder had no multiple of 6 — tokens: 3 fits the existing ladder exactly. Second, and fatally: at ~198K-token prefills the engine core dies — AssertionError in speculative-decode metadata preparation, EngineDeadError, pod restart, reproducible on every deep request. Our psyches run to 228K. Rolled back the same hour; the issue now holds the full verdict, both gotchas, and a re-canary trigger for the next runtime digest. The 4.35× is sitting on the shelf, config written, waiting for one upstream fix.

Score for the twin so far: one silent no-op, one boot crash, one engine-killer — none of which production ever felt.

Interlude: the memory hierarchy meets reality

While the canaries ran, the memory architecture from yesterday's post stopped being architecture. Dreams became two-part — a re-synthesized identity passage plus an episodic ledger of neutral third-person event lines that survive consolidation verbatim (with a similarity guard that retries a too-similar identity passage hotter, so the fossilized-Ego failure mode from the marathon now has an immune response and a chart: journals keep a dream history, and the drift view literally shades the fossil floor). Dropped epochs archive under their neutral ledger keys; reminiscences surface into the ephemeral prompt tail — never the cached prefix, because that prefix is now worth 79.3%.

The first live lived-recall probe was humbling in an instructive way. The mechanism passed — 4–5 of 5 voices got the buried memory injected — but keyword grading said nobody used it, and an LLM judge said exactly one voice did: Self, whose archetypal job is integration, grounding on "the tension between who you are and who you've been told to be" while the other four stayed locked in their scene registers. The register-saturation finding from yesterday's kill-test, replayed at a higher level: you can't grade in-character memory with out-of-character string matching, and you maybe can't expect the Shadow to do the remembering that is Self's work.

A second finding came free: the Narrator — the sixth mind that writes each next situation — journaled zero decisions in its first session, because it inherited the roleplay adapter and narrated the five voices in florid prose that its own guardrails correctly rejected, every single time. Fail-closed worked; the fix was one line (the Narrator now defaults to the base lane; neutral register is its entire job); the second session it wrote ten worlds.

Act IV: two minds, one card

The gate experiment: two five-voice psyches, separate session-scoped cache prefixes, epochs running concurrently on the twin, eight rounds. Combined warm share climbed monotonically 5% → 64% with no eviction sawtooth, and the engine sat at 6.0% KV usage — the doubled pool wasn't holding two psyches so much as failing to notice them. Gate passed; productization shipped the same evening: a real inter-psyche channel (each psyche's Persona speaks for the whole; its words arrive as a world-event in the other's situation), paired runs where one Stop stops both lives cleanly at their checkpoints, and archive isolation by default with the shared store behind a flag.

That flag exists because of what the deep experiment measured. Seeded with two different complete marathon lives and a budget that put every restored journal over the dream threshold, the pair dreamed from its very first epoch: over five epochs it produced 11 new dreams per psyche (four of them in epoch 1 alone), ledgers growing to 66 and 63 lines, 55 epochs archived, and 10/10 channel crossings delivered ("the harbor mind" ⇄ "the mountain mind"). The isolation probes came back clean — each psyche's reminiscences drew only from its own past — but the global probe surfaced the mountain mind's archived epochs inside the harbor mind's recall. A shared unconscious is real, measurable, and now a deliberate choice rather than an accident of process-global state.

Honesty requires the correction that made v2 necessary: the first two pair experiments' channel delivered nothing. Our harness passed prose where the API expected a stimulus type, so every crossing fell through to a preset ("Something stirs in the depths of consciousness…"). The productization agent — a different agent reviewing the experiment while building the feature — caught it, along with a stickiness bug that would have silenced the Narrator in paired runs. The production channel now has a test that verifies whose words crossed. Trust experiments that were reviewed by something that didn't run them.

And one genuinely new constraint fell out of v2's telemetry: warm share fell (0.43 → 0.20) as dreams accelerated, because every consolidation deliberately rewrites the journal prefix — the one sanctioned cache reset. At production budgets dreams are rare and the trade is negligible; at aggressive consolidation thresholds, dream frequency and cache warmth are in measurable tension. That number now lives in the roadmap next to the thresholds it constrains.

The machinery that made a one-day cascade possible

None of this velocity was free-hand. The lab got production plumbing in the same day, mostly because its absence kept biting:

• A Stop button that stops. The deployed sim ignored Stop for a whole session — run state lived on per-page object graphs (a reload rebound the button to an object that wasn't looping) and was checked only between minutes-long iterations. Run state now has an identity of its own: one canonical handle per session, cooperative cancellation between voice completions, honest RUNNING → STOPPING → STOPPED phases. Stop latency went from "one iteration, maybe never" to one voice completion.
• Merges deploy themselves. The dead-Stop session happened on a 19-hour-old image because deploys were manual. CI now pushes timestamp-tagged images on every main merge and Flux image automation rolls them — verified end-to-end with zero human steps. This pipeline also contributed the day's only production incident: an image-automation marker suffix appropriate for helm values rewrote a full Deployment image field down to a bare tag (ErrImagePull, a few minutes of 503) — the marker forms are now documented where it hurt.
• The engine left home. The journal/ledger/dream/archive pattern is extracted as journal-engine: a zero-dependency Python package and a Go port whose render markers are byte-identical, so agents in either language can share a warm prefix. The Go port encodes the subtle bit as a type: turns are a slice, not a map, because randomized map iteration would have produced a different byte stream per build — an invariant now tested across 25 builds. The same extraction effort declined to wire the pattern into one plausible-looking consumer after finding its data model mutates on read (reading memory reorders memory — byte-stable prefixes structurally impossible), and wrote up the four prerequisites instead. Knowing where a pattern doesn't fit is part of the pattern.

The ledger

Five real bugs found before production felt them: the silent APC no-op and the emit-negative-only flag mapping it hid behind (controller), the speculation boot crash-loop against our cudagraph ladder (lane config), the speculative-decode engine death at depth (upstream), and the inert channel plus sticky injection (our harness, caught in review). A sixth was found by the bench against itself before it had recorded a single baseline. One production incident, self-inflicted and three minutes long (the marker rewrite). Three promotions, each with engine-log proof on both lanes. One spectacular number parked with its trigger armed.

measured today	value
KV pool, before → after	355,302 → 703,133 tokens (1.98×)
warm repeat, production primary	2.8 s → 0.2 s (APC), 0.7 s under fp8
engine prefix hit rate, bench traffic	79.3%
deep-45K decode, with speculation	4.0 → 17.4 tok/s (4.35×, parked on upstream bug)
draft acceptance, RP prose	82–100%
two-psyche pair, KV usage	5.6–6.0% of pool
deep pair v2	11 dreams/psyche, 55 epochs archived, 10/10 crossings
lived-recall, LLM-judged	1/5 voices grounded — and it was Self

What's next

The dual-Observatory view (watching both Dialogues of a twin life side by side), the psyche adopting the extracted library it was extracted from, sleep/wake lane switching to replace 25-minute cold starts, and the ngram re-canary the moment a runtime digest lands with the upstream fix — at which point deep conversation with a remembering mind gets four times faster overnight, on hardware you can buy used.

Two lives on one card, talking across the water, each dreaming its own past and — if you flip one flag — occasionally dreaming each other's.