Five Voices, 200K Tokens, One Consumer GPU: Giving Simulated Minds a Whole Life to Remember

TL;DR

• My Jungian psyche lab — five archetype agents conversing over a constrained message network — used to remember its last 10 exchanges. Now each agent carries an append-only journal of its entire life, rendered into every prompt, targeting the platform's 228,352-token input window.
• The trick that makes this affordable on a single 24 GB consumer GPU is prompt shape: [fixed identity][append-only journal][this turn]. The journal only ever grows at the tail, so vLLM's prefix cache can re-serve the whole life as a warm KV prefix and each turn prefills only the delta. [Update 2026-07-26: the shape is right, but it was not yet being rewarded — prefix caching was disabled on this lane during the run described here. See the dated correction below.]
• When a journal crosses 92% of its budget, the agent enters a dream cycle: the model distills its own oldest epochs — in character, in first person — into a core memory that replaces them. The consolidation loop is designed to keep long sessions within budget; the psyche's early life survives as memory, with the information loss measured in the postscript.
• Budgets are fleet-aware: five journals at 64K tokens each fit inside the lane's measured 355,302-token KV pool. A focused run can push a single journal toward the full 228K ceiling.
• The instrumented marathon: a 75-epoch life at a deliberately small 16K budget — 139 dream cycles, ~3M prompt tokens served, compressions like 14,685 → 7,643 tokens per sleep, ~11K-token prompts, and a token ledger that self-calibrated from an assumed 4.0 chars/token to a measured 4.34–4.43 across all five voices independently. It survived a real power outage mid-dream (see the postscript).
• Everything below runs on hardware you can buy used: an RX 7900 XTX serving a 9B model with a roleplay LoRA at 229K context, a Radeon VII doing embeddings, and a GTX 980 Ti painting the dream imagery.

---

Update (2026-07-26): one correction and a doubled pool

This post is a dated artifact — it describes the lab as of 2026-07-24, and the two days after it published changed three of its numbers and corrected one of its claims. The sequel has the full story: Twin Life on a Doubled Pool: One Day of Canary-Driven Inference Engineering. The short version:

• Prefix caching was disabled during the marathon. The prompt layout was designed for reuse, but the model configuration and controller kept the cache off. All 130 recorded per-voice readings reported cached_tokens: null and prefix_cache_hit_rate: null. I had described the intended benefit as an observed result. The affected claims below are corrected.
• The payoff arrived on 2026-07-25, and it was immediate — precisely because the prompt shape was already correct. Once the flag was fixed and promoted: 79.3% engine prefix-cache hit rate on bench traffic, and a warm repeat on the production lane going from 2.8 s to 0.2 s with cached_tokens: 8976/9028. The warm repeat was 14× faster in that check. It is a before/after example, not a fleet-wide speedup.
• The KV pool doubled. fp8_e4m3 KV cache took the measured pool from 355,302 → 703,133 tokens (1.98×). Every budget derivation below is still the right method, but the input number is now 703,133 — which is why the sequel runs two complete five-voice psyches on the one card, passing messages across a channel, instead of one.
• The memory hierarchy shipped. Dreams now carry an episodic ledger of neutral event lines that survives consolidation verbatim, with a similarity guard against exactly the fossilized-Ego failure documented in this post's postscript.
• One metric below is mislabeled. The "time-to-first-byte in the tens of milliseconds" figures are not time-to-first-byte. See the note in the results table.

---

The lab

Psyche-simulation is the oldest experiment on the FlexInfer platform: five agents — Shadow, Persona, Anima/Animus, Self, and Ego — talk to each other through a communication graph that only allows certain pairs to speak (Shadow can needle Persona; only under measured stagnation do emergency channels open). Each turn is an LLM completion; a NiceGUI front end streams the dialogue live with sentiment analysis, a network mandala, and a generative "Dreamscape" that paints each archetype as an evolving portrait.

It predates the platform. It once drove a single Ollama node through a hand-rolled proxy with 10 requests per minute of app-side throttling to survive. Every platform capability we've added since — model routing by name or LoRA adapter, prefix-cache-aware scheduling, scale-to-zero, observability headers — eventually got exercised by this lab first. So when our production 9B lane was promoted to a 229,376-token context window, the obvious question was: what would it take for the sim to actually use it?

The answer at the time: nothing could. Each agent kept a rolling deque of its last 10 exchanges. The prompt put the current situation before the chat history, which meant every single turn invalidated the KV cache after the system prompt. We had built a platform with a six-figure token window and prefix-cache routing, and our flagship lab was sending it 2K-token goldfish prompts that never cached.

The engine: an append-only life

The overhaul replaced the deque with a journal — the agent's entire subjective existence, chronological and append-only:

• every situation the world posed,
• everything it heard from the other archetypes (deduplicated against stale re-deliveries),
• everything it said.

The journal renders into every prompt. The prompt structure is the entire performance story:

[system]  archetype identity — fixed for the agent's lifetime
[user]    journal render    — append-only between dream cycles
          "now" block       — this turn's situation, voices, ask

The invariant that matters: between dream cycles, an earlier render is a strict string prefix of every later render. That's not a style preference — it's the contract vLLM's prefix cache lives on. Break it anywhere (a mutated system prompt, a "helpful" dynamic header, re-sorted history) and the whole life gets re-prefilled from scratch every turn. We pinned the invariant with a unit test, because it's exactly the kind of property a well-meaning refactor silently destroys.

Two design consequences fell out of taking the invariant seriously:

1. Prompt adaptation had to move. The sim's DynamicPromptManager nudges agents ("Focus on finding common ground…") based on measured conflict and stagnation. It used to mutate the prompt template — compounding, cache-hostile. Now guidance is ephemeral: it renders at the prompt tail, inside the only region that changes per turn.
2. The token math has to be honest. A self-calibrating ledger estimates journal depth in tokens and corrects itself against the exact prompt_tokens the lane reports (streamed calls get theirs via stream_options: {include_usage: true}, because response headers are sent before generation finishes and can't carry counts). In last night's run the ledger drifted from its assumed 4.0 chars/token to a measured 4.39 — a ~10% error we'd otherwise have carried into every budget decision.

Dream cycles: forgetting as a feature

An append-only life eventually fills any window. At 92% of budget, the agent stops and dreams: the oldest epochs are handed back to the model with its own prior core memory and any surfaced unconscious material, and it's asked — in character — to return one replacement memory that can stand on its own: who it has become, what it learned about the others, unresolved tensions, turning points.

The replacement memory takes the top of the journal and the distilled epochs are removed. Each consolidation includes the prior core memory, which keeps the summary from growing with every cycle. This is a lossy summarization scheme, not a model of human memory; the long run below shows which events it failed to retain. In the UI, the transition is explicit:

☽ Ego sinks into a consolidating dream — 3 epochs being distilled… ☽ Ego awakes; the oldest epochs live on as core memory (12,437 → 7,676 tokens)

A dream is also the one deliberate cache reset: the journal prefix changes wholesale, the next turn prefills cold, and the append-only clock starts again. That trade — one cold prefill per sleep in exchange for a bounded, coherent life — is the whole memory architecture in one sentence.

Fleet-aware budgets, not vibes

The naive budget is "the model takes 228K, so use 228K." The correct budget came from measuring the lane: the 24 GB card's KV pool holds 355,302 tokens at serving precision. Five archetypes near 228K each would thrash the pool and evict each other's warm prefixes every turn — cache hit rate collapses exactly when prompts are most expensive.

I chose a default journal budget of 64K tokens per agent: five journals total 320K tokens against the measured 355,302-token pool. That leaves some room for other prompt content, though eviction and runtime behavior still need measurement. Prefix caching was disabled during this first run, so the budget was a capacity estimate rather than evidence of warm-cache reuse.

For last night's observed run we shrank the vessel further (16K budgets) so dream cycles would happen on camera instead of at hour four. Same engine, same thresholds, smaller life.

Measurement correction, July 26, 2026. The proxy’s X-Flexinfer-Upstream-Ms header measures the wait for upstream response headers. On this streaming path those headers arrive before prefill and token generation, so 11–66 ms describes header latency, not time to first token. Per-turn generation still took seconds for roughly 700 output tokens per voice. The earlier TTFB interpretation was an error in this article.

The observatory

Long sessions broke the old UI before they broke anything else: one endless page of stacked panels, with the live dialogue and the instruments separated by a thousand pixels of scrolling. The rebuilt front end is a single viewport that never scrolls:

• a command bar (situation, iterations, begin/stop, stimulus, a confirm-gated reset — because "Reset" now deletes five lifetimes);
• the Dialogue as a permanent left stage — follow mode pins to the newest voice, per-archetype chips filter any voice in or out live;
• every instrument in tabs on the right: Memory (per-agent context gauges against budget, prefix-cache warm share, dream counts, a click-through journal reader), Chronicle (the whole session newest-first — one line per iteration with its situation and a sentiment dot per voice, dreams threaded between), plus the psyche metrics, mandala, latent-space map, and Dreamscape.

The Memory tab is the platform's observability made visible: per-turn prompt_tokens, cached-token share, and upstream latency come straight off the completion metrics every request already carries. One honest detail: when the proxy doesn't report cached tokens, warm share displays as "—", not a fake 0%. Dashboards that invent zeros teach you to distrust them.

Measurement correction, July 26, 2026. The proxy’s X-Flexinfer-Upstream-Ms header measures the wait for upstream response headers. On this streaming path those headers arrive before prefill and token generation, so 11–66 ms describes header latency, not time to first token. Per-turn generation still took seconds for roughly 700 output tokens per voice. The earlier TTFB interpretation was an error in this article.

What the run measured

An 80-iteration run, five voices, 16K budgets, streamed through the in-cluster proxy, tabulated from its first 11 iterations. From the persisted telemetry and journals:

Metric	Value	Notes
Dream cycles	11 in 11 iterations	first at epoch 6; all five voices within two epochs of each other
Compression per sleep	~45–50%	e.g. 14,685 → 7,643; 13,610 → 7,892; 12,656 → 6,302 tokens
Prompt size at depth	~10.9K tokens/turn	five of these per iteration, all streamed
Proxy→upstream header latency	tens of ms	11–66 ms; see the note below — this is not TTFB
Ledger calibration	4.0 → 4.34–4.43 chars/token	converged within ~10 completions of exact usage
Lane ceiling	229,376 ctx / 228,352 input	qwen35-9b + rank-64 RP LoRA, single RX 7900 XTX
KV pool	355,302 tokens	the number the 64K default budget is derived from; 703,133 since 2026-07-26

Measurement correction, July 26, 2026. The proxy’s X-Flexinfer-Upstream-Ms header measures the wait for upstream response headers. On this streaming path those headers arrive before prefill and token generation, so 11–66 ms describes header latency, not time to first token. Per-turn generation still took seconds for roughly 700 output tokens per voice. The earlier TTFB interpretation was an error in this article.

Everything persists: journals and core memories in Redis (7-day TTL), a per-iteration telemetry time series, and a collector that snapshots a session into a durable run directory — because "the demo worked" is not data, and TTLs that are right for an app are wrong for an experiment.

The stack, because it matters

The experiment runs on my K3s cluster using consumer hardware. The results below apply to that model, runtime, and workload; they are not a total-cost comparison with a hosted service.

• Text: Qwen3.5-9B (abliterated) with a rank-64 roleplay LoRA, served by vLLM on one AMD RX 7900 XTX (24 GB) at 229K context. Apps select it by adapter name through the platform proxy — moving the lane between nodes changes nothing upstream.
• Embeddings: bge-large on a Radeon VII, feeding the latent-space map of each archetype's drift.
• Imagery: SD1.5 on a GTX 980 Ti — a 2015 card — painting each archetype's evolving portrait; short i2v clips animate epoch transitions on the 7900 XTX between text batches.
• Control plane: K3s + Flux GitOps, Harbor registry, Redis state, Prometheus/Grafana, and the proxy's per-request observability headers stitching it together.

The platform contracts did the heavy lifting: OpenAI-compatible surface, model/adapter routing by header, session affinity, prefix-cache-aware scheduling, scale-to-zero for the lanes that sleep. The application's job was to shape its work to the platform — append-only prompts, session-scoped cache keys, budgets derived from measured pools — and that's precisely where application changes affected cache reuse.

Takeaways

• Long context is an architecture, not a parameter. The window was there for months; it was worthless until the prompt became append-only and the budget came from the KV pool instead of the model card.
• Prefix caching rewards discipline you must design for. One invariant — earlier renders are prefixes of later renders — decided the entire cost profile. Pin it with a test. [Update 2026-07-26: this held up better than we knew, and for a different reason than we thought. The discipline bought nothing during the marathon, because the cache was off. It paid the day the flag was fixed: 79.3% hit rate and a 14× warm repeat, with no application change at all. Design for the cache before you have it, and verify against the engine's boot log that you actually have it.]
• Forgetting needs to be designed with the same care as remembering. In-character consolidation that re-integrates the prior core kept the memory block bounded in this run, but sometimes retained voice at the expense of events.
• Trust your gauges only if they refuse to lie. Exact usage from the final stream chunk, a self-calibrating ledger, and "—" instead of fabricated zeros caught real gaps that averages would have hidden. [Update 2026-07-26: this bullet originally credited the gauges with catching "a proxy header regression." There was no proxy regression. The "—" was reporting a genuinely disabled prefix cache, correctly, the whole time — the gauge refused to lie and we misread it anyway. Which sharpens the takeaway rather than softening it: an honest instrument is only half the system; the other half is reading it as evidence about the engine instead of about the instrument.]
• Measure the whole memory path. This 24 GB Radeon supported the five-agent workload at the tested budget. Cache configuration, prompt overhead, concurrency, and consolidation quality still limit what fits and what remains useful.

Postscript: what the 75-epoch run showed

The draft above ended with the psyche still running. It then ran for 75 epochs across an entire day — through two planned restarts and one genuine power outage — and the marathon rewrote the ending. Final tally: 139 dream cycles, ~3M prompt tokens (2.03M in the final act alone, from the persisted telemetry), and every voice's core still bounded at ~4K chars after thirty-plus consolidations each.

Three findings only a long run could produce:

The world froze, and the instruments caught it. Our situation-evolution heuristic had a fixed point: under sustained stagnation it returned the same sentence — "Seeking new perspectives and breaking patterns" — every epoch, from epoch 6 to 31. The situation literally was the pattern it claimed to break. Worse, the persisted conversation records proved that mid-run stimulus injections were being silently overwritten before any iteration ever consumed them: the "inject stimulus" button was a no-op while a run was live.

Both are fixed now (injections are sticky for exactly one iteration; no evolution branch may return the situation the psyche already has), but the lesson generalizes: closed loops need fixed-point analysis, and "the control did nothing" is invisible without durable records. Grep-the-database beat every impression we had formed from watching the stream.

Consolidation preserves identity, not biography. Ego's core memory opened with the verbatim text of its first dream for twelve consecutive consolidations — a per-voice attractor at dream temperature. And when cores did evolve, they consistently kept each archetype's register (Shadow's hunger, Self's integration language) while shedding events — the epochs, the other voices, the turning points the dream prompt explicitly asks for. Voice survives compression far better than story. If you build memory consolidation on in-character summarization, plan for this: similarity guards against verbatim carry-over, and instructions that privilege episodes over vibe.

The architecture earned its keep during the outage. Power dropped mid-run — one agent was literally mid-dream. The journals came through with zero corruption, because a journal only records a turn that actually completed: the engine fails clean by construction, not by handler. Recovery was a chained resume (PSYCHE_RESUME_FROM): the same five lives crossed three process generations and a dead cluster, and the narrative resumed with the world "returning after a long silence." The final epoch's memories sit on a used GPU that had rebooted from a power cut hours earlier.

The experiment's last image belongs to Ego. After 32 dreams, the rational mind's entire compressed biography is a single scene: standing before its Shadow, being told "why cling to those rigid definitions when the truth is so much richer?" We built the memory system; the psyche decided what was worth remembering.