Five Voices, 200K Tokens, One Consumer GPU: Giving Simulated Minds a Whole Life to Remember

TL;DR

• Our Jungian psyche lab — five archetype agents conversing over a constrained message network — used to remember its last 10 exchanges. Now each agent carries an append-only journal of its entire life, rendered into every prompt, targeting the platform's 228,352-token input window.
• The trick that makes this affordable on a single 24 GB consumer GPU is prompt shape: [fixed identity][append-only journal][this turn]. The journal only ever grows at the tail, so vLLM's prefix cache re-serves the whole life as a warm KV prefix and each turn prefills only the delta.
• When a journal crosses 92% of its budget, the agent enters a dream cycle: the model distills its own oldest epochs — in character, in first person — into a core memory that replaces them. The simulation runs indefinitely at depth; the psyche's early life survives as memory, the way memory actually works.
• Budgets are fleet-aware: five journals at 64K tokens each fit inside the lane's measured 355,302-token KV pool. A focused run can push a single journal toward the full 228K ceiling.
• The instrumented marathon: a 75-epoch life at a deliberately small 16K budget — 139 dream cycles, ~3M prompt tokens served, compressions like 14,685 → 7,643 tokens per sleep, ~11K-token prompts with time-to-first-byte in the tens of milliseconds, and a token ledger that self-calibrated from an assumed 4.0 chars/token to a measured 4.37 across all five voices independently. It survived a real power outage mid-dream (see the postscript).
• Everything below runs on hardware you can buy used: an RX 7900 XTX serving a 9B model with a roleplay LoRA at 229K context, a Radeon VII doing embeddings, and a GTX 980 Ti painting the dream imagery.

---

The lab

Psyche-simulation is the oldest experiment on the FlexInfer platform: five agents — Shadow, Persona, Anima/Animus, Self, and Ego — talk to each other through a communication graph that only allows certain pairs to speak (Shadow can needle Persona; only under measured stagnation do emergency channels open). Each turn is an LLM completion; a NiceGUI front end streams the dialogue live with sentiment analysis, a network mandala, and a generative "Dreamscape" that paints each archetype as an evolving portrait.

It predates the platform. It once drove a single Ollama node through a hand-rolled proxy with 10 requests per minute of app-side throttling to survive. Every platform capability we've added since — model routing by name or LoRA adapter, prefix-cache-aware scheduling, scale-to-zero, observability headers — eventually got exercised by this lab first. So when our production 9B lane was promoted to a 229,376-token context window, the obvious question was: what would it take for the sim to actually use it?

The answer at the time: nothing could. Each agent kept a rolling deque of its last 10 exchanges. The prompt put the current situation before the chat history, which meant every single turn invalidated the KV cache after the system prompt. We had built a platform with a six-figure token window and prefix-cache routing, and our flagship lab was sending it 2K-token goldfish prompts that never cached.

The engine: an append-only life

The overhaul replaced the deque with a journal — the agent's entire subjective existence, chronological and append-only:

• every situation the world posed,
• everything it heard from the other archetypes (deduplicated against stale re-deliveries),
• everything it said.

The journal renders into every prompt. The prompt structure is the entire performance story:

[system]  archetype identity — fixed for the agent's lifetime
[user]    journal render    — append-only between dream cycles
          "now" block       — this turn's situation, voices, ask

The invariant that matters: between dream cycles, an earlier render is a strict string prefix of every later render. That's not a style preference — it's the contract vLLM's prefix cache lives on. Break it anywhere (a mutated system prompt, a "helpful" dynamic header, re-sorted history) and the whole life gets re-prefilled from scratch every turn. We pinned the invariant with a unit test, because it's exactly the kind of property a well-meaning refactor silently destroys.

Two design consequences fell out of taking the invariant seriously:

1. Prompt adaptation had to move. The sim's DynamicPromptManager nudges agents ("Focus on finding common ground…") based on measured conflict and stagnation. It used to mutate the prompt template — compounding, cache-hostile. Now guidance is ephemeral: it renders at the prompt tail, inside the only region that changes per turn.
2. The token math has to be honest. A self-calibrating ledger estimates journal depth in tokens and corrects itself against the exact prompt_tokens the lane reports (streamed calls get theirs via stream_options: {include_usage: true}, because response headers are sent before generation finishes and can't carry counts). In last night's run the ledger drifted from its assumed 4.0 chars/token to a measured 4.39 — an 9% error we'd otherwise have carried into every budget decision.

Dream cycles: forgetting as a feature

An append-only life eventually fills any window. At 92% of budget, the agent stops and dreams: the oldest epochs are handed back to the model with its own prior core memory and any surfaced unconscious material, and it's asked — in character — to return one replacement memory that can stand on its own: who it has become, what it learned about the others, unresolved tensions, turning points.

The replacement memory takes the top of the journal; the distilled epochs drop; the life continues. The core stays bounded forever because each dream re-integrates the previous core rather than stacking summaries — early epochs compress progressively, like actual memory. In the UI the moment reads as ceremony:

☽ Ego sinks into a consolidating dream — 3 epochs being distilled… ☽ Ego awakes; the oldest epochs live on as core memory (12,437 → 7,676 tokens)

A dream is also the one deliberate cache reset: the journal prefix changes wholesale, the next turn prefills cold, and the append-only clock starts again. That trade — one cold prefill per sleep in exchange for a bounded, coherent life — is the whole memory architecture in one sentence.

Fleet-aware budgets, not vibes

The naive budget is "the model takes 228K, so use 228K." The correct budget came from measuring the lane: the 24 GB card's KV pool holds 355,302 tokens at serving precision. Five archetypes near 228K each would thrash the pool and evict each other's warm prefixes every turn — cache hit rate collapses exactly when prompts are most expensive.

So the default journal budget is 64K per agent: five warm lives fit the pool with headroom, every turn hits warm KV, and a focused single-agent experiment can still override toward the ceiling. Getting this number required nothing exotic — the platform exposes pool size, and the config comment now cites it — but it's the difference between an architecture that demos well and one that runs for days.

For last night's observed run we shrank the vessel further (16K budgets) so dream cycles would happen on camera instead of at hour four. Same engine, same thresholds, smaller life.

The observatory

Long sessions broke the old UI before they broke anything else: one endless page of stacked panels, with the live dialogue and the instruments separated by a thousand pixels of scrolling. The rebuilt front end is a single viewport that never scrolls:

• a command bar (situation, iterations, begin/stop, stimulus, a confirm-gated reset — because "Reset" now deletes five lifetimes);
• the Dialogue as a permanent left stage — follow mode pins to the newest voice, per-archetype chips filter any voice in or out live;
• every instrument in tabs on the right: Memory (per-agent context gauges against budget, prefix-cache warm share, dream counts, a click-through journal reader), Chronicle (the whole session newest-first — one line per iteration with its situation and a sentiment dot per voice, dreams threaded between), plus the psyche metrics, mandala, latent-space map, and Dreamscape.

The Memory tab is the platform's observability made visible: per-turn prompt_tokens, cached-token share, and upstream latency come straight off the completion metrics every request already carries. One honest detail: when the proxy doesn't report cached tokens (a header gap we found during this work, fix in flight), warm share displays as "—", not a fake 0%. Dashboards that invent zeros teach you to distrust them.

What the run measured

An 80-iteration life, five voices, 16K budgets, streamed through the in-cluster proxy. From the persisted telemetry and journals:

Metric	Value	Notes
Dream cycles	11 in 11 iterations	first at epoch 6; all five voices within two epochs of each other
Compression per sleep	~45–50%	e.g. 14,685 → 7,643; 13,610 → 7,892; 12,656 → 6,302 tokens
Prompt size at depth	~10.9K tokens/turn	five of these per iteration, all streamed
Time-to-first-byte	tens of ms	upstream connect readings of 11–66 ms on warm turns
Ledger calibration	4.0 → 4.39 chars/token	converged within ~10 completions of exact usage
Lane ceiling	229,376 ctx / 228,352 input	qwen35-9b + rank-64 RP LoRA, single RX 7900 XTX
KV pool	355,302 tokens	the number the 64K default budget is derived from

Everything persists: journals and core memories in Redis (7-day TTL), a per-iteration telemetry time series, and a collector that snapshots a session into a durable run directory — because "the demo worked" is not data, and TTLs that are right for an app are wrong for an experiment.

The stack, because it matters

This is the part I'd underline for anyone who thinks serious LLM systems require a cloud bill: the entire experiment runs on a small K3s cluster of consumer hardware.

• Text: Qwen3.5-9B (abliterated) with a rank-64 roleplay LoRA, served by vLLM on one AMD RX 7900 XTX (24 GB) at 229K context. Apps select it by adapter name through the platform proxy — moving the lane between nodes changes nothing upstream.
• Embeddings: bge-large on a Radeon VII, feeding the latent-space map of each archetype's drift.
• Imagery: SD1.5 on a GTX 980 Ti — a 2015 card — painting each archetype's evolving portrait; short i2v clips animate epoch transitions on the 7900 XTX between text batches.
• Control plane: K3s + Flux GitOps, Harbor registry, Redis state, Prometheus/Grafana, and the proxy's per-request observability headers stitching it together.

The platform contracts did the heavy lifting: OpenAI-compatible surface, model/adapter routing by header, session affinity, prefix-cache-aware scheduling, scale-to-zero for the lanes that sleep. The application's job was to shape its work to the platform — append-only prompts, session-scoped cache keys, budgets derived from measured pools — and that's precisely where all the leverage was.

Takeaways

• Long context is an architecture, not a parameter. The window was there for months; it was worthless until the prompt became append-only and the budget came from the KV pool instead of the model card.
• Prefix caching rewards discipline you must design for. One invariant — earlier renders are prefixes of later renders — decided the entire cost profile. Pin it with a test.
• Forgetting needs to be designed with the same care as remembering. In-character consolidation that re-integrates the prior core keeps the memory block bounded forever and reads better than any summarizer we could have bolted on.
• Trust your gauges only if they refuse to lie. Exact usage from the final stream chunk, a self-calibrating ledger, and "—" instead of fabricated zeros caught real gaps (including a proxy header regression) that averages would have hidden.
• Consumer hardware is not the constraint. A used 24 GB Radeon holds five simulated lifetimes in KV cache with room to spare. The constraint is whether your controllers, metrics, and harnesses are honest.

Postscript: what a 75-epoch life actually taught us

The draft above ended with the psyche still running. It then ran for 75 epochs across an entire day — through two planned restarts and one genuine power outage — and the marathon rewrote the ending. Final tally: 139 dream cycles, ~3M prompt tokens (2.03M in the final act alone, from the persisted telemetry), and every voice's core still bounded at ~4K chars after thirty-plus consolidations each.

Three findings only a long run could produce:

The world froze, and the instruments caught it. Our situation-evolution heuristic had a fixed point: under sustained stagnation it returned the same sentence — "Seeking new perspectives and breaking patterns" — every epoch, from epoch 6 to 31. The situation literally was the pattern it claimed to break. Worse, the persisted conversation records proved that mid-run stimulus injections were being silently overwritten before any iteration ever consumed them: the "inject stimulus" button was a no-op while a run was live. Both are fixed now (injections are sticky for exactly one iteration; no evolution branch may return the situation the psyche already has), but the lesson generalizes: closed loops need fixed-point analysis, and "the control did nothing" is invisible without durable records. Grep-the-database beat every impression we had formed from watching the stream.

Consolidation preserves identity, not biography. Ego's core memory opened with the verbatim text of its first dream for twelve consecutive consolidations — a per-voice attractor at dream temperature. And when cores did evolve, they consistently kept each archetype's register (Shadow's hunger, Self's integration language) while shedding events — the epochs, the other voices, the turning points the dream prompt explicitly asks for. Voice survives compression far better than story. If you build memory consolidation on in-character summarization, plan for this: similarity guards against verbatim carry-over, and instructions that privilege episodes over vibe.

The architecture earned its keep during the outage. Power dropped mid-run — one agent was literally mid-dream. The journals came through with zero corruption, because a journal only records a turn that actually completed: the engine fails clean by construction, not by handler. Recovery was a chained resume (PSYCHE_RESUME_FROM): the same five lives crossed three process generations and a dead cluster, and the narrative resumed with the world "returning after a long silence." The final epoch's memories sit on a used GPU that had rebooted from a power cut hours earlier.

The experiment's last image belongs to Ego. After 32 dreams, the rational mind's entire compressed biography is a single scene: standing before its Shadow, being told "why cling to those rigid definitions when the truth is so much richer?" We built the memory system; the psyche decided what was worth remembering.