FlexInfer docs

Gemma4 ROCm Status

Living status for Gemma4 quantized runtime gates on unified gfx1100 runtimes.

Gemma4 ROCm Status

This document tracks the current managed state of the Gemma4 quantized deployments on the unified gfx1100 runtime path.

It also defines a reusable quantized-artifact promotion gate for other families (Qwen, Llama, Mistral, etc.): the gate is motivated by Gemma4 findings but is intentionally model-family agnostic.

Update this document whenever a tuning change lands or a new blocker is found.

Current profiles

Model ID	Model CR	Node	Attention / KV path	Intent
`gemma4-31b-gptq`	`gemma4-31b-gptq`	`cblevins-7900xtx`	`TRITON_ATTN` + float16 KV	Current warm primary at the validated 2K ceiling (`minReplicas: 1`)
`gemma4-26b-a4b-gptq-long`	`gemma4-26b-a4b-gptq-long`	`cblevins-7900xtx`	`TRITON_ATTN` + FP8 KV	16K proof lane and only reconciled 26B profile (`minReplicas: 0`, priority 255 for explicit validation demand)

Current profile knobs

Model ID	`maxModelLen`	`maxNumBatchedTokens`	`gpuMemoryUtilization`	Serverless
`gemma4-31b-gptq`	`2048`	runtime default	`0.95`	`minReplicas: 1`
`gemma4-26b-a4b-gptq-long`	`16384`	`160`	`0.98`	`minReplicas: 0`

Latest baseline

Date: 2026-05-03

Current finding:

The reconciled Gemma4 serving surface is deduplicated to gemma4-31b-gptq and gemma4-26b-a4b-gptq-long. The base 26B profile and 22K canary remain on disk for reference, but are not reconciled.
gemma4-26b-a4b-gptq-long intentionally outranks the warm 31B primary (priority: 255 vs 250) so explicit 26B validation demand can preempt, serve, and hand the lane back after its idle timeout.
The current Gemma4 26B-A4B hybrid artifact (gptq-w4-g128-attnfp16-clean) is coherent and stable at 16K when served with FP8 KV and the validated ROCm fallback switches.
The same artifact is too large/risky for default 32K serving on a single 24 GB gfx1100 card.
A 32K FP16-KV boot failed cleanly on 2026-04-26: vLLM needed 6.88 GiB of KV cache after loading the hybrid artifact, but only 1.87 GiB remained. The canary is therefore narrowed to 16K with kvCacheDtype: fp8_e4m3 for live validation.
The 16K FP8-KV canary passed the long-context probe on 2026-04-26 on cblevins-7900xtx: 14,088 prompt tokens returned the expected gemma4-long-ok marker with zero pod restarts. The default 26B alias now uses those same runtime knobs while remaining scale-to-zero and non-primary.
vLLM reported 22,608 GPU KV cache tokens for the 22K FP8-KV boot. The separate gemma4-26b-a4b-gptq-22k manifest remains available for the next practical rung, but it is not part of the reconciled serving set.
The 22K canary passed an in-cluster 17,092-token prompt retention probe on 2026-04-26, but it is partially validated only. It is no longer reconciled until the 18K-22K target window is re-qualified with a less noisy runtime build.
The smaller dense-validated artifact path remains blocked behind the cblevins-5930k memory guard; do not uncordon that node or remove the taint just to accelerate this lane.

16K FP8-KV proof (2026-04-26)

Run ID: gemma4-long-context-20260426T142859-1f6000

Case	HTTP	Prompt tokens	Completion tokens	Elapsed	Output
`short-sanity`	`200`	`50`	`2`	`4.565s`	`4`
`medium-context`	`200`	`6088`	`8`	`456.216s`	`gemma4-medium-ok`
`long-context`	`200`	`14088`	`8`	`1014.343s`	`gemma4-long-ok`

Cluster evidence:

Pod: gemma4-26b-a4b-gptq-long-5678f5499d-8lhnd
Node: cblevins-7900xtx
Image: registry.harbor.lan/flexinfer/runtime@sha256:0b05b32b92e6ab99cd648837a9bf80cf3dd437275b1d97fb71378a9f829cdaac
Probe artifact: /tmp/flexinfer-gemma4-probes/gemma4-long-context-20260426T142859-1f6000/gemma4-long-context-20260426T142859-1f6000.md
Result after idle timeout: gemma4-26b-a4b-gptq-long returned to Idle; gemma4-31b-gptq returned to Ready.

Runtime note: the current image includes verbose gemma4_layer_debug warnings, which made 6K/14K prefill slow. Future images should keep the decoder-layer debug patch disabled unless actively chasing NaNs.

22K FP8-KV canary proof (2026-04-26)

The 22K profile boots successfully on cblevins-7900xtx; vLLM reports max_seq_len=22000, kv_cache_dtype=fp8_e4m3, and 22,608 GPU KV-cache tokens.

In-cluster 17K probe:

Case	HTTP	Prompt tokens	Completion tokens	Elapsed	Output
`17k-incluster`	`200`	`17092`	`10`	`1223.656s`	`gemma4-17k-ok`

Cluster evidence:

Job: gemma4-22k-incluster-probe
Probe pod: gemma4-22k-incluster-probe-hgzhc on k3s-w-10
Backend pod: gemma4-26b-a4b-gptq-22k-65d79f6b4c-8h4ws on cblevins-7900xtx
Finished: 2026-04-26T23:38:13Z
Result source: Kubernetes termination message on the completed probe pod

Operational finding:

Proxy keepalive now refreshes status.lastActiveTime during long in-flight requests; the 17K run advanced the timestamp through the full 20-minute request instead of being idled out.
A local port-forwarded 20K probe is not accepted as model evidence because the port-forward died with lost connection to pod. Use in-cluster jobs for future 18K-22K proofs.
An earlier 17K in-cluster run returned HTTP 200 and processed 17,092 prompt tokens, but the marker was truncated by max_tokens=8; use at least max_tokens=24 for marker probes at this profile.

Quantized-artifact promotion gate (reusable)

Use this gate before promoting any quantized artifact (any model family) from canary to default:

Artifact validation at target context
- correctness/repetition checks pass (probe script + production parser path)
- warm and cold runs are both clean
Memory headroom validation
- no OOM or allocator instability at target context on intended GPU class
Canary containment
- minReplicas: 0
- warmPolicy: ondemand (or equivalent non-primary policy)
- no alias/default swap while unvalidated
Explicit promotion change
- only after 1-3 pass, raise context/default aliases in a separate change

Controller enforcement:

Mark guarded models with flexinfer.ai/promotion-gate: quantized-artifact-v1.
Warm-primary promotion (serverless.minReplicas > 0, serverless disabled, or config.warmPolicy: primary) requires either flexinfer.ai/promotion-validation: passed or flexinfer.ai/promotion-evidence: <artifact-or-runbook-ref>.
Canary / scale-to-zero operation remains allowed without evidence so candidate artifacts can be probed deliberately.
The controller reports the decision in the Model PromotionGate condition.

Long-context readiness probe (promotion gate input)

Use scripts/probe-gemma4-long-context.sh for repeatable target-context validation before promoting any long-context quantized canary. The probe runs:

short sanity: 2 + 2
medium context: repeated-token prompt with a retained verification code
long context: default ~30k-token prompt with the same retained verification code

It writes both JSON and Markdown artifacts, records prompt/completion tokens and elapsed time, and exits nonzero if the model returns obvious garbage, repetition, or the wrong answer.

Common runs:

kubectl -n ai port-forward svc/litellm 18000:8000
ENDPOINT=http://127.0.0.1:18000 ./scripts/probe-gemma4-long-context.sh

Direct service URL with cluster metadata/log hints:

ENDPOINT=http://litellm.ai.svc.cluster.local:8000 \
  AUTH_TOKEN=${LITELLM_MASTER_KEY} \
  MODEL=gemma4-26b-a4b-gptq \
  POD_SELECTOR='app=gemma4-26b-a4b' \
  ./scripts/probe-gemma4-long-context.sh

Features working

Feature	Status	Notes
Unified `gfx1100` runtime path	Working	No separate debug runtime required
Managed Gemma4 CRD deployment	Working	26B 16K fallback + 31B warm primary + long canaries reconcile through Flux
LiteLLM aliases	Working	26B aliases are promoted to the validated 16K FP8-KV profile
Tool calling	Working	Gemma parser path remains enabled on baseline profiles
Conservative rollout gates	Working	Canary remains scale-to-zero and non-primary
16K baseline coherence	Working	Current hybrid serves coherently at 16K on gfx1100 with FP8 KV

Features still being chased

Feature	Status	Current read
Smaller long-context artifact	In progress	Needed before promoting beyond 16K baseline
16K promotion validation	Promoted to 26B alias	26B canary used FP8 KV and passed the 14K-token probe; the default 26B alias now carries those knobs
22K upper-bound validation	Partial pass	22K boots and 17,092 prompt tokens pass in-cluster; 18K-22K target proof still pending on a quieter runtime
32K promotion validation	Blocked on artifact	Current hybrid needs a smaller artifact; FP16 KV boot needs 6.88 GiB with only 1.87 GiB available
Compressed-tensors + FP8 KV lane	Planned canary	Separate compressed-tensors artifact work remains disabled/non-default until validated
AITER on ROCm	Blocked / deferred	`TRITON_ATTN` remains the stable path on RDNA3
Production-grade long-context default	Deferred	Keep long profiles non-primary until gate checks pass
Speculative decoding	Not started	No Gemma4 speculator path wired yet
FP8-centric KV path	Planned canary	Reference config only; not promoted

Gemma4 GPTQ Pipeline Models

26B-A4B MoE (GPTQ INT4)

Field	Value
ModelCache	`gemma4-26b-a4b-gptq`
Model CR	`gemma4-26b-a4b-gptq`
Source	`google/gemma-4-26B-A4B-it`
Node	`cblevins-7900xtx` (gfx1100)
Pipeline	Download BF16 (~27 GB) → Abliterate → GPTQ INT4 (~7-13 GB)
PVC	96 Gi (nvme-1r-gpu)
Shared Group	`7900xtx-textgen` (priority 200, on-demand fallback)
Aliases	`gemma4-26b`, `gemma4-26b-a4b`, `gemma4-moe`

MoE Architecture: 25.2B total / 3.8B active, 128 experts top-8, 30 layers (25 GDN + 5 full-attention). Current hybrid export is validated at 16K with FP8 KV; it is not promoted for 22K/32K service. The separate gemma4-26b-a4b-gptq-long canary is retained as a proof lane, and gemma4-26b-a4b-gptq-22k remains the upper-bound canary.

Abliteration safety: Only o_proj (shared attention output). Expert FFN weights auto-skipped. ablitateLmHead: false (save corruption bug).

Quantization config: sym=true, descAct=false, maxSamples=512 (MoE expert coverage), timeoutSeconds=43200 (12h for 640 expert modules).

31B Dense (GPTQ INT4)

Field	Value
ModelCache	`gemma4-31b-gptq`
Model CR	`gemma4-31b-gptq`
Source	`google/gemma-4-31B-it`
Node	`cblevins-7900xtx` (gfx1100)
Pipeline	GPTQ INT4 runtime serving from `gptq-w4-g128-keqv`
Status	Warm primary at 2K (`minReplicas: 1`)

Dense Architecture: 30.7B params, 60 layers (50 GDN + 10 full-attention). Requires 128 GB RAM node for abliteration + save overhead.

Abliteration: Both o_proj and down_proj (safe for dense models, no MoE experts). maxMemoryGB=96.

Quantization config: maxMemoryGB=96, maxSamples=256 (no MoE), timeoutSeconds=28800 (8h).

GPTQ Performance on ROCm

Model	Decode tok/s	Prompt tok/s	VRAM	Context
26B-A4B MoE INT4	~72	~1800	~17.7 GB weights + FP8 KV	16K baseline
31B Dense INT4	TBD	TBD	~20 GB	2K validated ceiling

ExLlama v2 kernels (HIP-compiled) with sym=true achieve 7x faster decode than AWQ on gfx1100.

Reference-only future canary (disabled)

The following is a reference snippet for compressed-tensors + FP8 KV experiments. Keep this out of live defaults until artifact validation gates pass.

# reference only: do not include in deploy/models/kustomization.yaml yet
apiVersion: ai.flexinfer/v1alpha2
kind: Model
metadata:
  name: gemma4-31b-gptq-fp8kv-canary
  annotations:
    flexinfer.ai/promotion-gate: quantized-artifact-v1
    flexinfer.ai/promotion-state: canary-reference-only
spec:
  backend: vllm
  source: pvc://gemma4-31b-gptq/gemma4-31b-gptq/compressed-tensors-fp8kv-candidate
  serverless:
    enabled: true
    minReplicas: 0
  config:
    quantization: compressed-tensors
    kvCacheDtype: fp8_e4m3
    maxModelLen: 16384
    warmPolicy: ondemand

Deployment Reliability (2026-04-13)

Feature	Status	Notes
GPUProfile watch	Working	Controller watches GPUProfile CRs; image changes trigger reconciliation
Image drift detection	Working	Stale running jobs auto-deleted on GPUProfile image update
Script version marker	Working	`FLEXINFER_SCRIPT_VERSION=v12` checked at job startup
Deploy automation	Working	`make deploy-quantizer QUANTIZER_ARCH=gfx1100`
Spec hash with image	Working	`quantSpecHashWithImage()` includes resolved image in hash

Next tuning queue

Rebuild the unified runtime with decoder-layer debug logging disabled by default, then re-run the 16K probe to measure prefill without instrumentation.
Re-run the 22K FP8-KV canary with in-cluster 18K-22K prompt probes using max_tokens >= 24.
Produce a smaller 26B artifact candidate for 32K validation.
Keep 22K+ long-context canaries non-primary (minReplicas: 0, warmPolicy: ondemand) until promotion criteria are met.
Validate compressed-tensors + FP8 KV on a dedicated canary before any alias/default changes.
Generalize this gate to additional quantized model families in shared docs/manifests.

gfx1100 canary closure queue (2026-05-02)

This section is a docs-only closure note. No live cluster validation was run for this update. Future canary evidence should have one canonical home in .loom/60-validation-matrix.md, then link back here and to the PR-2 readiness plan at docs/planning/rocm-gfx1100-deploy-swap-tracing-slice.md.

Gate	Acceptance criteria	Validation commands	Evidence target
`26B-dense-rerun-gate`	`gemma4-26b-a4b-gptq-dense` reaches the dense-module cosine gate with `cos.min >= 0.98`, artifact metadata validation passes, runtime smoke is coherent, and load/context numbers are compared against the validated 16K FP8-KV fallback.	`kubectl get modelcache gemma4-26b-a4b-gptq-dense -n flexinfer-system -o yaml`; `flexinfer quantize validate-artifact --path <mounted-artifact> --layout vllm-gptq --family gemma4-26b-a4b --run-generation`; `ENDPOINT=<url> MODEL=<dense-canary> ./scripts/probe-gemma4-long-context.sh`.	Update `.loom/60-validation-matrix.md` row `gemma4-26b-a4b-gptq-dense`; raw logs under `.loom/local/validation/gemma4-26b-a4b-gptq-dense/<timestamp>/`.
`E4B-turboquant-runtime-probe`	A digest-pinned gfx1100 TurboQuant runtime with `TQ4_SHARE_PRIMITIVES=1` boots E4B to Ready, reaches KV sizing, returns coherent short/medium output, and avoids repeated-token collapse across a small sequential request loop.	Build/promote the current `gfx1100-gemma4-turboquant-experimental` runtime profile; `kubectl logs -n flexinfer-system \| rg 'TQ4_SHARE_PRIMITIVES	KV
`31B-long-turboquant-posture-gate`	Keep 31B production at `maxModelLen: 2048` unless the TurboQuant canary reaches KV sizing without attention-construction OOM and then passes a 4096 -> 8192 -> 16384 coherence ladder with memory headroom. Failure should produce an explicit closed posture, not an ambiguous canary.	`kubectl get model gemma4-31b-gptq gemma4-31b-gptq-long -n flexinfer-system -o yaml`; `kubectl logs -n flexinfer-system <31b-long-pod> \| rg 'Model loading took	KV
`Qwen35-9B-gfx1100-validation-gate`	Staged `qwen35-9b-gptq-gfx1100` completes Download -> Abliterate -> GPTQ -> OCI publish on gfx1100, artifact validator passes, serving smoke is coherent, and the keep/retire decision for the gfx906 artifact is recorded.	`kubectl get modelcache qwen35-9b-gptq-gfx1100 -n flexinfer-system -o yaml`; `flexinfer quantize validate-artifact --path <mounted-artifact> --layout vllm-gptq --family qwen35 --run-generation`; `kubectl get pods -n flexinfer-system -o wide \| rg qwen35`; run a LiteLLM chat smoke against the gfx1100 canary.	Populate `.loom/60-validation-matrix.md` row `qwen35-9b-gptq-gfx1100`; raw logs under `.loom/local/validation/qwen35-9b-gptq-gfx1100/<timestamp>/`.