FlexInfer docs

GPTQ Quantization Runbook

Operating GPTQ quantization jobs — diagnosing failures, recovering from stalls, and tuning for slow or numerically-tricky hardware.

GPTQ Quantization Runbook

1. What this covers

Operational procedures for GPTQ INT4 quantization runs on FlexInfer, specifically:

Recognizing the failure modes that have bitten us in production.
Recovering a failed / stalled / exhausted ModelCache without a full re-download.
Picking sensible timeoutSeconds, maxMemoryGB, and Hessian-recovery tunables for a new model.
Responding to the Prometheus alerts the controller emits.

This is not a reference for the GPTQ algorithm itself — see docs/design/quantization-pipelines.md for architecture.

2. Pipeline at a glance

For a GPTQ ModelCache, the controller runs these Jobs in order:

downloader — pulls source weights from HuggingFace into the PVC.
abliterate (optional) — modifies weights to suppress refusal directions.
quantize — the GPTQModel run. This is the expensive one: typically 60–90% of pipeline wall time.
publish — packages the quantized output as an OCI artifact.

All four run as K8s batch/v1.Jobs with backoffLimit=2, so up to three attempts per phase before the ModelCache is marked Failed. Each Job has its own activeDeadlineSeconds derived from the ModelCache's spec.<phase>.timeoutSeconds.

3. What GPTQ actually does (and why it's so slow)

For each transformer layer:

Forward pass with calibration data through the layer's modules (~seconds per layer).
Hessian accumulation: X.T @ X in FP32 for each linear module (cheap, piggybacks on step 1).
Cholesky inverse of the regularized Hessian to solve for optimal quantized weights. This is where time goes. Each Cholesky on a 21504² FP32 matrix is O(n³/3) ≈ 3.3 TFLOPs. On gfx906 (Vega20) with ROCm LAPACK on CPU fallback, one Cholesky is ~40 s. On gfx1100 with native GPU LAPACK, it's seconds.
Pack the quantized weights into the INT4 format and replace the module.

A 59-layer model has ~413 modules (7 per layer × 59). Most succeed on the first try. A handful are numerically ill-conditioned and need the Hessian recovery path.

4. Hessian recovery — what, why, when

The problem

Some modules (especially large-fan-in ones like mlp.down_proj with input dim 21504) have Hessian matrices that are nearly singular on BF16 calibration activations — 7 mantissa bits of precision isn't enough when you're summing 58k samples worth of activation products. Cholesky fails on these matrices.

The recovery loop (see `build/scripts/quantize_gptq.py:_patched_hessian_inverse`)

Damp sweep (attempt 0): add damp × mean(diag) to the diagonal, where damp starts at GPTQ_DAMP_PERCENT_OVERRIDE (default 0.05) and steps by GPTQ_DAMP_AUTO_INCREMENT_OVERRIDE (default 0.1). Retry Cholesky after each step up to damp ≥ 1.0.
Diagonal floor (attempts 1–6): if the damp sweep failed, apply floor_base × 10^(attempt-1) to the diagonal and try ONE Cholesky (no damp sweep at this tier). floor_base is mean(diag) × GPTQ_HESSIAN_DIAG_FLOOR_SCALE (default 0.01) under the default mean mode.
Exhaustion: if all 6 floor attempts still fail, log it and return (None, 1.0) so GPTQModel treats the module as unquantized (falls through to its own default handling).

Why the defaults are what they are

Before MR !156, the floor was scaled by abs_max × 1e-6 — ~1000× smaller than damp × mean for typical activations, so attempts 1–3 were invisibly weak. Modules would burn 60 min per sweep × 7 sweeps = 7 hours just to exhaust. The mean × 0.01 default puts attempt 1 on a comparable scale to damp (20% of it), attempt 2 at 2× damp, attempt 3 at 20× damp — each attempt is meaningful.

damp_step=0.1 cuts the initial sweep from 95 Cholesky iterations (step=0.0015) to 10, which is the single biggest wall-clock win on slow Cholesky backends.

HIP error cascade (gfx906 specific)

A Cholesky failure on gfx906 can poison the HIP context — subsequent GPU linalg ops on the SAME tensor fault with torch.AcceleratorError: HIP error: invalid argument, including torch.isfinite(H).sum(). MR !156 wraps the pre-loop sanitize check and setup in try/except so a bad Hessian returns cleanly instead of crashing the whole job.

Do not move H to CPU as a workaround — the rocm/pytorch container is compiled without CPU LAPACK, so Cholesky on CPU dies with LAPACK library not found in compilation and EVERY module exhausts.

5. Picking timeouts and memory for a new model

Expected per-layer wall time

Hardware	~min per layer (no recovery)	~min per layer (w/ 1 problematic module)
gfx1100 (24 GB)	1–2	2–4
gfx906 (16 GB)	3–5	8–12

Problematic modules exhausting all 7 attempts add ~1 min each on the post-fix code (~10 min each with the pre-fix code — don't use old images).

`timeoutSeconds` rule of thumb

timeoutSeconds = (per_layer_wall × num_layers + save_minutes + 30_min_buffer) × 60

For 31B on gfx906 (59 layers × ~10 min + 30 min save + buffer) = ~11 h — use 43200 (12 h) today. The CRD max is 172800 (48 h) post-MR !161.

`maxMemoryGB` rule of thumb

Quantize phase: model size + ~10 GiB overhead for activations and Hessian accumulation.

Save phase can be the bottleneck — GPTQModel's save_quantized loads all shards and rewrites; peak RSS is close to the full FP16 model size. For 26B BF16 (~52 GB) we needed maxMemoryGB=40 (container limit 52 Gi with the 12 GiB GPU driver overhead) to clear save. See MR !157.

Node-selection gotcha

On the radeonvii / 5930k nodes we have one GPU each and also run the control plane (etcd). Container memory limits near node.allocatable - kubelet_reserved can tip the kubelet into eviction. Leave at least 5 GiB headroom.

6. Recovering a stuck / failed ModelCache

6.1 The ModelCache is `Failed` but the run got ~90% through

Happens on DeadlineExceeded. Three options:

Option A — raise the timeout and retry. Update deploy/modelcaches/<name>.yaml, commit, let Flux reconcile, then reset the failure state:

export KUBECONFIG=~/workspace/platform/gitops/.kube/k3s.yaml

# 1. Edit + commit deploy/modelcaches/<name>.yaml with a larger timeoutSeconds
# 2. Wait for Flux reconcile
flux reconcile source git flexinfer -n flux-system
flux reconcile kustomization flexinfer-models -n flux-system

# 3. Reset retry state so the controller will re-attempt
kubectl patch modelcache -n flexinfer-system <name> \
  --type=merge --subresource=status \
  -p '{"status":{"phase":"Quantizing","retryCount":0,"lastFailurePhase":null,"lastFailureTime":null}}'

Option B — let the existing Job clock age out of the grace window, then delete it. If the controller already created a Job with the old deadline, it will keep that deadline until recreated. Delete the Job once Flux has applied the new spec:

kubectl delete job -n flexinfer-system <name>-quantize
# Controller will recreate with the new deadline

Option C — accept the loss and run again from scratch. Reasonable if the model is small and the failure was early.

6.2 The ModelCache OOMKilled during save

Save phase OOMs are nasty: GPTQModel's resume only handles stage=quantizing, so an OOM at stage=saving restarts the whole quantize from layer 0. Fix the memory limit FIRST, then recover:

Update deploy/modelcaches/<name>.yaml with a larger spec.quantization.maxMemoryGB. (See §5 for sizing.)
Commit + reconcile Flux.
Delete the failed Job to force recreation with the new memory limit:
```
kubectl delete job -n flexinfer-system <name>-quantize
```
The controller will recreate the Job. A fresh pod with the new memory limit runs from layer 0 — yes, all progress is lost. This is unfortunate but unavoidable until we ship real per-layer resume.

6.3 Quantize is running but stuck (no layer progress for 30+ min)

Check for Hessian recovery exhaustion:

POD=$(kubectl get pods -n flexinfer-system \
  -l job-name=<name>-quantize \
  -o jsonpath='{.items[0].metadata.name}')

# Is the process alive?
kubectl top pod -n flexinfer-system $POD

# Latest recovery events
kubectl logs -n flexinfer-system $POD \
  | grep -E 'Hessian recovery|damp recovery|diagonal floor' \
  | awk '!seen[$0]++' | tail -20

Interpretation:

Many diagonal floor +N (attempt X/6) lines within a few minutes — recovery is working; just let it finish (attempt 6 is worst case ~5 min on gfx906 with damp_step=0.1).
One starting damp recovery with no follow-ups for 30+ min — damp sweep is still running. On pre-fix code this can be an hour; with the post-MR !156 fix it's ~6 min per sweep. If you're on old code, bump GPTQ_DAMP_AUTO_INCREMENT_OVERRIDE via the controller env and redeploy.
Hessian recovery exhausted — the module fell through to (None, 1.0) and the run should have continued. If it didn't, see the HIP cascade note in §4.

6.4 Job keeps getting recreated right after start

Check for image-drift preemption. MR !160 added a 10-minute grace window; runs older than that should be safe. Symptoms:

kubectl get events -n flexinfer-system \
  --field-selector involvedObject.name=<name>-quantize \
  --sort-by=.lastTimestamp | tail -15

If you see repeated SuccessfulCreate events with QuantizerImageDrift warnings within the first 10 min, the GPUProfile digest is disagreeing with the running Pod. Usually this is Flux reconciling — check if someone hand-patched the GPUProfile recently.

To preempt a long-running job with a new image intentionally:

kubectl annotate modelcache -n flexinfer-system <name> \
  flexinfer.ai/force-image-update=true --overwrite

Then delete the Job. The controller will recreate with the current GPUProfile image. Remove the annotation after.

6.5 Qwen3.5 recovery after unsafe abliteration

Use this path when a Qwen3.5 GPTQ artifact serves garbage text even though the quantization Job completed. This is the failure from GitLab issue #51: the GPTQ weights faithfully preserved a corrupted FP16 source produced by unsafe abliteration. The root-cause report is docs/dev/qwen35-gptq-root-cause.md.

Before rebuilding, prove the serving runtime is not the only problem:

kubectl apply -f deploy/debug/qwen35-gptq-direct-test.yaml
kubectl logs -n flexinfer-system job/qwen35-gptq-direct-test -f

If direct GPTQModel inference produces multilingual token noise or fails a deterministic smoke such as "2 + 2", treat the artifact as corrupted. Do not try to promote it by changing vLLM flags.

Safe recovery sequence:

Pause the parent Model that consumes the cache, or keep its manifest disabled in deploy/models/kustomization.yaml, so no runtime pod mounts the artifact during rebuild.
Rebuild from a clean upstream FP16 source. For the first recovery pass, remove spec.abliteration from the ModelCache and quantize directly with GPTQ (format: GPTQ, bits: 4, groupSize: 128, sym: true, descAct: false).
If the old download directory is already marked complete, clear the corrupted source through the parent ModelCache lifecycle rather than deleting child Jobs or pods. In GitOps, that usually means disabling/deleting the ModelCache CR, letting the controller clean up owned resources, then re-enabling it with the clean-source spec. Only perform manual PVC cleanup after confirming no Model or Job is mounting it.

Let the controller re-run download then quantize. Watch the status and logs:

kubectl get modelcache -n flexinfer-system qwen35-27b-opus-distill-gptq -w
kubectl logs -n flexinfer-system job/qwen35-27b-opus-distill-gptq-quantize -f

Validate the new GPTQ artifact with the direct GPTQModel Job before any vLLM deployment or alias promotion. Passing means short deterministic prompts are coherent and the "2 + 2" top token is sane, not random multilingual noise.
Only after direct validation passes, publish/pull the OCI cache and re-enable the serving Model.

If an abliterated Qwen3.5 artifact is still required after the direct GPTQ recovery passes, use the guarded pipeline only: skipGDNLayers: true, narrowly target full-attention layers, keep ablitateLmHead: false, and enforce a refusal norm/perplexity gate. Do not reintroduce all-layer abliteration.

6.6 RTN fallback quality gate

GPTQModel uses round-to-nearest (RTN) quantization when calibration does not exercise a sparse module enough to build a useful Hessian. This is expected for a small fraction of MoE experts, but a large fraction means the artifact no longer represents the requested GPTQ calibration.

After a fresh run, FlexInfer writes ${OUT_DIR}/.quantization-quality.json, emits a quantization_quality event, and fails the Job when the RTN fallback percentage exceeds the configured limit. The default gate applies after 100 completed GPTQ modules and allows at most 10% RTN fallbacks. Configure the controller with:

FLEXINFER_GPTQ_MAX_RTN_FALLBACK_PERCENT to change the percentage limit.
FLEXINFER_GPTQ_MIN_MODULES_FOR_FALLBACK_GATE to change the minimum sample.

The cached-artifact short circuit enforces the same sidecar, so a failed gate cannot pass on the next Job retry merely because .save-complete already exists. Older cached artifacts without a quality sidecar remain loadable but emit quantization_quality_unavailable; validate those from the persisted quantization log before promotion. Passing this guardrail does not replace the ModelExperiment coherence and long-context kill-tests.

7. Alert response matrix

These alerts ship from charts/flexinfer/templates/prometheusrule.yaml when alerting.enabled=true.

Alert	First check	Action
`FlexInferQuantizationDeadlineApproaching` (>80% of deadline)	`JobProgressPercent` vs `QuantizationLayerIndex`. If layer count is >75% of total, the time estimate is accurate — let it finish. If layer count is much lower, job is genuinely slow.	Consider bumping `timeoutSeconds` before the critical alert fires.
`FlexInferQuantizationDeadlineCritical` (>95%)	Same as above.	Either extend timeout now (§6.1) or let it die and accept the restart cost.
`FlexInferQuantizationJobFailed`	`kubectl describe job` + pod termination message	Classify: OOM → §6.2, DeadlineExceeded → §6.1, HIP error → §4.
`FlexInferQuantizationStalled` (45 min no layer progress)	Hessian events, §6.3	Usually just waiting on floor-attempt exhaustion or a genuinely slow module. If real stall (no CPU activity), restart the Pod.

8. Known limitations (as of today)

Mid-run resume: end-to-end shipped behind a flag (Phase A writer + Phase B reload). As of MR !163 (writer) and MR !165 (reload), completed decoder layers are persisted to .flexinfer-gptq-cache/layers/layer-NNN.safetensors AND swapped back into the live model on job restart so the GPTQModel looper's find_modules filter naturally skips them (it only walks nn.Linear/Conv1d/Conv2d — pre-swapped BaseQuantLinear children yield an empty subset). Controlled by FLEXINFER_GPTQ_RESUME_LAYERS (default true since 2026-06-11 — the llama33-70b build lost three ~layer-8 attempts to full restarts; set false on the controller to opt out). Graceful-fallback semantics: any failure during reload wipes the cache and proceeds with a clean full re-quantize — resume NEVER blocks a run.
Save-phase resume: marker shipped (MR !163). quantize_gptq.py now writes ${OUT_DIR}/.save-complete with a shard manifest (name / size / sha256) before the atomic rename. On job restart, the bash short-circuit verifies each shard's on-disk size against the manifest and skips save entirely when it matches. Partial saves (marker missing or size mismatch) still fall through to a full re-save — true per-shard resume inside save_quantized is NOT shipped. Mitigation for mid-save OOM is still to size memory correctly up front (§5).
make deploy-quantizer-full BuildKit stale content. Observed once — the remote builder shipped a quantizer image containing stale script content despite --no-cache. MR !160 added a post-build md5 parity check that catches this before push. If you see ERROR: Image script content mismatch, the check did its job; retry the deploy or docker system prune on the remote context.

8.1 Resume artifact layout

Unless FLEXINFER_GPTQ_RESUME_LAYERS=false, the PVC gains:

${MODEL_DIR}/.flexinfer-gptq-cache/
├── checkpoint.json                  # existing — progress + resume_config_hash
├── calibration-examples.pt          # existing — cached calibration tensors
└── layers/
    ├── manifest.json                # {version, config_hash, script_version, layers: [...]}
    ├── layer-000.safetensors        # packed int4 state for decoder layer 0
    ├── layer-001.safetensors
    └── ...

manifest.json is the source of truth; orphan layer-*.safetensors files (write succeeded, manifest update failed) are GC'd on each startup. A config_hash mismatch wipes the cache before any write.

${OUT_DIR}/.save-complete is written unconditionally (not flag-gated) on any v13+ run:

{
  "version": 1,
  "script_version": "v14",
  "completed_at": "2026-04-21T18:00:00Z",
  "shards": [
    {"name": "model-00001-of-00007.safetensors", "size_bytes": 4123456789, "sha256": "..."}
  ],
  "has_index": true,
  "has_config": true,
  "has_quantize_config": true
}

The bash short-circuit uses this to fast-path on job restart. Pre-v13 artifacts without the marker still fast-path via the legacy du-based heuristic.

8.2 Reload path events (Phase B)

Look for these Loki events to confirm resume behavior on a restarted job:

Event	Meaning	Action
`quantization_layer_cached`	Phase A writer persisted a layer (per layer, once)	Normal — cache is growing
`quantization_resume_loaded`	Phase B reload succeeded; N layers swapped before quantize	Expected on restart with a valid cache
`quantization_resume_fallback`	Reload failed for any reason; cache was wiped	Check the `reason` field; full re-quant will run normally
`quantization_layer_cache_reset`	Cache invalidated (config hash changed, or fallback-triggered)	Normal on config changes; investigate if unexpected

{namespace="flexinfer-system"} |= "quantization_resume"

Important safety guarantee: quantization_resume_fallback is NEVER fatal — any reload error path ends in a cache wipe + full re-quantize, never a job failure. If you see this event, the run will still complete correctly (just slowly, as if resume were disabled).

docs/design/quantization-pipelines.md — architecture.
build/scripts/quantize_gptq.py — the actual Python quantize driver (Hessian-recovery patch starts around patch_gptq_hessian_inverse).
pkg/quantization/gptq.go — controller-side Job construction.
controllers/modelcache_quantization.go — reconcile loop, image-drift handling, telemetry.
docs/dev/qwen35-gptq-root-cause.md — Qwen3.5 corrupted-source incident and recovery rationale.
MR !156 — Hessian recovery on gfx906.
MR !160 — image-drift grace window + deploy-script md5 check.
MR !161 — timeoutSeconds CRD cap raised to 48 h.
MR !162 — per-layer progress metric + alerts + this runbook.
MR !163 — .save-complete marker + per-layer state writer (Phase A).
MR !165 — per-layer reload path + looper-skip via natural find_modules filter (Phase B).

GPTQ Quantization Runbook

GPTQ Quantization Runbook

1. What this covers

2. Pipeline at a glance

3. What GPTQ actually does (and why it's so slow)

4. Hessian recovery — what, why, when

The problem

The recovery loop (see build/scripts/quantize_gptq.py:_patched_hessian_inverse)

Why the defaults are what they are

HIP error cascade (gfx906 specific)

5. Picking timeouts and memory for a new model

Expected per-layer wall time

timeoutSeconds rule of thumb

maxMemoryGB rule of thumb

Node-selection gotcha

6. Recovering a stuck / failed ModelCache

6.1 The ModelCache is Failed but the run got ~90% through

6.2 The ModelCache OOMKilled during save

6.3 Quantize is running but stuck (no layer progress for 30+ min)

6.4 Job keeps getting recreated right after start

6.5 Qwen3.5 recovery after unsafe abliteration

6.6 RTN fallback quality gate

7. Alert response matrix

8. Known limitations (as of today)

8.1 Resume artifact layout

8.2 Reload path events (Phase B)

9. Related reading

The recovery loop (see `build/scripts/quantize_gptq.py:_patched_hessian_inverse`)

`timeoutSeconds` rule of thumb

`maxMemoryGB` rule of thumb

6.1 The ModelCache is `Failed` but the run got ~90% through