Mills Operational Guardrails

Operator-facing guidance for keeping Loom Mills in a safe state when telemetry, storage, or authentication dependencies degrade.

This document complements docs/MILLS.md and docs/MILLS_RUNBOOK.md. Use it when /api/mills/status, /api/mills/capabilities, HUD, or alerts show that unattended Mills work may no longer be trustworthy.

Guardrail model

Mills composes several practical safety layers:

Layer	Where to check	Safe state
Policy kill switch	policy.enabled in the Mills policy ConfigMap	false closes work-creating REST admissions and background schedulers.
Quiescence proof	GET /api/mills/safety/quiescence	all durable and in-memory activity counts are zero before maintenance or shared-pod fault injection.
Destructive-action lease	POST /api/mills/safety/crash-lease + token renewal	blocks new admissions across the last quiescence read, identity checks, and bounded UID delete.
Capability breaker	GET /api/mills/capabilities	Any required red/stub row makes autonomy_ready=false.
Pipeline continuation breaker	Pipeline stage events and escalation reason	A running pipeline checks autonomy before each autonomous stage; a blocked verdict escalates the item instead of continuing toward MR, CI watch, or merge.

/readyz only says the service initialized. It is not an autonomy signal. Treat autonomy_ready=false as the authoritative fail-closed state for unattended writes.

Breaker thresholds

These thresholds are the operator-facing defaults to know during incident response.

Guardrail	Default threshold	Failure behavior	Operator action
Mills policy kill switch	policy.enabled=false	New queued work is not picked up. In-flight runs continue under the policy captured at start, then hit the continuation breaker if readiness is blocked.	Use for any ambiguous telemetry, storage, or auth incident where autonomous safety cannot be proven.
Capability readiness	All required rows green/real	autonomy_ready=false; blockers list the red/stub capability rows.	Remediate the named row, then verify /api/mills/capabilities.
Pipeline retry policy	max_attempts: 3, cooldown_seconds: 300	Retryable stage failures cool down and retry; exhausted items escalate.	Do not manually restart the same failure loop until the dependency is fixed.
Pipeline budget policy	$5 per pipeline run, $75 per day, 4 concurrent runs, 20 runs/day	Budget/concurrency gates stop new work from exceeding policy.	Raise only by policy change with an explicit reason and rollback point.
Council budget policy	$15 per council run, $50 per day	Council demand generation is held when budget is exhausted.	Check whether queue starvation is budget-driven before treating it as a model or storage failure.
Local MCP recv timeout breaker	3 consecutive local recv timeouts	The daemon tears down the stale local stdio transport so the next request can reconnect.	Inspect server logs after the third timeout; one timeout alone is not enough to declare the server dead.
Embedding provider breaker	3s per call, 3 consecutive failures, 30s cooldown	Embedding calls fail fast with embedder unavailable; callers should use fallback/non-vector paths where implemented.	Fix the provider or route around it; expect degraded semantic search until recovery.

Safe state procedure

1. Freeze new autonomous work:

   # In platform/gitops/k3s/mills/configmap-policy.yaml:
   #   enabled: false
   flux reconcile kustomization apps -n flux-system --with-source
   kubectl logs -n loom-mills deploy/loom-mills-operator --since=2m | grep "policy reloaded"

   Apply the ConfigMap-only closure first. Do not update a Deployment checksum or otherwise restart the operator until quiescence is proven; rollouts happen behind the already-closed barrier.

2. Confirm Mills is fail-closed:

   curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/status" | jq '{policy_enabled, autonomy_ready, autonomy_blockers}'
   curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/capabilities" | jq
   curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/safety/quiescence" | jq
   loom mills pipelines list --state=running

3. If a running pipeline is already past the point you trust, force-escalate it:

   curl -sf -X POST -H "Authorization: Bearer $LOOM_ADMIN_TOKEN" \
     "$LOOM_MILLS_OPERATOR_URL/api/mills/pipeline/runs/<run_id>/escalate"

4. Remediate the dependency class below.

5. Resume only after /api/mills/capabilities shows autonomy_ready=true, recent logs are clean, and any escalated item has a human disposition.

Telemetry failures

Telemetry failures are visibility failures first. They become autonomy blockers when they hide required capability state or make it impossible to verify what Mills is doing.

Common symptoms:

Symptom	Likely source	Safe interpretation
/metrics missing loom_mills_* series	Metrics listener, scrape, or KPI writer path	Mills may still run, but trend-based decisions are blind. Pause if you cannot inspect status another way.
/api/mills/kpis?window=1d returns 404 immediately after restart	KPI writer has not recorded its first post-start snapshot	Wait one scheduler tick, then recheck. Persistent 404 is a telemetry incident.
HUD shows stale Mills panels while CLI/status endpoints are fresh	HUD proxy/cache path	Prefer direct operator endpoints during the incident.
Missing cost or spawn telemetry on a failed stage	Spawn/harness telemetry path	Treat the stage result as incomplete; inspect the run event log and spawn logs before retrying.

Remediation workflow:

1. Check direct operator surfaces before derived dashboards:

   curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/status" | jq
   curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/capabilities" | jq
   curl -sf "$LOOM_MILLS_OPERATOR_URL/metrics" | grep '^loom_mills_' | head
   kubectl logs -n loom-mills deploy/loom-mills-operator --tail=200

2. If direct status is healthy but HUD is stale, keep Mills paused only if operators depend on HUD for approval. Otherwise continue using direct status during the HUD fix.

3. If KPI snapshots or metrics are missing across more than one scheduler tick, keep or set policy.enabled=false until visibility is restored.

4. After repair, verify a fresh KPI snapshot:

   curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/kpis?window=1d" | jq '{snapshot_at, metrics}'

Storage failures

The canonical store is SQLite on the mills-state Longhorn PVC. Storage is a hard autonomy dependency.

Common symptoms:

Symptom	Breaker row	Safe state
/api/mills/capabilities shows sqlite_store red	sqlite_store	autonomy_ready=false; do not resume until DB liveness and integrity are known.
Operator logs show database is locked, failed migrations, or integrity errors	sqlite_store or policy_loaded	Pause, capture diagnostics, and follow the DB recovery runbook.
Repo root checkout or .loom is missing/unwritable	repo_root	Read-only APIs can stay up, but unattended backlog execution must remain blocked.

Remediation workflow:

1. Capture the current state:

   kubectl get deploy,po,pvc -n loom-mills
   kubectl logs -n loom-mills deploy/loom-mills-operator --previous --tail=500
   kubectl logs -n loom-mills deploy/loom-mills-operator --tail=500
   curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/capabilities" | jq '.autonomy_blockers, .capabilities[] | select(.status!="green")'

2. For DB integrity errors, use the recovery procedure in docs/MILLS_RUNBOOK.md before replaying work.

3. For repo-root errors, verify the mounted checkout:

   kubectl exec -n loom-mills deploy/loom-mills-operator -- \
     sh -lc 'git -C "$LOOM_MILLS_REPO_ROOT" status --short && test -w "$LOOM_MILLS_REPO_ROOT/.loom"'

4. Resume only after sqlite_store, policy_loaded, and repo_root are green and autonomy_ready=true.

Authentication failures

Auth failures are fail-closed. Missing admin auth blocks mutating endpoints; missing or invalid downstream credentials block the required capability rows for autonomous work.

Common symptoms:

Symptom	Breaker row	Safe state
Admin endpoints return 401/403	admin_auth	Mutating endpoints fail closed; rotate or restore LOOM_ADMIN_TOKEN.
MR, CI watch, merge, or cleanup stages fail with GitLab auth errors	gitlab	Autonomy blocked; do not retry until token/project/API URL are fixed.
Spawn stages fail with unauthorized responses	hud_spawn	Plan-slice, implement, and self-review stages cannot be trusted.
Agent-context session cannot initialize	mcp_hub_session	Handoff, plan-slice, and coordination paths are degraded; autonomy blocked.
FlexInfer returns auth/model access errors	flexinfer	LLM-judged gates and research/council paths are not safe for unattended operation.

Remediation workflow:

1. Identify the failing auth boundary from capabilities:

   curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/capabilities" | \
     jq '.capabilities[] | select(.status!="green") | {id, status, mode, config_key, message}'

2. Rotate or restore the specific secret through GitOps when possible. Use direct cluster patches only for an emergency, and follow with a GitOps commit so Flux does not revert the fix.

3. Restart or roll the dependent deployment when the secret is env-backed rather than dynamically read.

4. Verify with a read-only probe first, then a dry run:

   loom mills status
   loom mills council dryrun

5. Resume policy only after the capability row is green and any failed pipeline run has been escalated, retried deliberately, or replaced by a new backlog item.

Resume checklist

Before setting policy.enabled=true again:

• /api/mills/capabilities returns autonomy_ready=true.
• autonomy_blockers is empty.
• loom mills pipelines list --state=running contains only runs you intend to keep.
• Recent operator logs contain no repeated storage, telemetry, or auth failures.
• Any forced escalation has a linked human note, GitLab issue, or agent-context handoff.
• The policy change has been applied through GitOps or reconciled back into GitOps after an emergency patch.