Loom Core docs

Quick status

Loom Mills — Operator Runbook

Day-2 procedures for loom-mills-operator. For architecture and policy reference, see docs/MILLS.md.

All kubectl commands assume KUBECONFIG=~/workspace/platform/gitops/.kube/k3s.yaml (or kc-k3s alias). All Mac CLI examples assume LOOM_MILLS_OPERATOR_URL and LOOM_ADMIN_TOKEN are exported. The operator runs in namespace loom-mills.

Quick status

loom mills status                                           # one-liner from Mac
kubectl get deploy,po,pvc -n loom-mills                     # cluster snapshot
kubectl logs -n loom-mills deploy/loom-mills-operator --tail=200
curl -sf $LOOM_MILLS_OPERATOR_URL/readyz                    # 200 once initialized
curl -sf $LOOM_MILLS_OPERATOR_URL/api/mills/capabilities | jq
curl -sf $LOOM_MILLS_OPERATOR_URL/metrics | grep '^loom_mills_'

/readyz is service readiness: the HTTP process, migrations, and policy manager are initialized. It is not an autonomy approval signal. Before enabling or trusting unattended writes, check /api/mills/status or /api/mills/capabilities and require autonomy_ready=true. When this is false, the scheduler may continue ticking for observability, but the reconciler will not start queued backlog items.

Common autonomy blockers are intentional fail-closed states: missing admin auth, missing or unwritable LOOM_MILLS_REPO_ROOT/.loom, unwired FlexInfer/GitLab/HUD spawn/MCP hub dependencies, NoOp dispatcher coverage on write stages, or fake council participants. Read-only APIs may stay healthy while these blockers are present.

In k3s, LOOM_MILLS_REPO_ROOT=/workspace/loom-core is backed by the same Longhorn RWO PVC as /var/lib/loom-mills. On boot, the operator uses the GitLab project token to clone or fast-forward that checkout. If credentials, network, git, or the checkout itself fail, the pod should still become service-ready, but /api/mills/capabilities must show repo_root red and autonomy_ready=false.

Pipeline-start recovery

A queued item is admitted with its run, workflow identity, capacity reservation, transition ledger row, and dispatch intent in one SQLite transaction. The starter runs only after that transaction commits. A pod crash between those steps is self-healing: the next reconciler tick leases the committed intent or recovers an acknowledged-but-still-queued top-level run and drives the same run ID forward.

Use the outbox gauge and structured events to distinguish a brief restart drain from a stuck starter:

curl -sf $LOOM_MILLS_OPERATOR_URL/metrics | \
  grep '^mills_pipeline_dispatch_outbox_pending '
kubectl logs -n loom-mills deploy/loom-mills-operator --since=30m | \
  grep -E 'reconciler\.(started|dispatch_dead_lettered|dispatch_ack_failed)'

The gauge may rise briefly during a restart and should then fall. Delivery is at-least-once by design, with the run ID as the starter idempotency key. Do not delete outbox or reservation rows to force progress: lease fencing prevents a late consumer from acknowledging another consumer's work, and manual deletion can hide an unaccepted start.

After the bounded retry ceiling, Mills marks the intent dead_letter, escalates the current run/backlog aggregate, synchronizes the workflow terminal state, and releases the reservation atomically. Investigate the associated reconciler.dispatch_dead_lettered event and starter configuration, then use the normal human-resolution/requeue path. Obsolete intents from an older aggregate are retired without escalating a newer run.

The operator remains a singleton. These recovery guarantees cover process restart under SQLite WAL; they do not provide multi-replica worker fencing.

Updating backlog items safely

POST /api/mills/backlog creates an item when its ID is new. Updating an existing ID requires the Revision returned by the latest backlog GET/list response. A stale write returns HTTP 409 with {"error":"stale-backlog-write"}. Re-read the item, merge the intended metadata change, and retry with the new Revision; do not blindly replay the old body because it may overwrite a lifecycle transition.

Pause and resume

The kill switch is policy.enabled: false. Edits to the mounted ConfigMap propagate via fsnotify within seconds; in-flight runs continue under their captured policy, but no new runs start.

Pause

# Edit platform/gitops/k3s/mills/configmap-policy.yaml in the gitops repo:
#   enabled: false
# Then reconcile:
flux reconcile kustomization apps -n flux-system --with-source

# Wait for hot-reload signal in operator logs:
kubectl logs -n loom-mills deploy/loom-mills-operator --since=2m | grep "policy reloaded"

# Confirm the reconciler is parked:
loom mills status                # expected: enabled=false, queue_depth=0 (no new picks)

The reconciler exits cleanly within one tick (≤60s). The HTTP and metrics listeners stay up so HUD reads continue working.

Resume

Reverse the edit (enabled: true or remove the field), reconcile, watch for the next policy reloaded log.

Emergency pause without GitOps

Only when GitOps is unavailable (e.g., Flux is down). Patches the mounted ConfigMap directly; must be reverted by a GitOps commit afterwards or Flux will fight you.

kubectl patch configmap -n loom-mills loom-mills-policy \
  --type merge -p '{"data":{"policy.yaml":"<full yaml with enabled: false>"}}'
# Restore via git/Flux as soon as the issue is resolved.

Force-escalate a stuck pipeline run

A run can wedge if a stage worker hangs (e.g., ci_watch waiting on a CI job that's been canceled). Force-escalation transitions the run to escalated, records a failure record in events, and (if configured) opens a GitLab issue + agent-context handoff.

# Find the run
loom mills pipelines list --state=running

# Escalate
curl -sf -X POST -H "Authorization: Bearer $LOOM_ADMIN_TOKEN" \
  $LOOM_MILLS_OPERATOR_URL/api/mills/pipeline/runs/<run_id>/escalate

# Confirm
loom mills pipelines list --state=escalated

The reconciler will not auto-retry escalated items; a human must edit the linked YAML or close the GitLab issue with a deliberate human-resolved label to unblock.

Pause / resume / fail a workflow run

Imperative workflow runs (the durable-journal runtime behind policy.workflows.enabled) have their own lifecycle mutations, separate from pipeline escalation. Use these to stop a wedged run live — e.g. a zombie whose spawn died terminally (the 2026-07-09 wf-canary loop) — instead of waiting for a code deploy.

# Find the run
curl -sf $LOOM_MILLS_OPERATOR_URL/api/mills/workflow/runs | jq '.runs[] | {id, state}'

# Pause (running → paused; takes effect between steps, on the next scheduler tick)
curl -sf -X POST -H "Authorization: Bearer $LOOM_ADMIN_TOKEN" \
  $LOOM_MILLS_OPERATOR_URL/api/mills/workflow/runs/<run_id>/pause

# Resume (paused → running)
curl -sf -X POST -H "Authorization: Bearer $LOOM_ADMIN_TOKEN" \
  $LOOM_MILLS_OPERATOR_URL/api/mills/workflow/runs/<run_id>/resume

# Fail (running|paused → error; TERMINAL — no un-fail. Optional audit reason.)
curl -sf -X POST -H "Authorization: Bearer $LOOM_ADMIN_TOKEN" \
  -d '{"reason":"zombie spawn, pod gone"}' \
  $LOOM_MILLS_OPERATOR_URL/api/mills/workflow/runs/<run_id>/fail

Expected output: the run's lifecycle view, e.g. {"id":"wf-canary-…","state":"paused","paused_at":"…"}. An invalid transition returns 409 with the current state. Pause/fail are eventually-consistent like the policy kill switch: an in-flight step finishes its current attempt first; the scheduler skips the run from the next tick.

Replay a council run

Useful when you want to verify a fix to the brief assembler, swap an ensemble member, or compare A/B outputs.

Dry-run (safe)

Runs the full council pipeline against a scratch DB; nothing is committed and no GitLab mutations happen. The sidecar + plan paths are returned for inspection.

loom mills council dryrun
# Output: sidecar JSON path, .loom/<NN>-… draft paths, total cost

Real replay with a different ensemble

# 1. Edit policy.council.ensemble in platform/gitops/k3s/mills/configmap-policy.yaml
#    e.g., swap editor.model from claude-opus to codex-gpt5.
# 2. flux reconcile.
# 3. Wait for "policy reloaded" log line.
# 4. Trigger:
loom mills council run

# 5. Inspect the new run:
loom mills eval list --subject=council
# HUD: Mills → Eval panel; sort by created_at desc.

When the run lands, compare cost + Loop A score + downstream Loop B attribution against the previous run with the same brief content (same roadmap_intents snapshot).

V2.1 will add first-class A/B replay UI; until then, the manual flow above is the supported path.

Audit a merged change

Find which council brief produced the slice that produced the merge:

# Get the pipeline run by MR iid:
curl -sf -H "Authorization: Bearer $LOOM_ADMIN_TOKEN" \
  "$LOOM_MILLS_OPERATOR_URL/api/mills/pipeline/runs?mr_iid=<iid>"

# Eval Loop B attribution shows the originating council run:
curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/eval?subject_kind=pipeline_run&subject_id=<run_id>"

# Loop A score for the council run that planned it:
curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/eval?subject_kind=council_run&subject_id=<council_run_id>"

For deeper inspection, the events table has the full per-stage trace (slog rows are also written via JSON to stderr; aggregated in Loki).

Recover from a corrupted DB

The canonical SQLite DB lives on a Longhorn RWO PVC. WAL replay handles most operator restarts; nothing should be needed for a clean kill. The procedures below are for the rare cases where the DB is unrecoverable.

Symptom: `database is locked` or schema check fails on boot

Capture diagnostics:

kubectl logs -n loom-mills deploy/loom-mills-operator --previous --tail=500 > /tmp/mills-prev-logs.txt
kubectl exec -n loom-mills deploy/loom-mills-operator -- sqlite3 /var/lib/loom-mills/state.db "PRAGMA integrity_check;"

If integrity_check returns anything other than ok, restore from the most recent nightly backup in MinIO:

# List backups (nightly CronJob writes one per day):
mc ls minio/loom-mills-backups/

# Pick a backup to restore:
BACKUP=2026-04-30T06-00-00.db

# Scale operator to 0 to release the PVC:
kubectl scale deploy -n loom-mills loom-mills-operator --replicas=0

# Restore via a one-shot Job (the manifest lives at
# platform/gitops/k3s/mills/jobs/restore-from-backup.yaml; render with the
# chosen filename):
kubectl create -n loom-mills -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  generateName: mills-restore-
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: restore
          image: registry.harbor.lan/library/loom-mills-operator:stable
          command: ["/bin/sh", "-euxc"]
          args:
            - |
              mc cp minio/loom-mills-backups/$BACKUP /var/lib/loom-mills/state.db
              sqlite3 /var/lib/loom-mills/state.db 'PRAGMA integrity_check;'
          volumeMounts:
            - { name: state, mountPath: /var/lib/loom-mills }
      volumes:
        - name: state
          persistentVolumeClaim:
            claimName: mills-state
EOF

# Watch the restore Job to completion:
kubectl logs -n loom-mills job/<job-name> -f

# Bring the operator back up:
kubectl scale deploy -n loom-mills loom-mills-operator --replicas=1
kubectl rollout status deploy -n loom-mills loom-mills-operator

After restore, run a council dryrun to confirm the brief assembler still works against the restored canonical state. Then resume normal operation.

Loss-window note

The nightly CronJob captures one snapshot per day. Items committed since the last snapshot are lost in a restore. Re-run the council to reproduce — roadmap_intents is idempotent and .loom/backlog/*.yaml is regenerated from the canonical store, so most state self-heals.

Inspect what the operator is doing right now

# Active runs (council + pipeline):
curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/status" | jq

# What gates fired in the last hour:
curl -sf "$LOOM_MILLS_OPERATOR_URL/api/mills/pipeline/runs?since=1h" | \
  jq '[.runs[].gate_outcomes[]] | group_by(.gate) | map({gate: .[0].gate, count: length})'

# What the budget enforcer thinks:
curl -sf "$LOOM_MILLS_OPERATOR_URL/metrics" | grep '^loom_mills_budget'

Production rollout staging

The default-on flip (slice 6.6) was landed for the in-binary policy default. Each cluster overlay still owns whether the mills is enabled in its ConfigMap. Stage rollouts in this order; never flip more than one environment per week.

Stage 1 — Local

A developer can run the operator against a local SQLite file and a local policy YAML for the smallest possible smoke test:

make build/loom-mills-operator
mkdir -p /tmp/loom-mills
./bin/loom-mills-operator \
  --db-path     /tmp/loom-mills/state.db \
  --policy-path testdata/policy.yaml \
  --listen      :8090 \
  --metrics-addr :9090

curl -sf localhost:9090/healthz                            # 200
curl -sf -H "Authorization: Bearer $LOOM_ADMIN_TOKEN" \
  localhost:8090/api/mills/status | jq

Local runs use the FakeReviewer/FakeEditor (cmd/loom-mills-operator/main.go: buildCouncilRunner) when no FlexInfer or HUD spawn is configured; this is sufficient for handler smoke tests but does not exercise real model calls.

Stage 2 — Dev cluster (kc-k3s)

# In the platform/gitops repo on a dev branch:
# Edit platform/gitops/k3s/mills/configmap-policy.yaml: enabled: true
# Commit, push, open MR, merge.

flux reconcile kustomization apps -n flux-system --with-source
kubectl rollout status deploy -n loom-mills loom-mills-operator

Soak for one week. Verify each acceptance criterion against the deployed cluster:

Criterion	Check
Council dryrun produces sidecar + 3 markdown docs in <8 min for <$5 with eval ≥0.7	`loom mills council dryrun` and inspect cost/score in HUD
End-to-end backlog → merged MR via fixture run	`loom mills backlog list --state=merged` after seeding a fixture
Eval Loops A/B/C populated	`loom mills eval list` returns recent rows for all three subject kinds
Idle-throttle drops reconciler cadence	`loom mills status` shows `next_tick_in_s` ≥ 240 when queue is empty
HUD `Mills` view renders four panels	Visit `<HUD>/mills`
Backups produced nightly	`mc ls minio/loom-mills-backups/` shows last 7 entries

Stage 3 — Production cluster

Only after dev has been green for 7 consecutive days. Same edit + reconcile flow, but watch the regression KPI:

watch -n 30 'curl -sf $LOOM_MILLS_OPERATOR_URL/metrics | grep loom_mills_regression'

If loom_mills_regression_count_total increases above zero in the first 24h, pause and investigate before any further flips. The kill switch (enabled: false) is the safest and lowest-blast-radius rollback.

GitLab issue importer (Slice 1a of .loom/43)

The operator can pull open issues labelled mills-eligible from its configured GitLab project (GITLAB_PROJECT env) and create one BacklogItem per fresh issue. Disabled by default; opt in via policy.

intake:
  gitlab:
    enabled: true
    eligible_label: "mills-eligible"   # required label on the issue
    poll_interval_seconds: 300         # 5 min
    default_priority: "P2"             # fallback if no priority:Px label

Label semantics on the issue side:

mills-eligible (required) — the importer's selector. Add this on any issue you want Mills to pick up.
priority:P0 / priority:P1 / priority:P2 / priority:P3 (optional) — sets the BacklogItem priority. Highest priority wins when multiple are present. Default is default_priority.

Dedup: BacklogItem ID is gl-<project_id>-<issue_iid>, so the importer is safe to re-run. Once an item exists, the importer does NOT update it — reconciler/council state transitions (queued → running → merged/escalated) are preserved across ticks.

Verify:

# After enabling, watch for the import line in operator logs:
kubectl -n loom-mills logs deploy/loom-mills-operator -f | \
  grep "gitlab importer"
# Check the backlog API for the new item:
loom mills backlog list | grep gl-

Don't enable until pipelines are reaching merge stage. Per the 2026-05-24 kill-test (.loom/local/handoffs/mills-autonomy-killtest-...) 100% of pipeline_runs were escalating at tests or plan_slice. Adding more intake before Slices 2c + 2e land would just deepen the escalation pile.

Source-branch push before MR

The mr stage publishes the spawn agent's commits to origin before calling GitLab CreateMR. Without this, GitLab accepts the MR row but it points at a branch with no head_sha, and ci_watch hangs forever waiting for a pipeline that can't exist.

There are TWO push paths (belt-and-suspenders):

Spawn-side push (primary): the implement stage prompt explicitly instructs the agent to git push -u origin HEAD as its final step. The spawn pod has git credentials configured. This is the path that actually fires today because the single- repo SpawnWorker does NOT use Mills' WorktreeAllocator — so the operator doesn't know the worktree path and can't push from outside the pod.
Operator-side push (fallback, currently inert): when jc.Run.WorktreePath is set, GitLabWorker.runMR runs the clients.GitBranchPusher before CreateMR. This path is exercised in the cross-repo integrator flow where Mills does allocate worktrees explicitly. A future slice should propagate the spawn-side WorkingDir back through SpawnResponse so the fallback also kicks in for single-repo runs.

Implementation: pkg/mills/clients/branch_pusher.go (clients.GitBranchPusher) shells git push --force-with-lease -u origin HEAD:<branch> from the run's WorktreePath.
Interface: pipeline.BranchPusher (4-line contract). The operator wires the production pusher onto GitLabWorker.BranchPusher at startup; tests inject fakes.
Idempotency: --force-with-lease lets retries safely overwrite a stale prior push without clobbering unrelated upstream work. A no-op push (HEAD already at origin) exits 0.

Failure mode (push errors before MR is created): the mr stage returns an error and the runner retries per the Slice 2c classification — a transport / quota error gets free retries; a real git push rejection counts against MaxAttempts.

v2 rollout staging (preview)

When .loom/94-implementation-plan-mills-v2-hierarchical-swarm-2026-05-02.md Phase 8 lands, each v2 feature flips behind its own policy flag with a 1-week soak between flips. Order:

policy.squads.enabled: true
policy.audit.enabled: true (advisory-only by default)
policy.council.debate.enabled.incident: true
policy.cross_repo.enabled: true — only after 3 successful loom-core+loom dogfood atomic merges
policy.adaptive_policy.enabled: true (manual-apply only)

Detail per slice in .loom/94-…2026-05-02.md Phase 8. The rollback playbook for v2 lives in MILLS_V2_ROLLBACK.md — covers feature flag disable, policy proposal revert, and DB restore from MinIO.

Useful loops

# Watch council runs as they complete:
watch -n 30 'loom mills eval list --subject=council | head -10'

# Watch escalation rate:
watch -n 60 'kubectl logs -n loom-mills deploy/loom-mills-operator --since=10m | jq -r "select(.msg==\"escalation\") | [.time, .item_id] | @tsv"'

# Diff the live policy vs. git:
kubectl get configmap -n loom-mills loom-mills-policy -o jsonpath='{.data.policy\.yaml}' | \
  diff - platform/gitops/k3s/mills/configmap-policy.yaml

Spawn pool capacity & multi-repo

Mills pipeline stages (plan_slice / research / implement) run as spawn pods. The spawn orchestrator is embedded in mobile-hud (internal/hud/spawn.go), not the operator — the operator only POSTs a spawn request to the HUD. So the spawn knobs are env vars on the mobile-hud Deployment (k8s/base/servers/mobile-hud/deployment.yaml), which deploys via the Flux loom-hub-servers Kustomization. The gitops override patch (platform/gitops/clusters/k3s/flux-system/kustomization-loom-hub-servers.yaml) does not touch these, so the base-manifest values are authoritative.

Env var	Meaning	Current
`SPAWN_MAX_CONCURRENT`	Global cap on active spawn pods (mobile-hud is a single replica, so this is a fleet-wide ceiling).	`10`
`SPAWN_MAX_CONCURRENT_BUILDS`	Concurrent spawn image builds (buildah); throttled independently so a higher run cap can't cause a build thundering herd.	`1`
`SPAWN_DEFAULT_CPU` / `SPAWN_DEFAULT_MEMORY_MB`	Per-spawn resource limits.	`0.5` / `4096`
`SPAWN_SYNC_MODE`	Workspace sync mode. `git-clone` = the pod clones the repo fresh (source of truth); the workspace mount is used only to fingerprint the repo for the Dockerfile.	`git-clone`
`SPAWN_GIT_BASE_URL`	Git base the pod clones from; rooted at the `services/` group.	`http://192.168.50.218/services`
`SPAWN_PROJECTS`	Comma-separated repos shown in the HUD/mobile spawn picker. Cosmetic — not an enforcement gate.	see manifest

Raising concurrency. Cluster allocatable memory is ~1.25 TiB (16 nodes), so even SPAWN_MAX_CONCURRENT at 10 (= 5 CPU + 40 GiB) is <4% of capacity. Capacity is not the constraint — demand is. Bump the value in the base manifest and let Flux roll mobile-hud; no gitops-patch change is needed.

Expanding to more repos. A stage targets a repo via the pipeline worker's Project (defaults to loom-core). For a repo to spawn cleanly:

Merge token reach — the Mills GitLab worker uses the services-group token (loom-mills-gitlab-group/api-token, Maintainer on group services), so only services/* repos can be opened+merged. Cross-group repos (libs/, platform/) need a separate token grant.
Git-clone group — the pod clones SPAWN_GIT_BASE_URL/<name>.git, i.e. the services/ group. Non-services repos would need a group-aware base URL.
Workspace fingerprint (best-effort) — resolveProjectPath looks for the repo on the loom-hub-workspace PVC (Longhorn RWO, mounted read-only) to fingerprint it for an accurate runtime image. In git-clone mode a repo that is not staged on the PVC no longer hard-fails: the orchestrator falls back to a lexical services/<name> path and a generic runtime image, and the init-container clone provides the real source. Staging the repo on the PVC still yields a better-fingerprinted image (correct language toolchain), so for build-heavy repos prefer to stage it. Populating/resizing that RWO PVC is a coordinated gitops op (single-writer; co-locate the writer with mobile-hud).

Multi-repo demand activation (S6)

The steps above make a stage able to run against a services/* repo. S6 makes demand itself target foreign repos: the plan-slice emitter sources an allowlist of non-home projects and stamps each emitted item's TargetProject, so the S2/S3/S4 routing carries the whole run cross-repo. Activation is two-key — both must be true or foreign demand stays inert:

Execution key — policy.cross_repo.enabled: true. Already flipped 2026-07-05 for the keystone (flexdeck!244). This alone changes nothing about demand; it only lets a foreign-targeted item execute instead of being skipped fail-closed by the reconciler.
Demand key — policy.cross_repo.demand_projects: [services/<repo>, …]. The emitter consults this list ONLY when enabled is also true (Policy.CrossRepoDemandProjects), so a stray allowlist can never source foreign demand while execution is off.

Procedure to onboard a repo (e.g. services/flexdeck):

Branch pipeline — the target repo must run a pipeline on the MR's source branch (flexdeck did this via flexdeck!242) so ci_watch has a green gate to merge on. Without it the autonomous merge can't confirm success.
Set the allowlist — add the repo to cross_repo.demand_projects in the gitops policy ConfigMap (platform/gitops/k3s/mills/…).
Roll the operator — the emitter snapshots policy at startup and fsnotify misses ConfigMap ..data swaps, so bump the deployment pod-checksum (or rollout-restart) to pick up the new allowlist.
Author demand — the emitter emits from Plans scoped to the target project that have a ready (plan_slice_emitter.ready_phase, default pending) slice with declared files, in the emitter's namespace (plan_slice_emitter.namespace). A foreign repo with no such Plan produces nothing; author a services/<repo> Plan with a ready slice to drive a run.
Verify — operator log plan-slice emitter created backlog item … demand, the item's pipeline runs against the target repo, an MR opens+merges in the target repo, and mills_autonomous_merges_real increments.

Rollback — remove the repo from demand_projects (stops new foreign demand) or set cross_repo.enabled: false (also fail-closes any already-queued foreign item on the next reconciler tick), then roll the operator.

Sources

cmd/loom-mills-operator/main.go (lifecycle, env vars, fakes vs. wired clients)
cmd/loom-mills-operator/auth.go (admin token loading)
pkg/mills/policy.go (enabled, IsEnabled, kill-switch semantics)
pkg/mills/policy_manager.go (fsnotify hot-reload)
pkg/mills/reconciler.go (one-tick exit on disable)
pkg/mills/scheduler.go (cron + idle throttle)
pkg/mills/eval/{judge,outcome_attributor,council_roi,cross_run}.go (Loops A/B/C)
pkg/mills/store/migrate.go (migrations on boot; safe replay)
platform/gitops/k3s/mills/ (manifests; not in this repo — lives in the gitops repo)
docs/MILLS.md (architecture + policy reference)
.loom/91-implementation-plan-agent-swarm-council-pipeline-2026-04-25.md §"Default-off rollout"

Quick status

Loom Mills — Operator Runbook

Quick status

Pipeline-start recovery

Updating backlog items safely

Pause and resume

Pause

Resume

Emergency pause without GitOps

Force-escalate a stuck pipeline run

Pause / resume / fail a workflow run

Replay a council run

Dry-run (safe)

Real replay with a different ensemble

Audit a merged change

Recover from a corrupted DB

Symptom: database is locked or schema check fails on boot

Loss-window note

Inspect what the operator is doing right now

Production rollout staging

Stage 1 — Local

Stage 2 — Dev cluster (kc-k3s)

Stage 3 — Production cluster

GitLab issue importer (Slice 1a of .loom/43)

Source-branch push before MR

v2 rollout staging (preview)

Useful loops

Spawn pool capacity & multi-repo

Multi-repo demand activation (S6)

Sources

Symptom: `database is locked` or schema check fails on boot