Skip to main content
Lab notebook · services/flexinfer

flexinfer climb

Where flexinfer started, the summit it is working toward, the rules it may not break, and every merge of the last 30 days, each tied to the roadmap waypoint it served or marked unplanned.

Latest leg: Eight unplanned operator merges co-moved with mixed metric shifts on 2026-11-20.

Now climbingchecking

Checking the operator for recent events…

Merge-queue moves and narrated legs appear here when the operator is reachable.

Base, summit and ropes

Base

The problem the climb started from

Self-hosted inference on a mixed homelab fleet (two gfx1100 cards, a Radeon VII and a GTX 980 Ti) breaks in ways nobody measures: lanes that hog a card, crash on load or time out behind a cold start. FlexInfer is the Kubernetes operator and OpenAI-compatible proxy that serves the lab's LLM, embedding, rerank, speech, image, audio and video lanes with scale-to-zero, breakers, reroute and per-GPU certification, climbing toward a fleet where every request the proxy routes is answered and every backend change is certified before it serves.

Source: ROADMAP.md#current-status · climbing since 2026-07-02

Summit

Where it is trying to get to, as measured

Proxy request success ratio (rolling 1d)
98.2%
target ≥ 99.5%
−1.7 pts since Nov 12, away from target
Roadmap gates met (Now list)
50%
target ≥ 100%
2 of 4 checked, 1 verified (1 claimed, unverified)
±0 pts since Nov 21
Master pipeline success ratio (7d)
77.4%
target ≥ 90%
+77.4 pts since Oct 23, toward target
Merges per week
37 merges
target ≥ 40 merges
+27 merges since Oct 23, toward target

Source: .loom/climb.yaml · as of 2026-11-21

Ropes

What must hold while climbing

  • Critical Conventions (read before touching the cluster)

    • Resource hierarchy: act on the parent CR, never the child. Scale/delete the ModelDeployment (not the Deployment), delete the ModelCache (not the ram-syncer DaemonSet), delete the Job (not the benchmark pod) — the controller recreates children. Details: operations.md.
    • ROCm gfx1100 (RX 7900 XTX): PyTorch backends need `TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1` (+ `HSA_OVERRIDE_GFX_VERSION=11.0.0`, `PYTORCH_ROCM_ARCH=gfx1100`) or attention ops SIGSEGV. Injected by `ROCmEnvVars()` in `backend/interface.go`.
    • ROCm gfx906 (Radeon VII): needs `HSA_ENABLE_SDMA=0` or you get memory-aperture violations. Do NOT set `HSA_OVERRIDE_GFX_VERSION` or the AOTriton flag on gfx906. Also via `ROCmEnvVars()`.
    • MLC-LLM quantization: use `q4f32_1` (NOT `q4f16`) for Qwen3 on ROCm — TVM segfault bug (mlc-ai/mlc-llm#3283). Production uses pre-compiled `modelLibPath` + `jitPolicy: "OFF"`; JIT adds 2-5 min startup.
    • Stuck Helm upgrades / rollouts: a NotReady node can leave pods stuck `Terminating`, blocking rollouts. Force-delete the stuck pods in `flexinfer-system`.

    AGENTS.md#critical-conventions-read-before-touching-the-cluster

  • Loom Agent Safety Policy (Generated)

    Canonical nudge for CLI hooks: > Dirty worktree detected. Treat pre-existing changes as baseline context, continue work, and stage/commit only files for the active task. Escalate only if new unexpected changes appear in files you are editing.

    • Pre-existing uncommitted/untracked files are baseline context, not an automatic blocker.
    • Continue on the current branch/worktree by default.
    • Stage and commit only files intentionally changed for the active task.
    • Escalate only when new unexpected changes appear in files you are editing, or when a branch/worktree switch is explicitly requested.
    • Dirty-worktree mode: `continue_scoped_commits`.

    AGENTS.md#loom-agent-safety-policy-generated

  • Resource Cleanup Procedures

    Before scheduling ANY workload to a GPU node, verify these resources are NOT running:

    • To stop a model deployment: Delete or scale the ModelDeployment (not the Deployment)
    • To stop RAM syncers: Delete the ModelCache with `storageStrategy: Memory`
    • To clean up benchmark pods: Delete the Job (pods are owned by Job)
    • From your workstation (before rebooting node):
    • Reboot the node (physically or via IPMI/SSH if accessible)
    • After node recovers, wait for it to rejoin:
    • Read the verdict — the reason and when it flipped: On the host, confirm with `dmesg | grep -iE 'amdgpu|smu'` (SMU message timeouts, `TransferTableSmu2Dram`, "Failed to export SMU metrics table") and `cat /sys/class/drm/card*/device/current_link_speed`.
    • Do not fight the label. The agent re-applies it every 30 s; editing it away only lets a lane cold-start into the dead card. If you must serve from a card the sentinel distrusts, set `--gpu-health-sentinel=false` on the agent DaemonSet and remove the label by hand — a deliberate, visible override.
    and 10 more
    • Recover the host. Every wedge so far needed a hard power cycle; a warm `systemctl reboot` with processes in state D hung the host for hours. Lanes stay parked automatically meanwhile; nothing to scale down by hand.
    • Verify before trusting the card again. After the host is back the agent starts fresh, the first clean cycle labels the node `ok`, and the controller's Node watch re-reconciles the parked Models within seconds — so a lane with `minReplicas: 1` will cold-start on its own. If the card has a history, run a load soak first (fp16 matmul burn for 8–10 min sampling `rocm-smi`, then check `dmesg`); park t…
    • Close out with a validation-matrix row and the `flexinfer_gpu_health_degraded_total` counter as evidence of what the sentinel saw.
    • Watch: `flexinfer_gpu_power_sampler_active{node}` should stay `1`; `0` means it stopped itself, and that node's agent log says why: On the host, `journalctl -k -f | grep -iE 'amdgpu|smu'` (or [path]` if journald is flooded) should show nothing new: SMU message timeouts, `TransferTableSmu2Dram` failures or "Failed to export SMU metrics table" after enabling are the signal to turn it of…
    • Turn off: remove the node from `agent.gpuPower.nodes` (or empty the list) and let Flux roll the agents. A sampler that stopped itself stays stopped until its pod restarts, so a self-stop needs no action beyond deciding whether to keep the node listed.
    • cblevins-7900xtx: do not list it until the PCIe/SMU soak verdict (see the XTX rows in `.loom/60-validation-matrix.md`).
    • Start with minimal resources - Don't deploy multiple models simultaneously
    • Use `local` mode for MLC-LLM - Lower memory footprint than `server` mode
    • Monitor with `kubectl logs -f` - Watch for early crash signs
    • Have cleanup commands ready - Don't let crashes cascade

    docs/agents/operations.md#resource-cleanup-procedures

Read from the roadmap and spec · as of 2026-11-21

Altitude

Summit metrics as progress toward their targets: the top line is the target met, and the legend turns each metric’s line on or off. Every merge is a tick on the time axis, coloured by how it was joined to the route; inferred joins are dashed because they are a similarity match, not a key. When merges are too dense to tell apart they become one bar per day, split the same way, and a bar opens that day’s merges. Focus or hover a tick or bar for detail; the arrow keys move along the walk. Shaded bands are weather (main red, outages, breaker and storm holds) merged into episodes; review holds, the queue waiting on a person, run as the thin lane along the top. An inverted red marker stands on a merge that was later reverted (a false summit); flags are release tags. Lines and merges co-move here; nothing on this chart claims one caused the other.

Main red, Oct 22 16:16 – Oct 23 16:20 UTC, 24.1 h, flexinfer: main pipeline red.Main red, Oct 24 17:42–18:11 UTC, 0.5 h, flexinfer: main pipeline red.Main red, Oct 24 19:11–19:44 UTC, 0.5 h, flexinfer: main pipeline red.Main red, Oct 26 02:50–07:07 UTC, 4.3 h, flexinfer: main pipeline red.Main red, Oct 26 21:19 – Oct 27 05:42 UTC, 8.4 h, flexinfer: main pipeline red.Main red, Oct 28 06:18–15:49 UTC, 9.5 h, 2 bands, flexinfer: main pipeline red.Main red, Oct 30 15:35–15:57 UTC, 0.4 h, flexinfer: main pipeline red.Main red, Oct 30 23:46 – Oct 31 16:14 UTC, 16.5 h, flexinfer: main pipeline red.Main red, Oct 31 17:28–21:01 UTC, 3.6 h, flexinfer: main pipeline red.Main red, Nov 1 18:58–20:49 UTC, 1.8 h, flexinfer: main pipeline red.Main red, Nov 2 14:46–15:38 UTC, 0.9 h, flexinfer: main pipeline red.Main red, Nov 2 16:58–18:28 UTC, 1.5 h, flexinfer: main pipeline red.Main red, Nov 3 05:39–05:51 UTC, 0.2 h, flexinfer: main pipeline red.Main red, Nov 3 18:08–20:17 UTC, 2.2 h, flexinfer: main pipeline red.Main red, Nov 3 21:39 – Nov 4 00:40 UTC, 3 h, flexinfer: main pipeline red.Main red, Nov 5 01:25–03:09 UTC, 1.7 h, flexinfer: main pipeline red.Main red, Nov 5 05:12–05:32 UTC, 0.3 h, flexinfer: main pipeline red.Main red, Nov 5 07:06–07:43 UTC, 0.6 h, flexinfer: main pipeline red.Main red, Nov 5 17:20–17:38 UTC, 0.3 h, flexinfer: main pipeline red.Main red, Nov 5 18:20–18:34 UTC, 0.2 h, flexinfer: main pipeline red.Main red, Nov 6 23:19 – Nov 7 22:55 UTC, 23.6 h, flexinfer: main pipeline red.Main red, Nov 11 20:28 – Nov 16 17:57 UTC, 117.5 h, flexinfer: main pipeline red.Main red, Nov 17 20:00 – Nov 18 14:40 UTC, 18.7 h, flexinfer: main pipeline red.Main red, Nov 19 19:16–19:23 UTC, 0.1 h, flexinfer: main pipeline red.Main red, Nov 19 20:31 – Nov 20 14:33 UTC, 18 h, flexinfer: main pipeline red.Main red, Nov 20 22:24 – Nov 21 00:14 UTC, 1.8 h, flexinfer: main pipeline red.Main red, Nov 21 03:34–04:08 UTC, 0.6 h, flexinfer: main pipeline red.Main red (still holding), Nov 21 05:46–11:04 UTC, 5.3 h, flexinfer: main pipeline red.
targetroadmap textinferredunplannedowner mergesMain red

Weather, false summits and base camps: tab to the first mark or hover one for its detail; the arrow keys move between marks.

2026-11-21 · 6 merges: 6 unplanned (6 owner merges).

2026-11-21MR !1371join: unplannedoperator lane

feat(scripts): video-gate harness for the in-pod Wan T2V gate

Served no written waypoint (off route: owner merge). Cost not recorded.

Commit 9edf349314e6 on GitHub →

Weather

28 episodes from 29 recorded bands (same kind, gaps of 30 minutes or less merged), about 266.1 hours held in the window (overlaps counted once); one is still holding.

  • Main red265.9 h28 episodes (29 bands), holding

The 10 longest episodes

  • Main red Nov 11 20:28 – Nov 16 17:57 UTC · 117.5 h · flexinfer's condition main pipeline red
  • Main red Oct 22 16:16 – Oct 23 16:20 UTC · 24.1 h · flexinfer's condition main pipeline red
  • Main red Nov 6 23:19 – Nov 7 22:55 UTC · 23.6 h · flexinfer's condition main pipeline red
  • Main red Nov 17 20:00 – Nov 18 14:40 UTC · 18.7 h · flexinfer's condition main pipeline red
  • Main red Nov 19 20:31 – Nov 20 14:33 UTC · 18 h · flexinfer's condition main pipeline red
  • Main red Oct 30 23:46 – Oct 31 16:14 UTC · 16.5 h · flexinfer's condition main pipeline red
  • Main red Oct 28 06:18–15:49 UTC · 9.5 h · 2 bands · flexinfer's condition main pipeline red
  • Main red Oct 26 21:19 – Oct 27 05:42 UTC · 8.4 h · flexinfer's condition main pipeline red
  • Main red (still holding) Nov 21 05:46–11:04 UTC · 5.3 h · flexinfer's condition main pipeline red
  • Main red Oct 26 02:50–07:07 UTC · 4.3 h · flexinfer's condition main pipeline red
Show the other 18 episodes
  • Main red Oct 24 17:42–18:11 UTC · 0.5 h · flexinfer's condition main pipeline red
  • Main red Oct 24 19:11–19:44 UTC · 0.5 h · flexinfer's condition main pipeline red
  • Main red Oct 30 15:35–15:57 UTC · 0.4 h · flexinfer's condition main pipeline red
  • Main red Oct 31 17:28–21:01 UTC · 3.6 h · flexinfer's condition main pipeline red
  • Main red Nov 1 18:58–20:49 UTC · 1.8 h · flexinfer's condition main pipeline red
  • Main red Nov 2 14:46–15:38 UTC · 0.9 h · flexinfer's condition main pipeline red
  • Main red Nov 2 16:58–18:28 UTC · 1.5 h · flexinfer's condition main pipeline red
  • Main red Nov 3 05:39–05:51 UTC · 0.2 h · flexinfer's condition main pipeline red
  • Main red Nov 3 18:08–20:17 UTC · 2.2 h · flexinfer's condition main pipeline red
  • Main red Nov 3 21:39 – Nov 4 00:40 UTC · 3 h · flexinfer's condition main pipeline red
  • Main red Nov 5 01:25–03:09 UTC · 1.7 h · flexinfer's condition main pipeline red
  • Main red Nov 5 05:12–05:32 UTC · 0.3 h · flexinfer's condition main pipeline red
  • Main red Nov 5 07:06–07:43 UTC · 0.6 h · flexinfer's condition main pipeline red
  • Main red Nov 5 17:20–17:38 UTC · 0.3 h · flexinfer's condition main pipeline red
  • Main red Nov 5 18:20–18:34 UTC · 0.2 h · flexinfer's condition main pipeline red
  • Main red Nov 19 19:16–19:23 UTC · 0.1 h · flexinfer's condition main pipeline red
  • Main red Nov 20 22:24 – Nov 21 00:14 UTC · 1.8 h · flexinfer's condition main pipeline red
  • Main red Nov 21 03:34–04:08 UTC · 0.6 h · flexinfer's condition main pipeline red

Route as written and walked

The waypoints the roadmap lists, in written order, against the merges that served them. A waypoint walked before one written ahead of it is a slip; one with no merge in the window is dimmed.

6 written, 4 served, 225 off route

WrittenWalkedUnplannedwritten ordermerge time123456Oct 23Oct 29Nov 4Nov 10Nov 16

6 written, 4 served, 225 off route: 6 written waypoints, 4 with a step in the window, 34 steps that served one and 225 off the written route. 2 waypoints were walked out of written order and 2 have no step in the window. The written rail is in the order the roadmap lists its waypoints, not on the time axis; the walked rail is.

Route as written

Now 1 waypoint

  1. 6.

    Now

    walking · ROADMAP.md:114 @ 9edf3493

    • MR !12432026-11-04join: roadmap text

Next 4 waypoints

  1. 1.

    Backend currency and gfx1100 vLLM gaps

    Next · walking · walked out of written order · ROADMAP.md:135 @ 9edf3493

    • MR !11392026-10-25join: inferred 0.83
    • MR !12822026-11-04join: inferred 0.79
    • MR !12852026-11-05join: inferred 0.85
    • MR !12882026-11-05join: inferred 0.87
    • MR !12902026-11-05join: inferred 0.86
    • MR !13082026-11-05join: inferred 0.83
  2. 2.

    Runtime images built and promoted by CI, no pin drift

    Next · planned · no step in this window · ROADMAP.md:149 @ 9edf3493

  3. 3.

    Put the idle gfx1100 cards to work for loom-core and Mills

    Next · done · walked out of written order · ROADMAP.md:158 @ 9edf3493

    • MR !12032026-11-01join: roadmap text
  4. 4.

    Python 3.14 model-tools promoted through a ModelCache publisher canary

    Next · unverified · no step in this window · ROADMAP.md:168 @ 9edf3493

Issues, specs and other waypoints 1 waypoint

  1. 5.

    Current Status

    walking · ROADMAP.md:20 @ 9edf3493

    • MR !11042026-10-22join: roadmap text
    • MR !11102026-10-23join: roadmap text
    • MR !11322026-10-24join: roadmap text
    • MR !11452026-10-24join: roadmap text
    • MR !11552026-10-25join: roadmap text
    • MR !11612026-10-25join: roadmap text
    • MR !11662026-10-25join: roadmap text
    • MR !11742026-10-26join: roadmap text
    • MR !11772026-10-26join: roadmap text
    • MR !11782026-10-26join: roadmap text
    • MR !11822026-10-26join: roadmap text
    • MR !11652026-10-27join: roadmap text
    • MR !11982026-10-30join: roadmap text
    • MR !12002026-10-30join: roadmap text
    • MR !12062026-10-30join: roadmap text
    • MR !12082026-10-31join: roadmap text
    • MR !12522026-11-02join: roadmap text
    • MR !12072026-11-02join: roadmap text
    • MR !12642026-11-03join: roadmap text
    • MR !12682026-11-03join: roadmap text
    • MR !12702026-11-03join: roadmap text
    • MR !12752026-11-04join: roadmap text
    • MR !12772026-11-04join: roadmap text
    • MR !12782026-11-04join: roadmap text
    • MR !12792026-11-04join: roadmap text
    • MR !12342026-11-04join: roadmap text

Off the written route

225 steps: 225 owner merges; the latest first.

  • MR !13712026-11-21owner mergesfeat(scripts): video-gate harness for the in-pod Wan T2V gate
  • MR !13702026-11-21owner mergesdocs(matrix): pre-register the in-pod Wan T2V gate (D5)
  • MR !13692026-11-21owner mergesdocs(matrix): D3 in-pod Qwen-Image passes with the allocator pinned
  • MR !13682026-11-21owner mergesfeat(models): serve Qwen-Image 2.1 in-pod on the XTX runtime, third attempt with the allocator pinned (D3) [LIVE-IMPACTING, owner window]
  • MR !13672026-11-21owner mergesdocs(matrix): KT-JOBS re-run 4 passes 9 of 9 on agent a541c524
  • MR !13662026-11-21owner mergesops(agent): roll the k3s agent to master a541c524 [LIVE-IMPACTING]
  • MR !13652026-11-20owner mergesfix(agent): degrade on probe-timeout only after two consecutive cycles
  • MR !13642026-11-20owner mergesops(proxy): roll the k3s proxy to master d90cb4bc [LIVE-IMPACTING]
  • MR !13632026-11-20owner mergesfix(proxy): name the contender on a multi-model card the two cannot share
  • MR !13622026-11-20owner mergesops(proxy): roll the k3s proxy to master 5a796de0 [LIVE-IMPACTING]
  • MR !13612026-11-20owner mergestest(kt-jobs): exclude the warm lane's own return from bar 2
  • MR !13602026-11-20owner mergesfix(proxy): keep the preempted wait through a release that hands the card on
Show the next 30 steps
  • MR !13592026-11-20owner mergesops(proxy): roll the k3s proxy to master 535f56fb [LIVE-IMPACTING]
  • MR !13582026-11-20owner mergesdocs(matrix): record the owner decision to keep the XTX warm lane in-pod
  • MR !13562026-11-19owner mergesfix(proxy): bound waits on preempted lanes, free abandoned queue slots
  • MR !13572026-11-19owner mergesfeat(diffusers): fail non-finite renders instead of serving garbage; add render gate
  • MR !13552026-11-19owner mergesfix(controller,runtime): make status.kvPool describe the in-pod load
  • MR !13542026-11-19owner mergesdocs(validation): record the KT-SWAP fallback arm of 2026-11-19
  • MR !13532026-11-19owner mergesdocs(validation): record the unchanged D3 retry of 2026-11-19
  • MR !13522026-11-19owner mergesrevert: serve Qwen-Image 2.1 in-pod on the XTX runtime, second attempt (gate failed again)
  • MR !13512026-11-19owner mergesfeat(models): serve Qwen-Image 2.1 in-pod on the XTX runtime, second attempt (D3 retry) [LIVE-IMPACTING, owner window]
  • MR !13502026-11-19owner mergesdocs(validation): record the D3 in-pod Qwen-Image gate failure of 2026-11-19
  • MR !13492026-11-19owner mergesrevert: serve Qwen-Image 2.1 in-pod on the XTX runtime (D3 failed its multi-render gate)
  • MR !12362026-11-19owner mergesfeat(models): serve Qwen-Image 2.1 in-pod on the XTX runtime (D1 step 3, lane 2/5) [LIVE-IMPACTING, owner window; multi-render gate first]
  • MR !13482026-11-19owner mergesfeat(models): serve the XTX warm lane in-pod, retry with the allocator pinned (D2) [LIVE-IMPACTING, owner window]
  • MR !13472026-11-18owner mergesdocs(validation): record the KT-JOBS re-run of 2026-11-18
  • MR !13462026-11-18owner mergesops(control-plane): roll the k3s controller and proxy to master 6500e342 [LIVE-IMPACTING, owner window]
  • MR !13452026-11-18owner mergesfix(controller): prune the EndpointSlices a removed Service selector leaves behind
  • MR !13442026-11-18owner mergesrevert: serve the XTX warm lane in-pod (D2 failed its live checks)
  • MR !13432026-11-18owner mergesfix(jobs): give a shared GPU back when the job queue for it empties
  • MR !12352026-11-18owner mergesfeat(models): serve the XTX warm lane in-pod on the runtime DaemonSet (D1 step 3, lane 1/5) [LIVE-IMPACTING, owner window]
  • MR !13422026-11-18owner mergesdocs(validation): record the KT-JOBS run of 2026-11-17 as fail on bars 2, 5 and 7
  • MR !13412026-11-17owner mergesdocs: refer to the transcription client generically in voice-stack docs
  • MR !13402026-11-17owner mergesfix(prewarm): drop the Harbor-deleted runtime@2e9652ed from the 7900xtx profile
  • MR !13382026-11-16owner mergestest(proxy): choose the job fixture's status before publishing the body
  • MR !13392026-11-16owner mergeschore: keep .loom planning notes out of the repository
  • MR !13372026-11-16owner mergesfix(ci): clear GO-2026-6859 with Go 1.26.11, give llama.cpp ROCm builds 2h
  • MR !13362026-11-11owner mergesdocs(build): point manual image builds at cluster-buildkit
  • MR !13352026-11-11owner mergesbuild: build images on cluster BuildKit instead of 7900xtx
  • MR !13322026-11-07owner mergesfix(deps): bump golang.org/x/net to v0.62.0 for CVE-2026-78661
  • MR !13312026-11-07owner mergesfeat(agent): inventory Local model caches and flag orphans
  • MR !13292026-11-07owner mergesfeat(controller): on-delete policy for Local cache directories (retain by default, delete via node Job)

183 older off-route steps are in the climb’s JSON.

Legs

One chapter per day, narrated from the day’s merges, roadmap changes, metric readings and weather. Every sentence carries its evidence: a chip opens the merge, the roadmap line as written, the metric point or the weather band it rests on, and a sentence whose evidence could not be published is left out and counted. A day narrated again says “revised” and keeps its earlier versions; a day the model could not narrate within the rules is written from a fixed template and marked as one.

15 chapters, newest first. Arrow keys move between chapters; Tab moves through a chapter’s citations.

  1. Leg for descending

    Eight unplanned operator merges co-moved with mixed metric shifts on 2026-11-20.

    Moved
    Merges per week · Nov 20: Summit metric “Merges per week” on 2026-11-20: 23 merges. Jumps to the altitude chart.Proxy request success ratio (rolling 1d) · Nov 20: Summit metric “Proxy request success ratio (rolling 1d)” on 2026-11-20: 98.4%. Jumps to the altitude chart.
    Slipped
    Master pipeline success ratio (7d) · Nov 20: Summit metric “Master pipeline success ratio (7d)” on 2026-11-20: 83.3%. Jumps to the altitude chart.
    • Merges per week 12 merges → 23 merges +17 merges rising over 7 days, toward target
    • Master pipeline success ratio (7d) 90% → 83.3% +66.7 pts rising over 7 days, toward target
    • Proxy request success ratio (rolling 1d) 98.4% → 98.5% −1.3 pts falling over 7 days, away from target

    Co-moved: On 2026-11-20, 8 changes merged while Merges per week rose from 12 to 23 (toward its target), Master pipeline success ratio (7d) fell from 0.9 to 0.8333 (away from its target) and Proxy request success ratio (rolling 1d) rose from 0.984 to 0.9851 (toward its target). This is co-movement on the same day only; the record does not show that any change moved a metric.

    Narrated by qwen38-27b-autoround-workhorse.

  2. Leg for ascending

    11 unplanned merges co-moved with metrics rising toward targets on 2026-11-19.

    Moved
    Merges per week · Nov 19: Summit metric “Merges per week” on 2026-11-19: 12 merges. Jumps to the altitude chart.Master pipeline success ratio (7d) · Nov 19: Summit metric “Master pipeline success ratio (7d)” on 2026-11-19: 90%. Jumps to the altitude chart.Proxy request success ratio (rolling 1d) · Nov 19: Summit metric “Proxy request success ratio (rolling 1d)” on 2026-11-19: 99.5%. Jumps to the altitude chart.
    • Merges per week 7 merges → 12 merges −11 merges falling over 7 days, away from target
    • Master pipeline success ratio (7d) 57.1% → 90% +25 pts rising over 7 days, toward target
    • Proxy request success ratio (rolling 1d) 97.5% → 98.4% −1.4 pts falling over 7 days, away from target

    Co-moved: On 2026-11-19, 11 changes merged while Merges per week rose from 7 to 12 (toward its target), Master pipeline success ratio (7d) rose from 0.5714 to 0.9 (toward its target) and Proxy request success ratio (rolling 1d) rose from 0.9755 to 0.984 (toward its target). This is co-movement on the same day only; the record does not show that any change moved a metric.

    Narrated by qwen38-27b-autoround-workhorse.

  3. Leg for descending

    Seven unplanned merges co-moved with mixed metric shifts on 2026-11-18.

    Moved
    Merges per week · Nov 18: Summit metric “Merges per week” on 2026-11-18: 7 merges. Jumps to the altitude chart.Proxy request success ratio (rolling 1d) · Nov 18: Summit metric “Proxy request success ratio (rolling 1d)” on 2026-11-18: 97.3%. Jumps to the altitude chart.
    Slipped
    Master pipeline success ratio (7d) · Nov 18: Summit metric “Master pipeline success ratio (7d)” on 2026-11-18: 57.1%. Jumps to the altitude chart.
    • Merges per week 5 merges → 7 merges −38 merges falling over 7 days, away from target
    • Master pipeline success ratio (7d) 60% → 57.1% −8.6 pts falling over 7 days, away from target
    • Proxy request success ratio (rolling 1d) 97.1% → 97.6% −2.2 pts falling over 7 days, away from target

    Co-moved: On 2026-11-18, 7 changes merged while Merges per week rose from 5 to 7 (toward its target), Master pipeline success ratio (7d) fell from 0.6 to 0.5714 (away from its target) and Proxy request success ratio (rolling 1d) rose from 0.9714 to 0.9756 (toward its target). This is co-movement on the same day only; the record does not show that any change moved a metric.

    Narrated by qwen38-27b-autoround-workhorse.

  4. Leg for descending

    Two unplanned operator merges co-moved with mixed metric trends on 2026-11-17.

    Moved
    Merges per week · Nov 17: Summit metric “Merges per week” on 2026-11-17: 5 merges. Jumps to the altitude chart.Master pipeline success ratio (7d) · Nov 17: Summit metric “Master pipeline success ratio (7d)” on 2026-11-17: 60%. Jumps to the altitude chart.
    Slipped
    Proxy request success ratio (rolling 1d) · Nov 17: Summit metric “Proxy request success ratio (rolling 1d)” on 2026-11-17: 96.7%. Jumps to the altitude chart.
    • Merges per week 2 merges → 5 merges −50 merges falling over 7 days, away from target
    • Master pipeline success ratio (7d) 0% → 60% −12.5 pts falling over 7 days, away from target
    • Proxy request success ratio (rolling 1d) 99.6% → 97.1% −2.4 pts falling over 7 days, away from target

    Co-moved: On 2026-11-17, 2 changes merged while Merges per week rose from 2 to 5 (toward its target), Master pipeline success ratio (7d) rose from 0 to 0.6 (toward its target) and Proxy request success ratio (rolling 1d) fell from 0.996 to 0.9708 (away from its target). This is co-movement on the same day only; the record does not show that any change moved a metric.

    Narrated by qwen38-27b-autoround-workhorse.

  5. Leg for descending

    Three unplanned operator merges occurred while the main pipeline remained red.

    Slipped
    Proxy request success ratio (rolling 1d) · Nov 16: Summit metric “Proxy request success ratio (rolling 1d)” on 2026-11-16: 99.2%. Jumps to the altitude chart.
    • Merges per week 2 merges → 2 merges −64 merges falling over 7 days, away from target
    • Master pipeline success ratio (7d) 0% → 0% −70.2 pts falling over 7 days, away from target
    • Proxy request success ratio (rolling 1d) 99.7% → 99.6% +0.4 pts rising over 7 days, toward target

    Co-moved: On 2026-11-16, 3 changes merged while Proxy request success ratio (rolling 1d) fell from 0.997 to 0.996 (away from its target). This is co-movement on the same day only; the record does not show that any change moved a metric.

    Narrated by qwen38-27b-autoround-workhorse.

  6. Leg for weathered

    Main pipeline red held the climb; cadence and CI metrics fell while serving improved.

    Moved
    Proxy request success ratio (rolling 1d) · Nov 15: Summit metric “Proxy request success ratio (rolling 1d)” on 2026-11-15: 99.7%. Jumps to the altitude chart.
    Slipped
    Merges per week · Nov 15: Summit metric “Merges per week” on 2026-11-15: 2 merges. Jumps to the altitude chart.Master pipeline success ratio (7d) · Nov 15: Summit metric “Master pipeline success ratio (7d)” on 2026-11-15: 0%. Jumps to the altitude chart.
    • Merges per week 6 merges → 2 merges −77 merges falling over 7 days, away from target
    • Master pipeline success ratio (7d) 16.7% → 0% −67.3 pts falling over 7 days, away from target
    • Proxy request success ratio (rolling 1d) 99.1% → 99.7% +0.1 pts rising over 7 days, toward target

    Co-moved: On 2026-11-15, 0 changes merged while Merges per week fell from 6 to 2 (away from its target), Master pipeline success ratio (7d) fell from 0.1667 to 0 (away from its target) and Proxy request success ratio (rolling 1d) rose from 0.9909 to 0.997 (toward its target). This is co-movement on the same day only; the record does not show that any change moved a metric.

    Narrated by qwen38-27b-autoround-workhorse.

  7. Leg for descending

    services/flexinfer stalled on 2026-11-14 with zero merges and red main pipeline.

    Slipped
    Merges per week · Nov 14: Summit metric “Merges per week” on 2026-11-14: 6 merges. Jumps to the altitude chart.Master pipeline success ratio (7d) · Nov 14: Summit metric “Master pipeline success ratio (7d)” on 2026-11-14: 16.7%. Jumps to the altitude chart.Proxy request success ratio (rolling 1d) · Nov 14: Summit metric “Proxy request success ratio (rolling 1d)” on 2026-11-14: 99.4%. Jumps to the altitude chart.
    • Merges per week 23 merges → 6 merges −91 merges falling over 7 days, away from target
    • Master pipeline success ratio (7d) 65% → 16.7% −52.1 pts falling over 7 days, away from target
    • Proxy request success ratio (rolling 1d) 99.8% → 99.1% −0.4 pts falling over 7 days, away from target

    Co-moved: On 2026-11-14, 0 changes merged while Merges per week fell from 23 to 6 (away from its target), Master pipeline success ratio (7d) fell from 0.65 to 0.1667 (away from its target) and Proxy request success ratio (rolling 1d) fell from 0.9977 to 0.9909 (away from its target). This is co-movement on the same day only; the record does not show that any change moved a metric.

    Narrated by qwen38-27b-autoround-workhorse.

  8. Leg for descending

    services/flexinfer stalled on 2026-11-13 with zero merges and a red main pipeline.

    Slipped
    Merges per week · Nov 13: Summit metric “Merges per week” on 2026-11-13: 23 merges. Jumps to the altitude chart.Master pipeline success ratio (7d) · Nov 13: Summit metric “Master pipeline success ratio (7d)” on 2026-11-13: 65%. Jumps to the altitude chart.Proxy request success ratio (rolling 1d) · Nov 13: Summit metric “Proxy request success ratio (rolling 1d)” on 2026-11-13: 99.6%. Jumps to the altitude chart.
    • Merges per week 45 merges → 23 merges −85 merges falling over 7 days, away from target
    • Master pipeline success ratio (7d) 65.7% → 65% −3.1 pts falling over 7 days, away from target
    • Proxy request success ratio (rolling 1d) 99.8% → 99.8% +0.4 pts rising over 7 days, toward target

    Co-moved: On 2026-11-13, 0 changes merged while Merges per week fell from 45 to 23 (away from its target), Master pipeline success ratio (7d) fell from 0.6571 to 0.65 (away from its target) and Proxy request success ratio (rolling 1d) fell from 0.9982 to 0.9977 (away from its target). This is co-movement on the same day only; the record does not show that any change moved a metric.

    Narrated by qwen38-27b-autoround-workhorse.

  9. Leg for weathered

    Main pipeline red held the climb; cadence and CI metrics slipped away from targets.

    Moved
    Proxy request success ratio (rolling 1d) · Nov 12: Summit metric “Proxy request success ratio (rolling 1d)” on 2026-11-12: 99.9%. Jumps to the altitude chart.
    Slipped
    Merges per week · Nov 12: Summit metric “Merges per week” on 2026-11-12: 45 merges. Jumps to the altitude chart.Master pipeline success ratio (7d) · Nov 12: Summit metric “Master pipeline success ratio (7d)” on 2026-11-12: 65.7%. Jumps to the altitude chart.
    • Merges per week 55 merges → 45 merges −55 merges falling over 7 days, away from target
    • Master pipeline success ratio (7d) 72.5% → 65.7% +1.2 pts rising over 7 days, toward target
    • Proxy request success ratio (rolling 1d) 99.8% → 99.8% +0.4 pts rising over 7 days, toward target

    Co-moved: On 2026-11-12, 0 changes merged while Merges per week fell from 55 to 45 (away from its target), Master pipeline success ratio (7d) fell from 0.725 to 0.6571 (away from its target) and Proxy request success ratio (rolling 1d) rose from 0.9978 to 0.9982 (toward its target). This is co-movement on the same day only; the record does not show that any change moved a metric.

    Narrated by qwen38-27b-autoround-workhorse.

  10. Leg for descending

    Two unplanned operator merges co-moved with mixed metric shifts on 2026-11-11.

    Moved
    Master pipeline success ratio (7d) · Nov 11: Summit metric “Master pipeline success ratio (7d)” on 2026-11-11: 72.5%. Jumps to the altitude chart.Proxy request success ratio (rolling 1d) · Nov 11: Summit metric “Proxy request success ratio (rolling 1d)” on 2026-11-11: no sample that day in the public series. Jumps to the altitude chart.
    Slipped
    Merges per week · Nov 11: Summit metric “Merges per week” on 2026-11-11: 55 merges. Jumps to the altitude chart.
    • Merges per week 66 merges → 55 merges −26 merges falling over 7 days, away from target
    • Master pipeline success ratio (7d) 70.2% → 72.5% +6.5 pts rising over 7 days, toward target
    • Proxy request success ratio (rolling 1d) 99.5% → 99.8% +0 pts rising over 7 days, toward target

    Co-moved: On 2026-11-11, 2 changes merged while Merges per week fell from 66 to 55 (away from its target), Master pipeline success ratio (7d) rose from 0.7021 to 0.725 (toward its target) and Proxy request success ratio (rolling 1d) rose from 0.9952 to 0.9978 (toward its target). This is co-movement on the same day only; the record does not show that any change moved a metric.

    Narrated by qwen38-27b-autoround-workhorse.

  11. Leg for ascendingtemplate: written from the deterministic template, not by the model

    No changes merged on 2026-11-10

    Moved
    Master pipeline success ratio (7d) · Nov 10: Summit metric “Master pipeline success ratio (7d)” on 2026-11-10: 70.2%. Jumps to the altitude chart.Proxy request success ratio (rolling 1d) · Nov 10: Summit metric “Proxy request success ratio (rolling 1d)” on 2026-11-10: no sample that day in the public series. Jumps to the altitude chart.
    Slipped
    Merges per week · Nov 10: Summit metric “Merges per week” on 2026-11-10: 66 merges. Jumps to the altitude chart.
    • Merges per week 79 merges → 66 merges −6 merges falling over 7 days, away from target
    • Master pipeline success ratio (7d) 67.3% → 70.2% +8.5 pts rising over 7 days, toward target
    • Proxy request success ratio (rolling 1d) 99.3% → 99.5% +0 pts rising over 7 days, toward target

    Co-moved: On 2026-11-10, 0 changes merged while Merges per week fell from 79 to 66 (away from its target), Master pipeline success ratio (7d) rose from 0.6727 to 0.7021 (toward its target) and Proxy request success ratio (rolling 1d) rose from 0.9926 to 0.9952 (toward its target). This is co-movement on the same day only; the record does not show that any change moved a metric.

  12. Leg for descendingtemplate: written from the deterministic template, not by the model

    No changes merged on 2026-11-09

    Slipped
    Merges per week · Nov 9: Summit metric “Merges per week” on 2026-11-09: 79 merges. Jumps to the altitude chart.Master pipeline success ratio (7d) · Nov 9: Summit metric “Master pipeline success ratio (7d)” on 2026-11-09: 67.3%. Jumps to the altitude chart.Proxy request success ratio (rolling 1d) · Nov 9: Summit metric “Proxy request success ratio (rolling 1d)” on 2026-11-09: no sample that day in the public series. Jumps to the altitude chart.
    • Merges per week 97 merges → 79 merges +7 merges rising over 7 days, toward target
    • Master pipeline success ratio (7d) 68.8% → 67.3% +0.6 pts rising over 7 days, toward target
    • Proxy request success ratio (rolling 1d) 99.6% → 99.3% −0.7 pts falling over 7 days, away from target

    Co-moved: On 2026-11-09, 0 changes merged while Merges per week fell from 97 to 79 (away from its target), Master pipeline success ratio (7d) fell from 0.6875 to 0.6727 (away from its target) and Proxy request success ratio (rolling 1d) fell from 0.9959 to 0.9926 (away from its target). This is co-movement on the same day only; the record does not show that any change moved a metric.

  13. Leg for ascendingtemplate: written from the deterministic template, not by the model

    No changes merged on 2026-11-08

    Moved
    Master pipeline success ratio (7d) · Nov 8: Summit metric “Master pipeline success ratio (7d)” on 2026-11-08: 68.8%. Jumps to the altitude chart.Proxy request success ratio (rolling 1d) · Nov 8: Summit metric “Proxy request success ratio (rolling 1d)” on 2026-11-08: no sample that day in the public series. Jumps to the altitude chart.
    Slipped
    Merges per week · Nov 8: Summit metric “Merges per week” on 2026-11-08: 97 merges. Jumps to the altitude chart.
    • Merges per week 108 merges → 97 merges +26 merges rising over 7 days, toward target
    • Master pipeline success ratio (7d) 68.1% → 68.8% −3.6 pts falling over 7 days, away from target
    • Proxy request success ratio (rolling 1d) 99.5% → 99.6% −0.4 pts falling over 7 days, away from target

    Co-moved: On 2026-11-08, 0 changes merged while Merges per week fell from 108 to 97 (away from its target), Master pipeline success ratio (7d) rose from 0.6812 to 0.6875 (toward its target) and Proxy request success ratio (rolling 1d) rose from 0.9948 to 0.9959 (toward its target). This is co-movement on the same day only; the record does not show that any change moved a metric.

  14. Leg for descending

    Four unplanned operator merges co-moved with mixed metric shifts on 2026-11-07.

    Moved
    Merges per week · Nov 7: Summit metric “Merges per week” on 2026-11-07: 108 merges. Jumps to the altitude chart.Master pipeline success ratio (7d) · Nov 7: Summit metric “Master pipeline success ratio (7d)” on 2026-11-07: 68.1%. Jumps to the altitude chart.
    Slipped
    Proxy request success ratio (rolling 1d) · Nov 7: Summit metric “Proxy request success ratio (rolling 1d)” on 2026-11-07: no sample that day in the public series. Jumps to the altitude chart.
    • Merges per week 100 merges → 108 merges +38 merges rising over 7 days, toward target
    • Master pipeline success ratio (7d) 64.5% → 68.1% −9.4 pts falling over 7 days, away from target
    • Proxy request success ratio (rolling 1d) 99.5% → 99.5% −0.4 pts falling over 7 days, away from target

    Co-moved: On 2026-11-07, 4 changes merged while Merges per week rose from 100 to 108 (toward its target), Master pipeline success ratio (7d) rose from 0.6452 to 0.6812 (toward its target) and Proxy request success ratio (rolling 1d) fell from 0.9954 to 0.9948 (away from its target). This is co-movement on the same day only; the record does not show that any change moved a metric.

    Narrated by qwen38-27b-autoround-workhorse.

  15. Leg for ascendingtemplate: written from the deterministic template, not by the model

    17 changes merged on 2026-11-06

    Moved
    Merges per week · Nov 6: Summit metric “Merges per week” on 2026-11-06: 100 merges. Jumps to the altitude chart.Proxy request success ratio (rolling 1d) · Nov 6: Summit metric “Proxy request success ratio (rolling 1d)” on 2026-11-06: no sample that day in the public series. Jumps to the altitude chart.
    Slipped
    Master pipeline success ratio (7d) · Nov 6: Summit metric “Master pipeline success ratio (7d)” on 2026-11-06: 64.5%. Jumps to the altitude chart.
    • Merges per week 81 merges → 100 merges +27 merges rising over 7 days, toward target
    • Master pipeline success ratio (7d) 66% → 64.5% −17.1 pts falling over 7 days, away from target
    • Proxy request success ratio (rolling 1d) 99.5% → 99.5% +0.8 pts rising over 7 days, toward target

    Co-moved: On 2026-11-06, 17 changes merged while Merges per week rose from 81 to 100 (toward its target), Master pipeline success ratio (7d) fell from 0.66 to 0.6452 (away from its target) and Proxy request success ratio (rolling 1d) rose from 0.9946 to 0.9954 (toward its target). This is co-movement on the same day only; the record does not show that any change moved a metric.

Where this comes from

Live from the operatorgenerated 2026-11-21 11:08 UTC ()

Read from the Mills operator within the last ten minutes. Sanitized from the Mills chronicle read model: run, backlog and roadmap ids are hashed; titles, headlines and narrated sentences are scrubbed of paths, URLs, credentials, addresses and unpublished repository names; only climbs on published repositories are listed. Merge request numbers and short commit hashes are kept because they are public; so is the name of the model that narrated each leg, while its backend, prompts and traces are not. Merge request numbers are shown as text because the GitLab they live on needs a sign-in.