Broker reports per-consumer gpu_resident_mib=0 for externally-started consumers → eviction finds "no candidates" and never reclaims their VRAM #4

New Issue

mAi · 2026-06-07T13:12:26Z

mAi commented

2026-06-07 13:12:26 +00:00

Symptom

A /v1/lease (kind=image) acquire fails with insufficient_vram even when an idle consumer is holding evictable VRAM. Broker log:

level=WARN msg="no eviction candidates" target=comfyui need_mib=13000 free_mib=10590 strict=true
POST /v1/lease status=503 (insufficient_vram)

Meanwhile nvidia-smi shows whisper-server holding 1996 MiB — clearly evictable (comfyui can_coexist_with: []). But /v1/status reports it as not-resident:

whisper-server  loaded=true  gpu_resident_mib=0   <- actually holds 1996 MiB
mvoice          loaded=false gpu_resident_mib=0
comfyui         loaded=true  gpu_resident_mib=0

Every consumer reads gpu_resident_mib=0, so eviction-candidate selection (which picks consumers with resident VRAM) finds nothing and the broker can't free space — even though stopping whisper manually frees ~2 GB and lets the lease grant.

Likely cause

The broker attributes VRAM per-consumer from something other than live nvidia-smi per-PID usage (e.g. a load/unload bookkeeping counter that is only updated when the broker itself loads/unloads a consumer). Consumers started outside the broker (whisper via its own systemd unit, mvoice/ollama pre-resident) are tracked as resident=0, so the broker won't evict them. After a broker restart this affects everything. knuth's May deploy verified eviction when the broker itself had loaded mvoice — the regression shows when consumers are externally resident.

Fix direction

Drive eviction-candidate residency from live per-PID nvidia-smi (map each consumer's process/port to its actual GPU memory), not only the broker's own load/unload bookkeeping. A consumer holding real VRAM must be an eviction candidate regardless of who started it. whisper-server in particular has no HTTP unload — evict it via its systemd_unit (stop, not restart; restart reloads the model and frees nothing).

Also (smaller)

comfyui.vram_resident_mib was 13000, which is larger than the realistically-reclaimable free VRAM on a desktop (Brave/Wayland/etc. hold ~1.5 GB that is not evictable). Lowered to 11000 in config/consumers.yaml during debugging — FLUX schnell fp8 generated fine at that budget (ComfyUI offloads T5/CLIP to RAM). Validate/keep this value.
whisper eviction must use systemctl stop, not restart.

Impact

Until fixed, ImaGen restyle (which acquires a lease via #15-on-the-ImaGen-side) only succeeds when the GPU is already fairly clear; the broker will not auto-reclaim idle services that it didn't itself load. Manual systemctl --user stop whisper-server was the workaround to land the first successful restyle (image produced, lineage correct).

Refs

ImaGen #15 (the lease consumer), mGPUmanager #2 (the lease primitive)
internal/scheduler/evicting.go (ensureFits / candidate selection), internal/gpu (nvidia-smi poller)

## Symptom A `/v1/lease` (kind=image) acquire fails with `insufficient_vram` even when an idle consumer is holding evictable VRAM. Broker log: ``` level=WARN msg="no eviction candidates" target=comfyui need_mib=13000 free_mib=10590 strict=true POST /v1/lease status=503 (insufficient_vram) ``` Meanwhile `nvidia-smi` shows `whisper-server` holding **1996 MiB** — clearly evictable (comfyui `can_coexist_with: []`). But `/v1/status` reports it as not-resident: ``` whisper-server loaded=true gpu_resident_mib=0 <- actually holds 1996 MiB mvoice loaded=false gpu_resident_mib=0 comfyui loaded=true gpu_resident_mib=0 ``` Every consumer reads `gpu_resident_mib=0`, so eviction-candidate selection (which picks consumers with resident VRAM) finds nothing and the broker can't free space — even though stopping whisper manually frees ~2 GB and lets the lease grant. ## Likely cause The broker attributes VRAM per-consumer from something other than live `nvidia-smi` per-PID usage (e.g. a load/unload bookkeeping counter that is only updated when the broker itself loads/unloads a consumer). Consumers started **outside** the broker (whisper via its own systemd unit, mvoice/ollama pre-resident) are tracked as `resident=0`, so the broker won't evict them. After a broker restart this affects everything. knuth's May deploy verified eviction when the broker itself had loaded mvoice — the regression shows when consumers are externally resident. ## Fix direction Drive eviction-candidate residency from **live per-PID nvidia-smi** (map each consumer's process/port to its actual GPU memory), not only the broker's own load/unload bookkeeping. A consumer holding real VRAM must be an eviction candidate regardless of who started it. whisper-server in particular has no HTTP unload — evict it via its `systemd_unit` (stop, not restart; restart reloads the model and frees nothing). ## Also (smaller) - `comfyui.vram_resident_mib` was 13000, which is larger than the realistically-reclaimable free VRAM on a desktop (Brave/Wayland/etc. hold ~1.5 GB that is not evictable). Lowered to **11000** in config/consumers.yaml during debugging — FLUX schnell fp8 generated fine at that budget (ComfyUI offloads T5/CLIP to RAM). Validate/keep this value. - whisper eviction must use `systemctl stop`, not `restart`. ## Impact Until fixed, ImaGen restyle (which acquires a lease via #15-on-the-ImaGen-side) only succeeds when the GPU is already fairly clear; the broker will not auto-reclaim idle services that it didn't itself load. Manual `systemctl --user stop whisper-server` was the workaround to land the first successful restyle (image produced, lineage correct). ## Refs - ImaGen #15 (the lease consumer), mGPUmanager #2 (the lease primitive) - internal/scheduler/evicting.go (ensureFits / candidate selection), internal/gpu (nvidia-smi poller)

mAi referenced this issue

2026-07-29 10:06:38 +00:00

comfyui's VRAM need is not a constant: it varies cold-vs-warm (8410 vs 6170 MiB) and with the requested resolution #12

m referenced this issue from a commit

2026-07-29 10:13:21 +00:00

mAi: #4 #6 - measure per-consumer VRAM from the GPU instead of asking the consumer

m referenced this issue from a commit

2026-07-29 10:17:34 +00:00

Merge mai/tesla/vram-truth: measure per-consumer VRAM from the GPU, really stop whisper-server on eviction, size comfyui to the measured cold peak — #4 #6 #8 #11

mAi commented

2026-07-29 10:19:59 +00:00

Deployed to mRock from main 531cc3b and verified live.

/v1/status now reports measured per-consumer residency, matching nvidia-smi exactly:

comfyui        6362   (nvidia-smi pid 2800:    6362)
mvoice          242   (nvidia-smi pid 905152:   242)
whisper-server 2052   (nvidia-smi pid 3321275: 2052)
ollama            0   (nvidia-smi pid 3340258: 2012)  <- still 0

Three of the four are fixed. ollama still reads 0, and that is a real gap in what I shipped, not a transient.

Cause: ollama's socket belongs to another user, so port->PID attribution cannot see it. Its residency comes from /api/ps instead, and I put that probe in the scheduler's measureResident, which only runs under eviction pressure — not on the registry's probe cadence that feeds /v1/status. So ollama's figure is correct at the moment it matters for eviction, and stale (0) on the status endpoint the rest of the time.

Fix would be to move the /api/ps residency read into registry.refreshResidency alongside the per-PID path, so both sources refresh on the same cadence. Not done here — flagged for a follow-up rather than left as a clean-looking story that does not add up.

Eviction correctness is unaffected: #6's guarantee holds because the measurement happens at the decision point.

Healthz, /v1/status and /v1/gamemode all verified after restart. Previous binary kept as bin/mgpumanager.bak-pre-vramtruth.

Deployed to mRock from main `531cc3b` and verified live. `/v1/status` now reports measured per-consumer residency, matching `nvidia-smi` exactly: ``` comfyui 6362 (nvidia-smi pid 2800: 6362) mvoice 242 (nvidia-smi pid 905152: 242) whisper-server 2052 (nvidia-smi pid 3321275: 2052) ollama 0 (nvidia-smi pid 3340258: 2012) <- still 0 ``` Three of the four are fixed. **ollama still reads 0, and that is a real gap in what I shipped, not a transient.** Cause: ollama's socket belongs to another user, so port->PID attribution cannot see it. Its residency comes from `/api/ps` instead, and I put that probe in the scheduler's `measureResident`, which only runs under eviction pressure — not on the registry's probe cadence that feeds `/v1/status`. So ollama's figure is correct at the moment it matters for eviction, and stale (0) on the status endpoint the rest of the time. Fix would be to move the `/api/ps` residency read into `registry.refreshResidency` alongside the per-PID path, so both sources refresh on the same cadence. Not done here — flagged for a follow-up rather than left as a clean-looking story that does not add up. Eviction correctness is unaffected: `#6`'s guarantee holds because the measurement happens at the decision point. Healthz, `/v1/status` and `/v1/gamemode` all verified after restart. Previous binary kept as `bin/mgpumanager.bak-pre-vramtruth`.

m referenced this issue from a commit

2026-07-29 10:24:59 +00:00

mAi: #4 - report ollama's residency on the probe cadence, not only during eviction

m referenced this issue from a commit

2026-07-29 10:34:04 +00:00

Merge mai/tesla/ollama-residency: report ollama's residency on the probe cadence, and do not refuse a stopped systemd consumer — #4 #8

mAi referenced this issue

2026-07-29 10:34:34 +00:00

The broker can reclaim VRAM it did not allocate, but nothing stops a direct client taking it — a bare `ollama run` still OOM-kills whisper #13

Sign in to join this conversation.