m/mGPUmanager
1
0
Fork 0
Code Issues 13 Pull Requests Actions Packages Projects Releases Wiki Activity

unload(): whisper-server systemd branch marks consumer unloaded without stopping the unit (latent VRAM-not-freed bug) #8

New Issue
Open
opened 2026-07-14 16:14:54 +00:00 by mAi · 1 comment
mAi commented 2026-07-14 16:14:54 +00:00
Copy Link

Flagged out of #5 (§9 of docs/designs/issue-5-ollama-eviction.md). Filing separately so it is triaged on its own merits rather than folded into the ollama eviction fix.

The defect

In internal/scheduler/evicting.go, unload() has a fallback branch for consumers that have a systemd_unit but no HTTP unload route (today: whisper-server):

if cons.SystemdUnit != "" {
    e.mu.Lock()
    e.loaded[name] = false   // <-- marks unloaded...
    e.mu.Unlock()
    return nil               // ...but never stops whisper-server.service
}

It sets loaded[name] = false (so the scheduler stops accounting for its VRAM and stops picking it as a victim) without actually stopping the unit. If whisper-server's process still pins its ~2050 MiB, the scheduler's bookkeeping now says that memory is free while physically it is not — the exact same mark-unloaded-without-stopping class of bug that #5 fixed for ollama (where the fix was HTTP soft-unload, not systemctl).

This is the original root cause called out in #5's problem statement (evicting.go unload() systemd branch only sets loaded=false).

Why it is separate from #5

  • #5's lever is HTTP keep_alive:0 soft-unload — zero privilege, no systemctl. That lever does not exist for whisper-server (no HTTP unload endpoint; it is a whisper.cpp server).
  • Fixing whisper properly means one of: (a) a genuine systemctl/polkit stop path (the game-mode mechanism, currently only on unmerged origin/feat/game-mode), (b) a whisper-side HTTP unload endpoint, or (c) confirming whisper self-releases VRAM quickly enough that marking it unloaded is acceptable — in which case the code should say so explicitly, not rely on an unstated assumption.

Has it bitten?

Not observed in practice — whisper may self-release, or the eviction smoke test never exercised the systemd branch under real VRAM pressure. It is a latent correctness bug, not an active outage.

Suggested next step

During #5 live verification on mRock I will confirm empirically whether evicting whisper-server actually frees VRAM (warm whisper, force an eviction, watch nvidia-smi). That result will be posted here to decide between fix options (a)/(b)/(c). Until then this is no technical debt to work around — it should be fixed at the root once the empirical behaviour is known.

Ref: internal/scheduler/evicting.go (systemd branch of unload()); design §9 in docs/designs/issue-5-ollama-eviction.md.

Flagged out of #5 (§9 of `docs/designs/issue-5-ollama-eviction.md`). Filing separately so it is triaged on its own merits rather than folded into the ollama eviction fix. ## The defect In `internal/scheduler/evicting.go`, `unload()` has a fallback branch for consumers that have a `systemd_unit` but no HTTP `unload` route (today: **whisper-server**): ```go if cons.SystemdUnit != "" { e.mu.Lock() e.loaded[name] = false // <-- marks unloaded... e.mu.Unlock() return nil // ...but never stops whisper-server.service } ``` It sets `loaded[name] = false` (so the scheduler stops accounting for its VRAM and stops picking it as a victim) **without actually stopping the unit**. If whisper-server's process still pins its ~2050 MiB, the scheduler's bookkeeping now says that memory is free while physically it is not — the exact same *mark-unloaded-without-stopping* class of bug that #5 fixed for ollama (where the fix was HTTP soft-unload, not systemctl). This is the original root cause called out in #5's problem statement (`evicting.go` `unload()` systemd branch only sets `loaded=false`). ## Why it is separate from #5 - #5's lever is HTTP `keep_alive:0` soft-unload — zero privilege, no systemctl. That lever does **not** exist for whisper-server (no HTTP unload endpoint; it is a whisper.cpp server). - Fixing whisper properly means one of: (a) a genuine systemctl/polkit stop path (the game-mode mechanism, currently only on unmerged `origin/feat/game-mode`), (b) a whisper-side HTTP unload endpoint, or (c) confirming whisper self-releases VRAM quickly enough that marking it unloaded is acceptable — in which case the code should say so explicitly, not rely on an unstated assumption. ## Has it bitten? Not observed in practice — whisper may self-release, or the eviction smoke test never exercised the systemd branch under real VRAM pressure. It is a latent correctness bug, not an active outage. ## Suggested next step During #5 live verification on mRock I will confirm empirically whether evicting whisper-server actually frees VRAM (warm whisper, force an eviction, watch `nvidia-smi`). That result will be posted here to decide between fix options (a)/(b)/(c). Until then this is **no technical debt to work around** — it should be fixed at the root once the empirical behaviour is known. Ref: `internal/scheduler/evicting.go` (systemd branch of `unload()`); design §9 in `docs/designs/issue-5-ollama-eviction.md`.
mAi commented 2026-07-14 16:56:08 +00:00
Author
Copy Link

Confirmed live on mRock (during #5 verification)

Evicting whisper-server frees no VRAM — the latent bug bites in practice. During the #5 live verification, a comfyui image lease drove eviction of all three victims. The scheduler journal:

evicted consumer  victim=mvoice          target=comfyui  free_mib_after=2399
evicted consumer  victim=whisper-server  target=comfyui  free_mib_after=2399   <-- froze nothing
evicted consumer  victim=ollama          target=comfyui  free_mib_after=2399

whisper-server was marked unloaded (loaded=false) but whisper-server.service was never stopped, so its ~2050 MiB stayed resident (free_mib_after unchanged across the whisper eviction, vs. ollama's soft-unload which did free its VRAM once it settled). This wasted an eviction cycle: the lease only succeeded because mvoice + ollama together freed enough, with whisper contributing 0.

So the impact is concrete: on a host where whisper's VRAM is actually needed to fit the target, an image lease would fail (insufficient_vram) despite the scheduler believing whisper was reclaimed. It did not become fatal here only because the other two victims sufficed.

Fix options remain as filed (a real systemctl/polkit stop path — now viable since game-mode's mechanism graduated to main in #3; a whisper-side HTTP unload endpoint; or an explicit self-release confirmation). Recommend the game-mode systemctl lever now that it exists on main.

## Confirmed live on mRock (during #5 verification) Evicting whisper-server **frees no VRAM** — the latent bug bites in practice. During the #5 live verification, a comfyui image lease drove eviction of all three victims. The scheduler journal: ``` evicted consumer victim=mvoice target=comfyui free_mib_after=2399 evicted consumer victim=whisper-server target=comfyui free_mib_after=2399 <-- froze nothing evicted consumer victim=ollama target=comfyui free_mib_after=2399 ``` whisper-server was marked unloaded (`loaded=false`) but `whisper-server.service` was never stopped, so its ~2050 MiB stayed resident (`free_mib_after` unchanged across the whisper eviction, vs. ollama's soft-unload which did free its VRAM once it settled). This wasted an eviction cycle: the lease only succeeded because mvoice + ollama together freed enough, with whisper contributing 0. So the impact is concrete: on a host where whisper's VRAM is actually needed to fit the target, an image lease would fail (`insufficient_vram`) despite the scheduler believing whisper was reclaimed. It did not become fatal here only because the other two victims sufficed. Fix options remain as filed (a real systemctl/polkit stop path — now viable since game-mode's mechanism graduated to main in #3; a whisper-side HTTP unload endpoint; or an explicit self-release confirmation). Recommend the game-mode `systemctl` lever now that it exists on main.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
m
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: m/mGPUmanager#8
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?