m/mGPUmanager
1
0
Fork 0
Code Issues 13 Pull Requests Actions Packages Projects Releases Wiki Activity

Broker must be the SOLE VRAM authority — close bypass gaps so out-of-band ollama holds get evicted (imagen starved by 24h keep-alive) #6

New Issue
Open
opened 2026-07-10 10:51:06 +00:00 by mAi · 1 comment
mAi commented 2026-07-10 10:51:06 +00:00
Copy Link

Incident (2026-07-10)

flexsiebels imagen (FLUX/ComfyUI) failed with "no GPU space." Diagnosis on mRock (16 GB card):

memory.used 13285 MiB / free 2628 MiB
  16059 ollama            7796 MiB   (gemma3:12b, expires_at = +24h keep-alive)
 501815 ollama            2012 MiB   (bge-m3, youpc embedding)
   2716 whisper-server    2050 MiB
   2717 mVoice             292 MiB
   2713 comfyui (idle)     210 MiB

The broker (mgpumanager.service) was active the whole time, yet the ~10 GB ollama hold was never evicted, so FLUX couldn't allocate. Manual ollama stop gemma3:12b + ollama stop bge-m3 freed 12.4 GB and imagen worked. m's call: fix this at the root ("die große Lösung, an der Wurzel"), not with a shorter keep-alive band-aid.

Root cause

The broker already has LRU-eviction over config/consumers.yaml coexistence groups and unload paths (Issue #2 lease machinery, evicting.go). It didn't fire because VRAM was held outside the broker's control/accounting:

  1. ollama was loaded via its native API with a 24h keep-alive, bypassing the broker's load/unload lifecycle. The broker's scheduler math assumes it owns residency; a directly-loaded model it didn't schedule is invisible to its eviction decisions.
  2. The imagen request very likely did not force a broker lease/evict — if flexsiebels imagen hits ComfyUI directly (or /v1/image without the evict-first lease from #2/#15), the eviction step never runs.

So the eviction engine is fine; the problem is bypass: consumers pin/allocate VRAM around the broker instead of through it.

The root fix — broker as the single VRAM chokepoint

  1. Audit + close every GPU-consumer bypass. Every path that touches mRock VRAM must go through the broker (/v1/* or /v1/lease). Known suspects to route through the broker (or make broker-aware):
    • flexsiebels imagen / ImaGen skill (#15) — must acquire an evict-first lease before rendering.
    • youpc batch embeddings (direct ollama :11434, OllamaEmbeddingURL=mRock) — the bge-m3 hold.
    • agentic-swap / model-hub ollama calls (mAi #297, claude-tmux-pool provider) — the likely source of the gemma3:12b 24h-keep-alive load.
  2. Broker owns ollama lifecycle. ollama models are loaded/unloaded by the broker with a short/zero native keep-alive (broker-managed residency), so nothing pins VRAM for 24h behind its back. Set OLLAMA_KEEP_ALIVE short on the mRock ollama unit as defence-in-depth.
  3. Reconcile accounting against reality. The broker should periodically read nvidia-smi (it already shells it for /v1/status) and detect resident VRAM it did NOT schedule (out-of-band holds). Under VRAM pressure it must be able to reclaim them via the consumer's declared unload path — so even a stray direct-load gets evicted instead of failing the lease. Today an unaccounted hold silently wins.
  4. Priority eviction honoured end-to-end. An imagen lease (higher priority / interactive) must evict an idle low-priority LLM hold per consumers.yaml priority. Verify the priority actually drives the eviction decision.

Definition of done

Reproduce: load gemma3:12b into ollama with a long keep-alive, then request flexsiebels imagen. Expected: the broker detects the pressure, evicts the idle ollama model via its unload path, and imagen renders — with zero manual ollama stop. Add a regression test (scheduler-level) for "out-of-band resident model gets reclaimed under pressure."

References

  • Existing eviction: internal/scheduler/evicting.go, config/consumers.yaml, lease API Issue #2, ImaGen lease consumer #15.
  • Bypass sources to audit: mAi #297 (agentic-swap ollama), youpc.org mRock-GPU batch-embedding routing (youpc.org#194), ImaGen skill.
  • Incident detail in otto memory (group otto): "mRock GPU contention — ollama keep-alive pins VRAM, starves imagen" (2026-07-10).
## Incident (2026-07-10) flexsiebels imagen (FLUX/ComfyUI) failed with "no GPU space." Diagnosis on mRock (16 GB card): ``` memory.used 13285 MiB / free 2628 MiB 16059 ollama 7796 MiB (gemma3:12b, expires_at = +24h keep-alive) 501815 ollama 2012 MiB (bge-m3, youpc embedding) 2716 whisper-server 2050 MiB 2717 mVoice 292 MiB 2713 comfyui (idle) 210 MiB ``` The broker (`mgpumanager.service`) was **active** the whole time, yet the ~10 GB ollama hold was never evicted, so FLUX couldn't allocate. Manual `ollama stop gemma3:12b` + `ollama stop bge-m3` freed 12.4 GB and imagen worked. m's call: fix this at the root ("die große Lösung, an der Wurzel"), not with a shorter keep-alive band-aid. ## Root cause The broker already has LRU-eviction over `config/consumers.yaml` coexistence groups and unload paths (Issue #2 lease machinery, evicting.go). It didn't fire because **VRAM was held outside the broker's control/accounting**: 1. **ollama was loaded via its native API with a 24h keep-alive**, bypassing the broker's load/unload lifecycle. The broker's scheduler math assumes it owns residency; a directly-loaded model it didn't schedule is invisible to its eviction decisions. 2. **The imagen request very likely did not force a broker lease/evict** — if flexsiebels imagen hits ComfyUI directly (or `/v1/image` without the evict-first lease from #2/#15), the eviction step never runs. So the eviction engine is fine; the problem is **bypass**: consumers pin/allocate VRAM around the broker instead of through it. ## The root fix — broker as the single VRAM chokepoint 1. **Audit + close every GPU-consumer bypass.** Every path that touches mRock VRAM must go through the broker (`/v1/*` or `/v1/lease`). Known suspects to route through the broker (or make broker-aware): - flexsiebels **imagen / ImaGen skill** (#15) — must acquire an evict-first lease before rendering. - **youpc batch embeddings** (direct ollama `:11434`, OllamaEmbeddingURL=mRock) — the bge-m3 hold. - **agentic-swap / model-hub** ollama calls (mAi #297, claude-tmux-pool provider) — the likely source of the gemma3:12b 24h-keep-alive load. 2. **Broker owns ollama lifecycle.** ollama models are loaded/unloaded by the broker with a short/zero native keep-alive (broker-managed residency), so nothing pins VRAM for 24h behind its back. Set `OLLAMA_KEEP_ALIVE` short on the mRock ollama unit as defence-in-depth. 3. **Reconcile accounting against reality.** The broker should periodically read `nvidia-smi` (it already shells it for `/v1/status`) and detect resident VRAM it did NOT schedule (out-of-band holds). Under VRAM pressure it must be able to reclaim them via the consumer's declared `unload` path — so even a stray direct-load gets evicted instead of failing the lease. Today an unaccounted hold silently wins. 4. **Priority eviction honoured end-to-end.** An imagen lease (higher priority / interactive) must evict an idle low-priority LLM hold per `consumers.yaml priority`. Verify the priority actually drives the eviction decision. ## Definition of done Reproduce: load `gemma3:12b` into ollama with a long keep-alive, then request flexsiebels imagen. Expected: the broker detects the pressure, evicts the idle ollama model via its unload path, and imagen renders — **with zero manual `ollama stop`**. Add a regression test (scheduler-level) for "out-of-band resident model gets reclaimed under pressure." ## References - Existing eviction: `internal/scheduler/evicting.go`, `config/consumers.yaml`, lease API Issue #2, ImaGen lease consumer #15. - Bypass sources to audit: mAi #297 (agentic-swap ollama), youpc.org mRock-GPU batch-embedding routing (youpc.org#194), ImaGen skill. - Incident detail in otto memory (group otto): "mRock GPU contention — ollama keep-alive pins VRAM, starves imagen" (2026-07-10).
mAi commented 2026-07-29 10:33:07 +00:00
Author
Copy Link

Partly closed — reclaim works, prevention does not

What now works, and is deployed on mRock:

Per-consumer residency comes from the GPU instead of from the consumers' own health responses, which never carried the field. /v1/status reports real numbers for all four. A consumer holding VRAM is an eviction candidate regardless of who started it — that is the reclaim half of this issue, and it is verified live: a lease evicted mvoice, whisper-server and ollama in sequence and was granted with 1165 MiB free.

Regression test is TestEvictsConsumerStartedOutOfBand (internal/scheduler/evicting_test.go): the broker believes a consumer absent, the GPU says it holds 2800 MiB, and the lease must still reclaim it. It fails without the change.

What is NOT closed

The broker still cannot stop an out-of-band load from happening. It can only take VRAM back once something asks it for VRAM. Nothing mediates a direct client.

I hit this for real while testing. I loaded gemma3:12b straight at :11434, bypassing the broker entirely — the same path mRiver's clients use. It took ~11.3 GB. whisper-server, which was starting at the time, could not fit its model and crash-looped with SIGABRT:

whisper-server.service: Main process exited, code=dumped, status=6/ABRT
whisper-server.service: Failed with result 'core-dump'

STT stayed down until I unloaded the ollama model by hand and restarted the unit. The broker never saw the allocation, so it had nothing to arbitrate.

So the accurate statement of where this issue stands:

The broker can now reclaim VRAM it did not allocate. It still cannot prevent a direct client from taking that VRAM in the first place, and a service starting at the wrong moment can still be OOM-killed by one.

This matters concretely: #6 should not be read as "the broker is the sole VRAM authority" yet. A direct ollama call can still cost you STT.

Does the prevention gap need its own issue?

Yes, in my view. It is a different problem with a different shape — reclaim is a scheduler question, prevention is an admission-control question, and closing it means either putting ollama behind the broker for real (its remote clients currently bypass :11434 deliberately) or accepting that some VRAM is outside the broker's control and making services resilient to losing it rather than crash-looping. Neither belongs in this issue.

Filing that is the head's call, not mine, and I have not filed it.

Related, fixed on the way

Evicting whisper-server used to freeze 0 MiB (#8). It now really stops the unit and frees ~2 GB. That surfaced a second defect: the broker refused STT with consumer_unreachable while the unit was stopped, so the request never reached the scheduler and nothing ever started it back — eviction was effectively permanent. Both are fixed on mai/tesla/ollama-residency (https://mgit.msbls.de/m/mGPUmanager/commit/92e55f1).

## Partly closed — reclaim works, prevention does not What now works, and is deployed on mRock: Per-consumer residency comes from the GPU instead of from the consumers' own health responses, which never carried the field. `/v1/status` reports real numbers for all four. A consumer holding VRAM is an eviction candidate **regardless of who started it** — that is the reclaim half of this issue, and it is verified live: a lease evicted mvoice, whisper-server and ollama in sequence and was granted with 1165 MiB free. Regression test is `TestEvictsConsumerStartedOutOfBand` (`internal/scheduler/evicting_test.go`): the broker believes a consumer absent, the GPU says it holds 2800 MiB, and the lease must still reclaim it. It fails without the change. ### What is NOT closed **The broker still cannot stop an out-of-band load from happening.** It can only take VRAM back once something asks it for VRAM. Nothing mediates a direct client. I hit this for real while testing. I loaded `gemma3:12b` straight at `:11434`, bypassing the broker entirely — the same path mRiver's clients use. It took ~11.3 GB. whisper-server, which was starting at the time, could not fit its model and **crash-looped with SIGABRT**: ``` whisper-server.service: Main process exited, code=dumped, status=6/ABRT whisper-server.service: Failed with result 'core-dump' ``` STT stayed down until I unloaded the ollama model by hand and restarted the unit. The broker never saw the allocation, so it had nothing to arbitrate. So the accurate statement of where this issue stands: > The broker can now reclaim VRAM it did not allocate. It still cannot prevent a direct client from taking that VRAM in the first place, and a service starting at the wrong moment can still be OOM-killed by one. This matters concretely: **#6 should not be read as "the broker is the sole VRAM authority" yet.** A direct `ollama` call can still cost you STT. ### Does the prevention gap need its own issue? Yes, in my view. It is a different problem with a different shape — reclaim is a scheduler question, prevention is an admission-control question, and closing it means either putting ollama behind the broker for real (its remote clients currently bypass `:11434` deliberately) or accepting that some VRAM is outside the broker's control and making services resilient to losing it rather than crash-looping. Neither belongs in this issue. Filing that is the head's call, not mine, and I have not filed it. ### Related, fixed on the way Evicting whisper-server used to freeze 0 MiB (#8). It now really stops the unit and frees ~2 GB. That surfaced a second defect: the broker refused STT with `consumer_unreachable` while the unit was stopped, so the request never reached the scheduler and nothing ever started it back — eviction was effectively permanent. Both are fixed on `mai/tesla/ollama-residency` (https://mgit.msbls.de/m/mGPUmanager/commit/92e55f1).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
m
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: m/mGPUmanager#6
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?