mAi
3b3d828e9e
Replaces the MVP Passthrough with scheduler.Locked: a capacity-1 channel serialises every consumer's GPU work end-to-end. main.go switches to it. Behavioural contract: - Jobs that arrive while another job holds the GPU block on the channel until the holder finishes. Context cancellation aborts the wait cleanly (no leaked tokens, queue depth decremented). - Stats track queue_depth, in_flight, total_jobs, last_wait_ms, last_run_ms, oldest_queued — surfaced through /v1/status. - One lock for ALL consumers (not per-consumer): the design (§4.3) is explicit that grobgranular > GPU-stream-granular on single-GPU single-user hardware. mvoice + ollama + comfyui never run truly concurrently any more, which is the whole point — that's what produced the CUDA-OOM under load. Tests: - 5 goroutines hammer the scheduler concurrently → max in-flight = 1. - Cancellation while parked on the lock returns ctx.Err() and frees the queue slot. - Stats reflect in-flight + queue-depth transitions correctly. - Race detector clean. Schritt 5 will compose this with VRAM-pressure eviction: before acquiring the lock, check if the target consumer's resident cost fits under the current GPU headroom; if not, unload the LRU non-coexistent consumer first. Refs: m/mGPUmanager#1 (Schritt 4).
mGPUmanager
GPU-Inference-Control-Plane für mRock — Scheduler vor TTS/STT/LLM/Image-Gen mit globalem GPU-Lock + LRU-Eviction + einheitlicher /v1-Fassade. Konsumenten: mVoice, whisper-server, Ollama, ComfyUI/FLUX, später Furbotto. Go.
Full design: docs/design.md — Bestandsaufnahme, 10-Alternativen-Survey, Eviction-Algorithmus, Migrationspfad.
Was es macht
Auf mrock:8770 sitzt ein Go-Daemon, der:
/v1/tts,/v1/stt,/v1/llm,/v1/imageals einheitliche Konsumenten-Fassade exponiert,- jede Anfrage durch einen globalen GPU-Scheduler schleust (seriell, Queue),
- bei VRAM-Druck LRU-Eviction über die deklarierten Coexistenz-Gruppen aus
config/consumers.yamlfährt, - in
/v1/statusLive-GPU-Belegung + Consumer-Health + Scheduler-Statistiken zeigt, - niemals stille Fallbacks zurückgibt — Fehler kommen als strukturiertes
{error,message,consumer,retryable}.
Konsumenten-Registry
config/consumers.yaml deklariert pro Consumer:
url,health.{method,path}für Liveness-Probingpaths.<kind>.{method,path}— wie der Broker zu seinem TTS/STT/LLM/Image-Endpoint kommtvram_resident_mib— für die Scheduler-Mathe (Schritt 5)unload.{method,path,body}und optionalload.{method,path}— wie der Broker den Consumer aus dem VRAM räumt / wieder hochfährtcan_coexist_with: [..]— wer parallel resident sein darfpriority(0=low, 4=urgent),max_concurrency
Build + Deploy
make build # ./bin/mgpumanager
make test # go test ./...
make run # lokal gegen ./config/consumers.yaml
make deploy HOST=mrock # rsync + systemd reload + restart
Auf mRock läuft der Daemon als System-Unit (/etc/systemd/system/mgpumanager.service).
Endpoints
| Verb | Pfad | Verhalten |
|---|---|---|
| POST | /v1/tts |
Proxy zu routing.tts-Consumer (default: mvoice /api/synthesize) |
| POST | /v1/stt |
Proxy zu routing.stt-Consumer (default: mvoice /api/transcribe) |
| POST | /v1/llm |
Proxy zu routing.llm-Consumer (default: ollama /api/generate) |
| POST | /v1/image |
Proxy zu routing.image-Consumer (default: comfyui /prompt) |
| GET | /audio/* |
Proxy zu audio_proxy-Consumer (wa.sh fetcht generiertes Audio so) |
| GET | /v1/status |
Live-Snapshot: GPU + Consumer-Health + Scheduler-Stats |
| GET | /healthz |
Broker-Liveness (200 OK) |
Fehler-Schema
Jeder Broker-eigene Fehler hat die Form:
{
"error": "consumer_unreachable",
"message": "upstream mvoice last probe failed: connection refused",
"consumer": "mvoice",
"retryable": true
}
Codes: consumer_unreachable, no_consumer, scheduler_error, bad_consumer_url, bad_request. Pass-through-4xx/5xx vom Consumer landet unverändert beim Client.
Phase 1 Status (Issue #1)
- ✅ Schritt 0 — ComfyUI persistent (
systemd: comfyui.service) - ✅ Schritt 1 —
mvoice /api/admin/{load,unload}(mai/knuth/admin-load-unload @ mVoice) - ✅ Schritt 2 — Routing-Façade +
/v1/status(passthrough scheduler) - ☐ Schritt 3 — wa.sh auf Broker umgestellt
- ☐ Schritt 4 — Queue + globaler GPU-Lock
- ☐ Schritt 5 — Coexistenz-Gruppen + LRU-Eviction