Multi-model backend expansion + compare harness #10
Reference in New Issue
Block a user
No description provided.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Goal
m: "small platform where I can easily use new models without changing my frontend look and feel"
Test the plug-and-play promise of the imagen platform by adding 3–4 new local models behind the existing CLI / HTTP / skill API, plus a
imagen compareharness that runs the same prompt across all enabled backends side-by-side. flexsiebels/imagineUI must stay untouched — that's the deliverable, not a side-effect.Context
State after #1–#9:
comfyuiadapter is hardcoded to a FLUX.1-schnell workflow (internal/backend/comfyui.go)replicateadapter merged but unused (no token)mockfor testing/imagineUI is a pure consumer ofimagenviaimagen.jobsqueue — backend swap should be invisibleThe 2026 landscape moved significantly since the FLUX.1 baseline. May 2026 web research surfaced new models that fit our RTX 4070 Ti SUPER (16GB).
Candidate models
Core (do these):
Stretch (only if Core lands cleanly):
Scope
Part A — Backend architecture (decide before adding models)
internal/backend/comfyui.gocurrently embeds a fixed FLUX-schnell workflow. To plug new model families cleanly, choose ONE:Path 1 (lean general): generic
comfyui-workflowadapterworkflow_template: path/to/workflow.jsonfield per yaml instance${prompt},${negative},${seed},${steps},${width},${height},${model}slotsinternal/backend/workflows/comfyui(hardcoded FLUX-schnell) for back-compat or migrate it to a workflow filePath 2 (explicit): one adapter per model family
comfyui-flux.go,comfyui-sd3.go,comfyui-hidream.goWorker decides after wiring the first new model. Document the choice in a
docs/backends.mdrationale section.Part B — Add the models
For each candidate (Core first):
~/ComfyUI/models/...)imagen generate "test" --backend <new-instance>produces an imagedocs/backends.mdPart C — Compare harness
image.Imagecomposite, no ImageMagick dependency)Part D — Optional: tie into series
imagen generate --series --models a,b,cwrites the comparison as a realimagen.seriesrow so it surfaces in flexsiebels/imagine/series/<id>. Decide during implementation — useful or scope-creep?Acceptance
docs/backends.mdimagen compareproduces side-by-side output + contact sheet/imagineUI unchanged and still works against the new backends (smoke test the live site)docs/backends.mdcovers setup per model + chosen architecture rationalesystemctl --user restart imagen-workerafter merge (daemon-rebuild trap from #9)Out of scope
References
imagen/handover-2026-05-11Shift-1 progress checkpoint
Path 1 (workflow-template) architecture shipped:
internal/backend/workflow_template.go— loads + substitutes JSON templates with type-preserving placeholders (${seed}→ int64,${prompt}→ string,${cfg}→ float64). Whole-value match only; partial matches are preserved literally so prompts containing the placeholder syntax don't corrupt.internal/backend/comfyui.go— refactored to load workflow from embedded FS or filesystem path. Back-compat preserved:workflow:defaults toflux1-schnellso existing configs keep working.flux1-schnell.json(migrated from hardcoded),flux2-klein.json(FLUX.2 Klein 4B w/ EmptyFlux2LatentImage + CLIPLoader type=flux2 + FluxGuidance),sd35-medium.json(CheckpointLoaderSimple bundled-clips variant + ModelSamplingSD3 + KSampler dpmpp_2m/sgm_uniform/cfg 4.5).internal/config/config.go) gainsflux2-klein-localandsd35-medium-localinstances. Per-template knobs (vae,clip,clip_l,clip_t5,dtype,shift,guidance) live in yaml — adding a new model is yaml + JSON, no Go code.Tests: 11 existing comfyui tests + 6 new workflow_template tests all green (
go test ./...).Next:
imagen comparesubcommand — sequential N-backend run + contact-sheet PNG + sidecar JSON.docs/backends.md— architecture rationale + per-model HF download paths + VRAM/latency./imagineUI against the new backends (the deliverable: UI must stay unchanged).Shift-1 complete
Branch:
mai/hermes/issue-10-multi-modelCommit:
8435817ce1949c9334b3e47c1dcf3889b5dcf069Architecture decision — Path 1 (workflow-template adapter)
One
comfyuiadapter, workflows as data. Adding a new model is yaml + JSON — never Go. Rationale + comparison documented indocs/backends.md.What landed
internal/backend/workflow_template.go—embed.FS-backed loader + token-substitution with type-preserving whole-value placeholders.${prompt}→ string,${seed}→ int64,${cfg}→ float64. Partial matches like"prefix ${prompt} suffix"are deliberately left literal so a prompt containing the placeholder syntax can't corrupt the workflow.internal/backend/comfyui.go— refactored. The hardcoded FLUX-schnell graph migrated toworkflows/flux1-schnell.jsonwith identical node IDs (6, 8, 9, 10, 11, 12, 13, 27, 30, 31). Back-compat preserved:workflow:defaults toflux1-schnellso existing configs keep working unchanged.flux2-klein.json(FLUX.2 Klein 4B w/EmptyFlux2LatentImage+CLIPLoader type=flux2+FluxGuidance),sd35-medium.json(CheckpointLoaderSimple bundled-clips variant + ModelSamplingSD3 + KSampler dpmpp_2m/sgm_uniform/cfg 4.5).cmd/imagen/compare.go— new subcommand. Sequential N-backend run (one GPU on mRock; parallel would OOM), per-backend PNG, sidecar JSON with per-model metadata + errors, composite contact sheet via the Goimagepackage (no ImageMagick dep).flux2-klein-local+sd35-medium-localinstances added toimagen config initoutput.docs/backends.md— Path 1 vs Path 2 rationale + per-model HF download paths + how-to-add-a-new-workflow + compare-harness reference.Tests
11 existing comfyui + 6 new workflow_template + 5 new compare tests. All green.
Live smoke proof
Both backends produced PNGs. Sidecar JSON has the new
workflow: "flux1-schnell"metadata key (additive — old consumers ignore unknown keys). Contact sheet rendered 1584×912 with backend label + latency under each cell. FLUX.1 latency 7.5s @ 768×768 fp8, VRAM peak 13030 MiB.The worker pipeline (
workerPipeline.Runincmd/imagen/worker.go) uses the samebuildBackend+Request→Generateflow as the CLI just exercised. Metadata is purely additive. flexsiebels/imagineUI API surface is unchanged.What's parked (deliberately, with operator notes in
docs/backends.md)flux-2-klein-base-4b-fp8.safetensors+qwen_3_4b.safetensors+flux2-vae.safetensorsunder~/dev/comfyui/models/; SD3.5:sd3.5_medium_incl_clips_t5xxlfp8scaled.safetensorsinmodels/checkpoints/). Once downloaded,imagen generate --backend flux2-klein-localand--backend sd35-medium-localshould produce images without any further code changes — that's the deliverable.Deploy notes
~/.config/imagen.yamldoesn't yet have the new yaml blocks (the sample is whatimagen config initwould print). To pick upflux2-klein-local+sd35-medium-local, either re-runimagen config init > ~/.config/imagen.yaml(overwrites) or copy the new blocks manually.systemctl --user restart imagen-worker.serviceto pick up the new binary (daemon-rebuild trap from #9). If the worker isn't yet installed there, the rebuild lands implicitly with the next deploy.Setting
needs-review.Shipped — merged into main as
7caf975(work commit8435817).Architecture: Path 1 (workflow-template adapter). One
comfyuiadapter, workflows as data. Adding a new model is now yaml + JSON, never Go.Bundled workflows: flux1-schnell (migrated from hardcoded), flux2-klein, sd35-medium.
Verified:
go build ./...clean,go test ./...6 packages green (including 11 existing comfyui + 6 new workflow_template + 5 new compare tests)imagen compareran mock + flux1-schnell-local at 768×768, both PNGs written, sidecar JSON correct, contact sheet renders/imagineUI API surface unchanged (the deliverable)imagen-worker.servicerestarted on the new binary at 17:32:41Smoke-pending: flux2-klein and sd35-medium model files need to be downloaded onto mRock (paths documented in
docs/backends.md). Workflows are JSON, no further Go changes needed.Out of scope: HiDream-O1-Image-Dev, Z-Image-Turbo, Qwen Image — left as easy follow-ups (just drop a new workflow JSON + yaml block).
Label:
done.