design(t-paliad-151): Paliadin Tailscale SSH route to mRiver
Inventor design for routing Paliadin from paliad.de's Dokploy container on mLake to mRiver via Tailscale + SSH, preserving m's Claude Code subscription instead of paying Anthropic API tokens. Three sub-designs covering m's four locked decisions (2026-05-07 22:35): - network_mode: host on paliad (m overrode the sidecar recommendation; Phase A explicitly tests traefik compatibility under host mode) - server-side paliadin-shim with one RPC per turn (run-turn / reset / health / bootstrap), authorized_keys command= restriction, from=mlake - env-var routing trigger (PALIADIN_REMOTE_HOST) + Paliadin interface split: LocalPaliadinService keeps the laptop PoC, RemotePaliadinService shells out to ssh m@mriver paliadin-shim - ed25519 keypair via Dokploy secret PALIADIN_SSH_PRIVATE_KEY, written to a chmod 600 tmpfile at startup; pinned host key via PALIADIN_KNOWN_HOSTS Verified live before designing: mRiver tmux+claude present, mLake Tailscale active and sees mRiver, paliad Dockerfile is alpine-minimal, no authorized_keys on mRiver yet. No assumptions left from CLAUDE.md. Includes: friendly error code mriver_unreachable extending t-paliad-150, single-flight rate limit, security review (defence-in-depth via command=/from= restrictions), three-phase rollout (manual proof → Dockerfile bake → polish), file-level deliverables for the coder shift. Inventor stops here — no code shipped. Awaiting m's go/no-go. Refs m/paliad#12
This commit is contained in:
m
592
docs/design-paliadin-tailscale-ssh-2026-05-07.md
Normal file
592
docs/design-paliadin-tailscale-ssh-2026-05-07.md
Normal file
@@ -0,0 +1,592 @@
|
|||||||
|
# Paliadin: route prod via Tailscale SSH to mRiver
|
||||||
|
|
||||||
|
**Issue:** m/paliad#12 — t-paliad-151
|
||||||
|
**Date:** 2026-05-07
|
||||||
|
**Author:** noether (inventor)
|
||||||
|
**Supersedes nothing.** Extends `docs/design-paliadin-2026-05-07.md` (the Phase 0 PoC) with a third deployment path between "laptop-only PoC" and "Anthropic API direct".
|
||||||
|
**Related:** t-paliad-146 (PoC ship), t-paliad-150 (`friendlyErrorMessage` pattern).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Goal
|
||||||
|
|
||||||
|
Make Paliadin reachable from `paliad.de` (Dokploy on mLake) without losing m's Claude Code subscription, by routing each turn over Tailscale + SSH from the paliad container to mRiver, where the existing long-lived `tmux` + `claude` PoC keeps running.
|
||||||
|
|
||||||
|
**Non-goals (v1):**
|
||||||
|
|
||||||
|
- Multi-host failover.
|
||||||
|
- Encryption beyond SSH-over-tailnet (already E2E-encrypted by Tailscale's WireGuard layer).
|
||||||
|
- Anthropic API fallback when mRiver is offline — show a friendly error instead.
|
||||||
|
- Wake-on-LAN of mRiver.
|
||||||
|
- Multi-tenant or multi-firm variants.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Live state — what was verified before designing
|
||||||
|
|
||||||
|
A design built on stale facts rots fast. These were probed on 2026-05-07, not assumed from CLAUDE.md or memory:
|
||||||
|
|
||||||
|
| Fact | How verified | Result |
|
||||||
|
|---|---|---|
|
||||||
|
| mRiver = `100.99.98.203`, has tmux + claude | this worker runs on mRiver; `tmux -V` → `tmux 3.6a`; `which claude` → `/home/m/.local/bin/claude` | confirmed |
|
||||||
|
| mLake (`100.99.98.201`) has Tailscale running | `ssh m@mlake tailscale status` | confirmed; mRiver visible as `active; direct [2a02:4780:41:3fbc::1]:41641` |
|
||||||
|
| paliad container Dockerfile is alpine:3.21 minimal, no SSH, no tailscaled | `Dockerfile` | confirmed (only `ca-certificates`) |
|
||||||
|
| paliad compose runs default Docker bridge (no `network_mode`) | `docker-compose.yml` | confirmed |
|
||||||
|
| mRiver has no `~/.ssh/authorized_keys` yet | `ls ~/.ssh/` | confirmed — file must be created in Phase A |
|
||||||
|
| `/tmp/paliadin/` does not exist on mRiver yet | `ls /tmp/paliadin` | confirmed — created on first turn (paliadin.go:185 `os.MkdirAll`) |
|
||||||
|
| `paliad-paliadin` tmux session is not currently running on mRiver | `tmux ls` | not present; the existing PoC creates it on demand |
|
||||||
|
|
||||||
|
**Implication for design:** the paliad container needs new infrastructure on three axes — network reachability of the tailnet, an SSH client + identity, and a service-layer code path that talks to a remote tmux instead of a local one. Each axis is its own sub-design below.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Locked decisions (m, 2026-05-07 22:35)
|
||||||
|
|
||||||
|
m made four design-shaping calls via the inventor's `AskUserQuestion` pass. They are recorded here verbatim because every downstream choice in §4–§6 follows from them.
|
||||||
|
|
||||||
|
| # | Question | m's choice |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | Container Tailscale shape | **`network_mode: host` on paliad** |
|
||||||
|
| 2 | SSH-to-mRiver protocol granularity | **Server-side `paliadin-shim` (one RPC per turn)** |
|
||||||
|
| 3 | Routing trigger | **Env var `PALIADIN_REMOTE_HOST` + interface split** |
|
||||||
|
| 4 | SSH private key storage | **Dokploy secret env var `PALIADIN_SSH_PRIVATE_KEY`** |
|
||||||
|
|
||||||
|
Decision (1) was *not* the inventor's recommendation — host mode has known interaction risk with traefik (§4.2). m is overriding the recommendation; this design accepts the call and codifies a Phase A test step that gates the rollout on traefik still working under host mode. If Phase A blows up, the fallback is to revisit (1) in a follow-up issue, not to silently swap to a sidecar.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Sub-design A — Container Tailscale shape
|
||||||
|
|
||||||
|
### 4.1 Shape: `network_mode: host`
|
||||||
|
|
||||||
|
paliad's container shares mLake's network namespace. `tailscale0` (mLake's tailnet interface) is directly visible from inside the container. Outbound `ssh m@100.99.98.203` reaches mRiver over the tailnet without any sidecar, userspace tailscaled, SOCKS proxy, or auth-key flow inside the container.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# docker-compose.yml diff
|
||||||
|
services:
|
||||||
|
web:
|
||||||
|
build: .
|
||||||
|
network_mode: host # NEW
|
||||||
|
# remove: expose: ["8080"] # host mode means port is on the host directly
|
||||||
|
environment:
|
||||||
|
- PORT=8080
|
||||||
|
...
|
||||||
|
# NEW Paliadin remote-routing knobs
|
||||||
|
- PALIADIN_REMOTE_HOST=${PALIADIN_REMOTE_HOST} # 100.99.98.203
|
||||||
|
- PALIADIN_REMOTE_USER=${PALIADIN_REMOTE_USER} # m
|
||||||
|
- PALIADIN_SSH_PRIVATE_KEY=${PALIADIN_SSH_PRIVATE_KEY}
|
||||||
|
- PALIADIN_KNOWN_HOSTS=${PALIADIN_KNOWN_HOSTS} # one-line ssh-keyscan output
|
||||||
|
restart: unless-stopped
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.2 Trade-off accepted: traefik routing under host mode
|
||||||
|
|
||||||
|
paliad.de's TLS is provided by Dokploy's traefik on the `dokploy-network` overlay. With `network_mode: host`, paliad is no longer attached to that overlay. Two failure modes are possible:
|
||||||
|
|
||||||
|
- **(M1)** traefik can't discover the service via Docker DNS → 502 at the edge.
|
||||||
|
- **(M2)** traefik routes via host loopback (`http://127.0.0.1:8080` or `host.docker.internal`) and works fine.
|
||||||
|
|
||||||
|
Recent Dokploy versions configure traefik with both `loadbalancer.server.url` and Docker labels; (M2) is the documented host-mode path. **Phase A explicitly tests this** (§7) before any code is written; if (M1) materialises, the design rolls back to the sidecar variant of decision 1 in a follow-up issue.
|
||||||
|
|
||||||
|
Other host-mode side-effects to flag in operations:
|
||||||
|
|
||||||
|
- paliad listens on host port 8080 directly. Any other compose service binding 8080 conflicts.
|
||||||
|
- paliad's outbound DNS uses host resolver (no Docker-internal `web` etc.). Currently fine: paliad's only network deps are external (Supabase, SMTP, GitHub raw). No service on `dokploy-network` is referenced by name.
|
||||||
|
- The container can reach **every** Tailscale node, not just mRiver. Mitigations live in §5 (key restriction) and §5.2 (`from=` clause on mRiver authorized_keys).
|
||||||
|
|
||||||
|
### 4.3 Dockerfile diff
|
||||||
|
|
||||||
|
```dockerfile
|
||||||
|
# Final stage adds the SSH client only. Tailscale is provided by the host.
|
||||||
|
FROM alpine:3.21
|
||||||
|
RUN apk add --no-cache ca-certificates openssh-client # +openssh-client (~1MB)
|
||||||
|
WORKDIR /app
|
||||||
|
COPY --from=backend /paliad /app/paliad
|
||||||
|
COPY --from=frontend /app/frontend/dist /app/dist
|
||||||
|
EXPOSE 8080
|
||||||
|
CMD ["/app/paliad"]
|
||||||
|
```
|
||||||
|
|
||||||
|
Image-size delta: alpine `openssh-client` is ~1.1 MB compressed — negligible. No tailscaled, no entrypoint script, no extra processes inside the container.
|
||||||
|
|
||||||
|
### 4.4 What does NOT change
|
||||||
|
|
||||||
|
- No Tailscale auth-key inside paliad. The container inherits the host's tailnet binding, so there is no per-container Tailscale identity to rotate. mLake's existing Tailscale auth is the only one in scope.
|
||||||
|
- No tailscaled process inside the container.
|
||||||
|
- No new sidecar container.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Sub-design B — SSH identity, restricted shim, host-key pinning
|
||||||
|
|
||||||
|
### 5.1 Identity: dedicated ed25519 keypair `paliad-prod`
|
||||||
|
|
||||||
|
One keypair, generated once on mRiver during Phase A, used by every paliad-prod deploy:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On mRiver (Phase A bootstrap):
|
||||||
|
ssh-keygen -t ed25519 -N "" -C "paliad-prod $(date +%Y-%m-%d)" -f /tmp/paliad-prod-key
|
||||||
|
# Public key → mRiver authorized_keys (see 5.2)
|
||||||
|
# Private key → Dokploy secret store as PALIADIN_SSH_PRIVATE_KEY
|
||||||
|
shred -u /tmp/paliad-prod-key # only the encrypted/secret-stored copies survive
|
||||||
|
```
|
||||||
|
|
||||||
|
Rotation: regenerate, push public key to mRiver authorized_keys, update Dokploy secret, redeploy. No code change needed — paliad's startup re-reads the env var on every boot.
|
||||||
|
|
||||||
|
The private key is delivered to the container as a multi-line env var. At process start, paliad writes it to a tmpfile so OpenSSH can use it:
|
||||||
|
|
||||||
|
```go
|
||||||
|
// cmd/server/main.go (sketch)
|
||||||
|
func loadPaliadinSSHKey() (string, error) {
|
||||||
|
blob := os.Getenv("PALIADIN_SSH_PRIVATE_KEY")
|
||||||
|
if blob == "" { return "", nil } // remote mode disabled
|
||||||
|
f, err := os.CreateTemp("", "paliadin-id_ed25519-")
|
||||||
|
if err != nil { return "", err }
|
||||||
|
if err := os.Chmod(f.Name(), 0o600); err != nil { return "", err }
|
||||||
|
if _, err := f.WriteString(blob); err != nil { return "", err }
|
||||||
|
if err := f.Close(); err != nil { return "", err }
|
||||||
|
return f.Name(), nil // path passed to RemotePaliadinService
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The tmpfile lives at `/tmp/paliadin-id_ed25519-<rand>` for the container's lifetime. On container restart, a fresh tmpfile is written. We never persist the key to a volume.
|
||||||
|
|
||||||
|
### 5.2 mRiver `authorized_keys` entry
|
||||||
|
|
||||||
|
```
|
||||||
|
command="/home/m/.local/bin/paliadin-shim",no-pty,no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-user-rc,from="100.99.98.201" ssh-ed25519 AAAA...PUBKEY... paliad-prod
|
||||||
|
```
|
||||||
|
|
||||||
|
Each restriction matters:
|
||||||
|
|
||||||
|
- `command=` — every `ssh m@mriver …` invocation runs the shim regardless of what the client asked for. The client's requested command is exposed as `$SSH_ORIGINAL_COMMAND` for the shim to dispatch on.
|
||||||
|
- `no-pty,no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-user-rc` — defence-in-depth: even if someone steals the key and bypasses the shim's argument validation, they can't get an interactive shell, can't tunnel ports, can't pivot via agent forwarding.
|
||||||
|
- `from="100.99.98.201"` — only accept connections from mLake's tailnet IP. Defends against the "container has full tailnet visibility" host-mode side-effect from §4.2: if the key leaks off mLake, it can't be replayed from another tailnet host.
|
||||||
|
|
||||||
|
### 5.3 Host-key pinning
|
||||||
|
|
||||||
|
`StrictHostKeyChecking=accept-new` is too loose for a long-lived production identity (one-time MITM during first connect substitutes a different key forever). Instead:
|
||||||
|
|
||||||
|
- During Phase A, run `ssh-keyscan -t ed25519 100.99.98.203` on mLake.
|
||||||
|
- Capture the single output line.
|
||||||
|
- Store as Dokploy secret `PALIADIN_KNOWN_HOSTS`.
|
||||||
|
- At container startup, write to `/tmp/paliadin-known_hosts` chmod 644.
|
||||||
|
- Pass to OpenSSH via `-o UserKnownHostsFile=/tmp/paliadin-known_hosts -o StrictHostKeyChecking=yes`.
|
||||||
|
|
||||||
|
If mRiver's host key ever rotates (rare; only on disk wipe / fresh OS), Phase A runs again and the secret is updated. SSH refuses to connect with a clear "host key changed" error, which surfaces as `mriver_unreachable` to the user — exactly the right blast-radius (loud failure, no silent connect to a substitute host).
|
||||||
|
|
||||||
|
### 5.4 The shim — `paliadin-shim`
|
||||||
|
|
||||||
|
A bash script on mRiver at `/home/m/.local/bin/paliadin-shim`. It is the **only** thing the paliad-prod key is allowed to invoke, and it dispatches on `$SSH_ORIGINAL_COMMAND`. Three RPCs:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
# paliadin-shim — server-side RPC for paliad's remote-tmux turns.
|
||||||
|
# Invoked via authorized_keys command= with $SSH_ORIGINAL_COMMAND set.
|
||||||
|
set -euo pipefail
|
||||||
|
umask 077
|
||||||
|
|
||||||
|
readonly TMUX_SESSION="${PALIADIN_TMUX_SESSION:-paliad-paliadin}"
|
||||||
|
readonly RESPONSE_DIR="${PALIADIN_RESPONSE_DIR:-/tmp/paliadin}"
|
||||||
|
readonly TIMEOUT_S=60
|
||||||
|
readonly TURN_ID_RE='^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$'
|
||||||
|
|
||||||
|
mkdir -p "$RESPONSE_DIR"
|
||||||
|
|
||||||
|
# Parse $SSH_ORIGINAL_COMMAND. Format: "<verb> <arg1> <arg2> …"
|
||||||
|
read -r -a argv <<< "${SSH_ORIGINAL_COMMAND:-}"
|
||||||
|
verb="${argv[0]:-}"
|
||||||
|
|
||||||
|
ensure_pane() {
|
||||||
|
if ! tmux has-session -t "$TMUX_SESSION" 2>/dev/null; then
|
||||||
|
tmux new-session -d -s "$TMUX_SESSION"
|
||||||
|
fi
|
||||||
|
# Find or create the @paliadin-scope=chat window.
|
||||||
|
local target=""
|
||||||
|
while read -r idx; do
|
||||||
|
scope=$(tmux show-window-option -t "$TMUX_SESSION:$idx" -v @paliadin-scope 2>/dev/null || true)
|
||||||
|
if [[ "$scope" == "chat" ]]; then target="$TMUX_SESSION:$idx"; break; fi
|
||||||
|
done < <(tmux list-windows -t "$TMUX_SESSION" -F '#{window_index}')
|
||||||
|
if [[ -z "$target" ]]; then
|
||||||
|
idx=$(tmux new-window -t "$TMUX_SESSION" -n claude-paliadin -P -F '#{window_index}' claude)
|
||||||
|
target="$TMUX_SESSION:$idx"
|
||||||
|
# Wait for claude to settle (60s bound; matches Go waitForPaneReady).
|
||||||
|
for _ in $(seq 1 120); do
|
||||||
|
pane=$(tmux capture-pane -t "$target" -p 2>/dev/null || true)
|
||||||
|
if [[ "$pane" == *"❯"* || "$pane" == *"│"* ]]; then break; fi
|
||||||
|
sleep 0.5
|
||||||
|
done
|
||||||
|
tmux set-window-option -t "$target" @paliadin-scope chat
|
||||||
|
tmux set-window-option -t "$target" @fix-name claude-paliadin
|
||||||
|
# Bootstrap system prompt — reuses the Go service's prompt text.
|
||||||
|
# The Go side sends this via the `bootstrap` RPC on first turn instead
|
||||||
|
# of duplicating the prompt here. See §6.4.
|
||||||
|
fi
|
||||||
|
echo "$target"
|
||||||
|
}
|
||||||
|
|
||||||
|
case "$verb" in
|
||||||
|
health)
|
||||||
|
# Liveness check — used by paliad to short-circuit when mRiver is offline.
|
||||||
|
# Returns "ok" iff tmux + claude are reachable.
|
||||||
|
tmux has-session -t "$TMUX_SESSION" 2>/dev/null \
|
||||||
|
|| tmux new-session -d -s "$TMUX_SESSION"
|
||||||
|
command -v claude >/dev/null && echo ok || { echo no-claude; exit 1; }
|
||||||
|
;;
|
||||||
|
|
||||||
|
bootstrap)
|
||||||
|
# First-turn-only: ensure pane exists and inject the system prompt.
|
||||||
|
# $1 = base64-encoded prompt body (avoids quoting hell).
|
||||||
|
target=$(ensure_pane)
|
||||||
|
prompt=$(printf '%s' "${argv[1]:?missing prompt}" | base64 -d)
|
||||||
|
tmux send-keys -t "$target" -l -- "$prompt"
|
||||||
|
tmux send-keys -t "$target" Enter
|
||||||
|
sleep 2 # give claude a moment to absorb
|
||||||
|
echo ok
|
||||||
|
;;
|
||||||
|
|
||||||
|
run-turn)
|
||||||
|
# $1 = turn_id (UUID); $2 = base64-encoded user message.
|
||||||
|
turn_id="${argv[1]:?missing turn_id}"
|
||||||
|
[[ "$turn_id" =~ $TURN_ID_RE ]] || { echo >&2 "bad turn_id"; exit 2; }
|
||||||
|
msg=$(printf '%s' "${argv[2]:?missing message}" | base64 -d)
|
||||||
|
target=$(ensure_pane)
|
||||||
|
out="$RESPONSE_DIR/$turn_id.txt"
|
||||||
|
rm -f "$out"
|
||||||
|
# Envelope matches what paliadin_prompt.go expects.
|
||||||
|
tmux send-keys -t "$target" -l -- "[PALIADIN:$turn_id] $msg"
|
||||||
|
tmux send-keys -t "$target" Enter
|
||||||
|
# Poll for the response file. Same shape as Go pollForResponse.
|
||||||
|
for _ in $(seq 1 $((TIMEOUT_S * 5))); do
|
||||||
|
if [[ -s "$out" ]]; then
|
||||||
|
sleep 0.05 # settle
|
||||||
|
cat "$out"
|
||||||
|
rm -f "$out"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
sleep 0.2
|
||||||
|
done
|
||||||
|
echo >&2 "paliadin: response timeout after ${TIMEOUT_S}s"
|
||||||
|
exit 124
|
||||||
|
;;
|
||||||
|
|
||||||
|
reset)
|
||||||
|
# /clear the conversation; next turn starts fresh.
|
||||||
|
target=$(ensure_pane)
|
||||||
|
tmux send-keys -t "$target" -l -- "/clear"
|
||||||
|
tmux send-keys -t "$target" Enter
|
||||||
|
echo ok
|
||||||
|
;;
|
||||||
|
|
||||||
|
*)
|
||||||
|
echo >&2 "paliadin-shim: unknown verb '$verb'"
|
||||||
|
exit 2
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
```
|
||||||
|
|
||||||
|
Why a shim instead of raw tmux-over-SSH:
|
||||||
|
|
||||||
|
- One SSH round-trip per turn (~50 ms over tailnet) vs ~10–20 round-trips for the granular pattern.
|
||||||
|
- Argument validation lives in one place (UUID regex on turn_id, base64 for messages, fixed verb list) — easier to audit than a regex over `$SSH_ORIGINAL_COMMAND` matching `tmux send-keys …`.
|
||||||
|
- mRiver-side concerns (response polling, settle delays, pane-readiness) stay on mRiver, which is where the tmux state lives. The Go service stops caring about local file polling at all.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Sub-design C — Service-layer integration, routing, reliability
|
||||||
|
|
||||||
|
### 6.1 Interface split
|
||||||
|
|
||||||
|
The current `*PaliadinService` becomes an interface with two implementations: `LocalPaliadinService` (the existing tmux code, renamed) and `RemotePaliadinService` (the new SSH code). Construction picks one at startup based on `PALIADIN_REMOTE_HOST`.
|
||||||
|
|
||||||
|
```go
|
||||||
|
// internal/services/paliadin.go (after refactor)
|
||||||
|
|
||||||
|
type Paliadin interface {
|
||||||
|
RunTurn(ctx context.Context, req TurnRequest) (*TurnResult, error)
|
||||||
|
ResetSession(ctx context.Context) error
|
||||||
|
ListRecentTurns(ctx context.Context, callerID uuid.UUID, limit int) ([]PaliadinTurn, error)
|
||||||
|
Stats(ctx context.Context, callerID uuid.UUID) (*PaliadinStats, error)
|
||||||
|
IsOwner(ctx context.Context, userID uuid.UUID) (bool, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// LocalPaliadinService wraps the current tmux PoC (laptop / dev path).
|
||||||
|
type LocalPaliadinService struct { /* identical to today's PaliadinService */ }
|
||||||
|
|
||||||
|
// RemotePaliadinService talks to a paliadin-shim over SSH on mRiver.
|
||||||
|
type RemotePaliadinService struct {
|
||||||
|
db *sqlx.DB
|
||||||
|
users *UserService
|
||||||
|
sshHost string // 100.99.98.203
|
||||||
|
sshUser string // m
|
||||||
|
sshKeyPath string // /tmp/paliadin-id_ed25519-<rand>
|
||||||
|
knownHosts string // /tmp/paliadin-known_hosts
|
||||||
|
turnMu sync.Mutex
|
||||||
|
|
||||||
|
// Health-check cache.
|
||||||
|
healthMu sync.Mutex
|
||||||
|
healthOK bool
|
||||||
|
healthCheckedAt time.Time
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
DB access (`ListRecentTurns`, `Stats`, `IsOwner`) is identical for both — they only read `paliad.paliadin_turns`. They live in a shared `paliadinDB` helper struct embedded in both implementations.
|
||||||
|
|
||||||
|
### 6.2 Wiring at startup
|
||||||
|
|
||||||
|
```go
|
||||||
|
// cmd/server/main.go (excerpt)
|
||||||
|
var paliadin services.Paliadin
|
||||||
|
remoteHost := os.Getenv("PALIADIN_REMOTE_HOST")
|
||||||
|
switch {
|
||||||
|
case remoteHost != "":
|
||||||
|
keyPath, err := loadPaliadinSSHKey()
|
||||||
|
if err != nil { log.Fatalf("paliadin: load ssh key: %v", err) }
|
||||||
|
if keyPath == "" { log.Fatalf("paliadin: PALIADIN_REMOTE_HOST set but no PALIADIN_SSH_PRIVATE_KEY") }
|
||||||
|
knownHosts, err := loadPaliadinKnownHosts()
|
||||||
|
if err != nil { log.Fatalf("paliadin: load known_hosts: %v", err) }
|
||||||
|
paliadin = services.NewRemotePaliadinService(db, userSvc, services.RemotePaliadinConfig{
|
||||||
|
SSHHost: remoteHost,
|
||||||
|
SSHUser: cmpOr(os.Getenv("PALIADIN_REMOTE_USER"), "m"),
|
||||||
|
SSHKeyPath: keyPath,
|
||||||
|
KnownHostsPath: knownHosts,
|
||||||
|
})
|
||||||
|
log.Printf("paliadin: remote mode → ssh %s@%s", "m", remoteHost)
|
||||||
|
case localTmuxAvailable():
|
||||||
|
paliadin = services.NewLocalPaliadinService(db, userSvc, "", "")
|
||||||
|
log.Printf("paliadin: local tmux mode")
|
||||||
|
default:
|
||||||
|
paliadin = services.NewDisabledPaliadinService(db, userSvc)
|
||||||
|
log.Printf("paliadin: disabled (no remote host, no local tmux)")
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`NewDisabledPaliadinService` exists today implicitly via the `ErrTmuxUnavailable` path; making it explicit gives the constructor a clear name and the handler doesn't have to special-case `nil`.
|
||||||
|
|
||||||
|
### 6.3 SSH invocation pattern
|
||||||
|
|
||||||
|
`RemotePaliadinService` runs every RPC through the same helper:
|
||||||
|
|
||||||
|
```go
|
||||||
|
func (s *RemotePaliadinService) callShim(ctx context.Context, args ...string) ([]byte, error) {
|
||||||
|
sshArgs := []string{
|
||||||
|
"-i", s.sshKeyPath,
|
||||||
|
"-o", "UserKnownHostsFile=" + s.knownHostsPath,
|
||||||
|
"-o", "StrictHostKeyChecking=yes",
|
||||||
|
"-o", "BatchMode=yes",
|
||||||
|
"-o", "ConnectTimeout=3",
|
||||||
|
"-o", "ServerAliveInterval=10",
|
||||||
|
"-o", "ServerAliveCountMax=3",
|
||||||
|
s.sshUser + "@" + s.sshHost,
|
||||||
|
"--",
|
||||||
|
}
|
||||||
|
sshArgs = append(sshArgs, args...)
|
||||||
|
c, cancel := context.WithTimeout(ctx, 70*time.Second) // shim has its own 60s; +10s for SSH overhead
|
||||||
|
defer cancel()
|
||||||
|
cmd := exec.CommandContext(c, "ssh", sshArgs...)
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
cmd.Stdout = &stdout; cmd.Stderr = &stderr
|
||||||
|
if err := cmd.Run(); err != nil {
|
||||||
|
return nil, fmt.Errorf("paliadin: ssh shim %v: %w (stderr: %s)", args, err, stderr.String())
|
||||||
|
}
|
||||||
|
return stdout.Bytes(), nil
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`RunTurn` becomes:
|
||||||
|
|
||||||
|
```go
|
||||||
|
func (s *RemotePaliadinService) RunTurn(ctx context.Context, req TurnRequest) (*TurnResult, error) {
|
||||||
|
s.turnMu.Lock()
|
||||||
|
defer s.turnMu.Unlock()
|
||||||
|
|
||||||
|
if err := s.healthGate(ctx); err != nil {
|
||||||
|
return nil, err // ErrMRiverUnreachable, picked up by handler
|
||||||
|
}
|
||||||
|
|
||||||
|
turnID := uuid.New()
|
||||||
|
started := time.Now().UTC()
|
||||||
|
if err := s.insertTurnRow(ctx, …); err != nil { return nil, err }
|
||||||
|
|
||||||
|
// First-turn-only: bootstrap the system prompt on mRiver. Detected by
|
||||||
|
// checking whether any prior turn for this user has succeeded.
|
||||||
|
if err := s.ensureBootstrapped(ctx); err != nil {
|
||||||
|
_ = s.markTurnError(ctx, turnID, "bootstrap_failed")
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
msg := sanitiseForTmux(req.UserMessage)
|
||||||
|
msgB64 := base64.StdEncoding.EncodeToString([]byte(msg))
|
||||||
|
body, err := s.callShim(ctx, "run-turn", turnID.String(), msgB64)
|
||||||
|
if err != nil {
|
||||||
|
_ = s.markTurnError(ctx, turnID, classifySSHError(err))
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Same trailer-parse + audit-row writes as Local, factored into shared helper.
|
||||||
|
return s.completeTurnFromBody(ctx, turnID, started, string(body))
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6.4 System prompt bootstrap
|
||||||
|
|
||||||
|
The local PoC calls `paliadinSystemPrompt(s.responseDir)` once when it creates the pane. The remote path needs the same hook. Two options that don't require duplicating the German prompt body to mRiver:
|
||||||
|
|
||||||
|
- **Lazy bootstrap (chosen):** the first `RunTurn` after a paliad-prod restart sends the system prompt via `bootstrap` RPC, then runs the actual turn. Subsequent turns skip the bootstrap. State is per-process: `RemotePaliadinService.bootstrapped` boolean guarded by mutex.
|
||||||
|
- Eager bootstrap at startup is rejected — it forces every container start to wait for mRiver to be online, which couples paliad's boot to mRiver's availability.
|
||||||
|
|
||||||
|
Lazy bootstrap means the very first turn after a paliad redeploy pays a ~3 s extra cost (claude pane spin-up + system prompt absorb). Acceptable for a single-user PoC.
|
||||||
|
|
||||||
|
### 6.5 Health-check gating (`mriver_unreachable`)
|
||||||
|
|
||||||
|
Every `RunTurn` first calls `healthGate(ctx)`:
|
||||||
|
|
||||||
|
- Cached for 10 s. If last check was <10 s ago and was OK, skip the probe.
|
||||||
|
- Otherwise: `s.callShim(ctx, "health")` with a 3 s timeout. On success, set cache OK; on failure, return `ErrMRiverUnreachable`.
|
||||||
|
|
||||||
|
Why 10 s: short enough that "I just woke my laptop" propagates inside one user retry; long enough that a busy chat doesn't probe on every turn.
|
||||||
|
|
||||||
|
```go
|
||||||
|
var ErrMRiverUnreachable = errors.New("paliadin: mriver unreachable")
|
||||||
|
|
||||||
|
func (s *RemotePaliadinService) healthGate(ctx context.Context) error {
|
||||||
|
s.healthMu.Lock()
|
||||||
|
defer s.healthMu.Unlock()
|
||||||
|
if s.healthOK && time.Since(s.healthCheckedAt) < 10*time.Second {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
c, cancel := context.WithTimeout(ctx, 3*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
out, err := s.callShim(c, "health")
|
||||||
|
s.healthCheckedAt = time.Now()
|
||||||
|
if err != nil || strings.TrimSpace(string(out)) != "ok" {
|
||||||
|
s.healthOK = false
|
||||||
|
return fmt.Errorf("%w: %v", ErrMRiverUnreachable, err)
|
||||||
|
}
|
||||||
|
s.healthOK = true
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6.6 Friendly error code (extends t-paliad-150)
|
||||||
|
|
||||||
|
`friendlyErrorMessage` already maps `tmux_unavailable` to a localised message. We add one new code:
|
||||||
|
|
||||||
|
- `mriver_unreachable` → DE: *"mRiver ist offline — Paliadin nicht erreichbar. Mach mRiver an, oder nutze Paliadin lokal mit `./paliad`."* / EN: *"mRiver is offline — Paliadin can't reach it. Wake mRiver, or run Paliadin locally with `./paliad`."*
|
||||||
|
|
||||||
|
Implementation: one new `case` in the SSE-error switch in `frontend/src/client/paliadin.ts`'s `friendlyErrorMessage`, plus matching i18n keys (`paliadin.error.mriver_unreachable.de` / `.en`). Server-side: `paliadin` HTTP handler maps `errors.Is(err, services.ErrMRiverUnreachable)` to `event: error\ndata: {"code":"mriver_unreachable","message":"..."}\n\n`.
|
||||||
|
|
||||||
|
### 6.7 Rate limit
|
||||||
|
|
||||||
|
A runaway loop on the paliad side could DOS the SSH connection. Cheapest cap: enforce one in-flight turn at a time via `turnMu` (already exists in the local PoC). On top of that, a rolling cap of N=20 turns/min in `RemotePaliadinService` rejects with `ErrRateLimited` (mapped to a friendly `paliadin.error.rate_limited`). PoC has one user (m); the cap is a paranoid safety, not a real throttle.
|
||||||
|
|
||||||
|
### 6.8 What about ControlMaster?
|
||||||
|
|
||||||
|
Decision-2's chosen path (server-side shim with one RPC per turn) makes ControlMaster optional. The shim collapses ~10 raw-tmux ops into a single SSH connect — that's already the latency win ControlMaster would buy.
|
||||||
|
|
||||||
|
Adding it on top would save ~30–50 ms per turn but adds:
|
||||||
|
|
||||||
|
- A persistent `~/.ssh/cm-*` socket inside the container.
|
||||||
|
- Cleanup logic on shutdown.
|
||||||
|
- A subtle interaction with the SSH BatchMode + ConnectTimeout settings.
|
||||||
|
|
||||||
|
Verdict: skip ControlMaster in v1. If turn latency over Tailscale is measured >300 ms in practice and hot enough to matter, add it in a follow-up; the call site is one helper.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Phasing
|
||||||
|
|
||||||
|
### Phase A — manual proof-of-concept (no Dockerfile change yet)
|
||||||
|
|
||||||
|
Goal: validate the round-trip end-to-end on a deployed paliad, before touching the image.
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
|
||||||
|
1. **Generate keypair** on mRiver: `ssh-keygen -t ed25519 -N "" -C "paliad-prod" -f /tmp/paliad-prod-key`.
|
||||||
|
2. **Install shim** at `/home/m/.local/bin/paliadin-shim` (script from §5.4), `chmod 755`.
|
||||||
|
3. **Write authorized_keys** with the public key + restrictions from §5.2.
|
||||||
|
4. **Capture mRiver host key**: `ssh-keyscan -t ed25519 100.99.98.203 > /tmp/paliad-known_hosts` from mLake.
|
||||||
|
5. **Confirm host networking trade-off (§4.2):** flip the running paliad-prod compose to `network_mode: host` on a temporary branch; redeploy via Dokploy; verify `https://paliad.de/` still serves correctly via traefik. **Gate**: if traefik 502s, abort Phase A and revisit decision 1 in a follow-up issue.
|
||||||
|
6. **Smoke-test SSH from inside the container**:
|
||||||
|
```
|
||||||
|
docker exec -it paliad-prod sh
|
||||||
|
apk add --no-cache openssh-client # one-shot, before Dockerfile change
|
||||||
|
ssh -i /tmp/key -o UserKnownHostsFile=/tmp/known_hosts m@100.99.98.203 health
|
||||||
|
# expected: "ok"
|
||||||
|
ssh … run-turn $(uuidgen) "$(printf 'Hallo Paliadin' | base64 -w0)"
|
||||||
|
# expected: response body, then ".../uuid.txt" cleaned up
|
||||||
|
```
|
||||||
|
7. **Wire env vars manually** via Dokploy UI for one deploy; confirm `/paliadin` works end-to-end against mRiver.
|
||||||
|
|
||||||
|
If Phase A passes: codify into Phase B. If it fails on (5), the design rolls back to a sidecar in a new issue (decision 1 follow-up). If it fails elsewhere, fix the shim or the SSH config; the architecture is fine.
|
||||||
|
|
||||||
|
### Phase B — bake into Dockerfile + Dokploy secrets
|
||||||
|
|
||||||
|
1. Dockerfile: add `openssh-client` to the final stage (§4.3).
|
||||||
|
2. compose: add `network_mode: host` and the four new env vars (§4.1).
|
||||||
|
3. Dokploy secrets: register `PALIADIN_REMOTE_HOST=100.99.98.203`, `PALIADIN_REMOTE_USER=m`, `PALIADIN_SSH_PRIVATE_KEY=...`, `PALIADIN_KNOWN_HOSTS=...`.
|
||||||
|
4. Code: refactor `PaliadinService` to the interface split (§6.1–§6.2). New file `internal/services/paliadin_remote.go`. Tests: `paliadin_remote_test.go` mocks `callShim` to verify `RunTurn` audit-row writes, error mapping, and `healthGate` caching.
|
||||||
|
5. Ship under one PR; tag t-paliad-151 done.
|
||||||
|
|
||||||
|
### Phase C — friendly errors + monitoring
|
||||||
|
|
||||||
|
1. `paliadin.error.mriver_unreachable` i18n keys + `friendlyErrorMessage` case (§6.6).
|
||||||
|
2. `/admin/paliadin` shows last health-probe result + last successful turn timestamp.
|
||||||
|
3. Optional: `mai-mesh` integration to surface mRiver-offline events to m on Telegram (out-of-band; not gating).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Security review summary
|
||||||
|
|
||||||
|
| Risk | Mitigation |
|
||||||
|
|---|---|
|
||||||
|
| Stolen private key → arbitrary SSH on mRiver | `command=` shim restriction + `from="100.99.98.201"` + ed25519 key + private key only in Dokploy secret store (encrypted at rest) |
|
||||||
|
| Stolen private key → tailnet-wide SSH from non-mLake host | `from="100.99.98.201"` clause |
|
||||||
|
| Container compromise → key extraction | Key written to tmpfile chmod 600, only root inside container can read; alpine container has no shell-on-error trampolines |
|
||||||
|
| Host-key MITM during connect | Pinned `known_hosts`; `StrictHostKeyChecking=yes` |
|
||||||
|
| Shim argument injection (e.g. via `run-turn $(rm -rf /)`) | Shim parses positional args from `$SSH_ORIGINAL_COMMAND` via `read -r -a`; never passes args to a subshell `eval`; turn_id validated by UUID regex; message body always base64-decoded into a single shell variable, never re-evaluated |
|
||||||
|
| Runaway loop → SSH flood | Single-flight `turnMu` + 20/min rolling cap |
|
||||||
|
| `network_mode: host` widens blast radius | The `command=` + `from=` restrictions on mRiver mean container compromise = "can run shim verbs against mRiver only", not "shell on mRiver" |
|
||||||
|
| PaliadinOwnerEmail bypass | Unchanged from PoC: gate is in Go (`/paliadin` 404s for any other user). Even if mRiver SSH key leaks, attacker still needs paliad session as `m@hoganlovells.com`. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Out-of-scope clarifications (for review)
|
||||||
|
|
||||||
|
These were called out in the issue but the design intentionally does not solve them, to keep v1 tight. Each is acknowledged so review knows it wasn't an oversight:
|
||||||
|
|
||||||
|
- **Wake-on-LAN of mRiver:** out of scope. v1's UX when mRiver is asleep is the friendly error from §6.6. Future work: integrate with `mai-mesh` capability fallback.
|
||||||
|
- **Multi-host failover:** out of scope. Only mRiver is targeted.
|
||||||
|
- **Anthropic API fallback when mRiver offline:** out of scope per CLAUDE.md (`ANTHROPIC_API_KEY` reserved for production-v1, unused in PoC).
|
||||||
|
- **ControlMaster:** v1 ships without; revisit if turn latency >300 ms in practice (§6.8).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. File-level deliverables (for the coder shift)
|
||||||
|
|
||||||
|
When this design is approved and the coder shift starts, the work splits roughly into:
|
||||||
|
|
||||||
|
- `Dockerfile` — `+openssh-client`.
|
||||||
|
- `docker-compose.yml` — `network_mode: host`, four new env entries.
|
||||||
|
- `internal/services/paliadin.go` — extract `Paliadin` interface; rename existing to `LocalPaliadinService`; pull DB-only methods (`ListRecentTurns`, `Stats`, `IsOwner`) into a shared embedded `paliadinDB` so both implementations get them for free.
|
||||||
|
- `internal/services/paliadin_remote.go` — new file: `RemotePaliadinService`, `RemotePaliadinConfig`, `callShim`, `healthGate`, `ensureBootstrapped`, `classifySSHError`, `ErrMRiverUnreachable`.
|
||||||
|
- `internal/services/paliadin_remote_test.go` — unit tests with a mocked `callShim`.
|
||||||
|
- `cmd/server/main.go` — env-var-based wiring (§6.2), `loadPaliadinSSHKey`, `loadPaliadinKnownHosts`.
|
||||||
|
- `frontend/src/client/paliadin.ts` — one `case` in `friendlyErrorMessage` for `mriver_unreachable`.
|
||||||
|
- `frontend/src/i18n.ts` — two new keys (`paliadin.error.mriver_unreachable.de` / `.en`).
|
||||||
|
- `scripts/paliadin-shim` — server-side script (§5.4); copied to mRiver during Phase A, not part of any container.
|
||||||
|
- `docs/project-status.md` — note Phase 0.5 (PoC) → Phase 0.6 (Tailscale-SSH prod route).
|
||||||
|
|
||||||
|
No DB migrations needed — `paliad.paliadin_turns` schema already covers everything (`error_code` field already accepts free-form codes including `mriver_unreachable`).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. Open questions for review
|
||||||
|
|
||||||
|
- **Q (m):** Phase A test step 5 expects traefik to keep working under host-mode. If a quick search confirms Dokploy's traefik can't route to host-network services without manual `loadbalancer.server.url` config, we should know before Phase A. Worth a 5-minute Dokploy doc check before merging Phase B.
|
||||||
|
- **Q (m):** Should the `paliadin-shim` script live in this repo (`scripts/paliadin-shim`) and be version-pinned, or is it a one-off that lives only on mRiver? Repo location lets us audit changes; mRiver-only keeps deploy footprint smaller.
|
||||||
|
- **Q (m):** `ANTHROPIC_API_KEY` env var reservation in compose comments — keep the comment line for production-v1, or strip it now that this design supersedes that path for the foreseeable future?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Inventor stops here.** No code shipped this shift. Awaiting m's go/no-go on the design before the coder shift starts.
|
||||||
Reference in New Issue
Block a user