docs(backup-mode): t-paliad-246 — inventor design for admin Backup Mode
Folds the two unbuilt slices of t-paliad-214 (Slice 3 org async + Slice 4 scheduler) into one design under the new "Backup Mode" framing from m/paliad#77. Extends ExportService with WriteOrg + orgSheetQueries — no new service abstraction; reuses the existing writer/audit pipeline. Single new table (paliad.backups, mig 121) for the /admin/backups catalog UI; audit chain stays on paliad.system_audit_log. Escalates 4 material picks to head/m: storage backend (Supabase vs local), bundle format (.zip vs single .xlsx), paliadin_turns inclusion (conflict with prior m decision), scheduler cadence. Six non-material picks default silently to the issue's R. Coder gate intentionally not auto-flipped; head decides post-decisions.
This commit is contained in:
710
docs/design-backup-mode-2026-05-25.md
Normal file
710
docs/design-backup-mode-2026-05-25.md
Normal file
@@ -0,0 +1,710 @@
|
||||
# Paliad Backup Mode — system-wide admin "Admin Excel" snapshot
|
||||
|
||||
Design: cronus (inventor), 2026-05-25.
|
||||
Task: **t-paliad-246** / m/paliad#77.
|
||||
Branch: `mai/cronus/inventor-backup-mode`.
|
||||
Status: READY FOR REVIEW — no code yet, awaiting head/m go-no-go on §11 material picks.
|
||||
|
||||
---
|
||||
|
||||
## 0. TL;DR
|
||||
|
||||
Paliad already ships a per-scope data export (`/api/me/export`, `/api/projects/{id}/export`) built on `ExportService` with sheet-registry + xlsx + JSON + CSV bundling and a `system_audit_log` audit chain. Slice 3 (`org` scope, async) was the last unbuilt slice of t-paliad-214 — and that is exactly what "Backup Mode" wants, **plus** Slice 4 (scheduler). This design folds them together:
|
||||
|
||||
- **`org` scope of `ExportService`** — new `WriteOrg` + `orgSheetQueries(...)` registry. No new service.
|
||||
- **On-demand admin trigger** — `POST /api/admin/backups/run` from `/admin/backups`, gated by the existing `adminGate(users, ...)` middleware.
|
||||
- **Nightly scheduled trigger** — new `BackupScheduler` goroutine modelled on `ReminderService` (top-of-hour ticker, fires at 03:00 UTC).
|
||||
- **Persisted artifact catalog** — new table `paliad.backups` (one row per backup, `kind ∈ {scheduled, on_demand}`, status, storage URL, size, row counts).
|
||||
- **Storage** — open. (R from issue) Supabase Storage bucket `paliad-backups` with 90-day lifecycle; (R from prior design) local disk `PALIAD_EXPORT_DIR`. **Material pick — escalated.** Defaulting to **Supabase Storage** in this draft because m's issue text named it explicitly; m's prior pick favoured local disk to avoid object-store provisioning. See §11 Q1.
|
||||
- **Format** — open. (R from issue) single `.xlsx`; (R from prior design + shipped infra) `.zip` of xlsx + JSON + CSV. **Material pick — escalated.** Defaulting to **`.zip`** in this draft because (a) the writer abstraction already produces a zip, (b) JSON twin is the no-lock-in promise, (c) "Admin Excel" framing is satisfied by the xlsx-inside-the-zip. See §11 Q2.
|
||||
- **`paliadin_turns`** — open. (R from issue) "no redaction for admin"; (R from t-paliad-214 m decision) "never include in org export." **Material pick — escalated**, but defaulting to **EXCLUDE** because the prior decision was explicit, structural, and the most-sensitive-PII rationale still holds. See §11 Q3.
|
||||
- **Audit chain** — write to `paliad.system_audit_log` with `event_type ∈ {backup_created, backup_downloaded, backup_failed}`. No new audit infra.
|
||||
|
||||
Three slices: **A** (on-demand + workbook) ships the whole pipeline against the existing audit chain. **B** (scheduler + storage) adds nightly automation and the chosen artifact store. **C** (admin UI polish) makes `/admin/backups` a proper catalog browser.
|
||||
|
||||
---
|
||||
|
||||
## 1. Premises verified live (2026-05-25)
|
||||
|
||||
Verified against the codebase, not memory.
|
||||
|
||||
### 1.1 What's already shipped (t-paliad-214 Slices 1 + 2)
|
||||
|
||||
- `internal/services/export_service.go` — 1472 lines. `ExportService` with `WritePersonal`, `WriteProject`, sheet-registry abstraction (`sheetQuery`, `collectedSheet`), `buildXLSX/JSON/CSV/README` writers, `writeBundle` outer-zip writer, `WriteAuditRow` + `PatchAuditRowSuccess` + `PatchAuditRowFailure`.
|
||||
- `internal/handlers/export.go` — 290 lines. `GET /api/me/export` (personal), `GET /api/projects/{id}/export?direct_only=0|1` (project). Audit row written **before** the artifact generation; patched on success/failure.
|
||||
- `paliad.system_audit_log` (mig 102, shipped) — generic `event_type` text + `metadata jsonb`; `scope ∈ {org, project, personal}`; `actor_id` FK to `paliad.users` with `ON DELETE SET NULL`, `actor_email` captured at write time. RLS: self-read + admin-read. **Already supports `scope='org'`** — backup writes land here trivially.
|
||||
- `github.com/xuri/excelize/v2 v2.10.1` — already in `go.mod`. No new xlsx dependency needed.
|
||||
- Reference design doc: `docs/design-paliad-data-export-2026-05-19.md` — exhaustive (~600 lines), with m's Q-by-Q decisions captured in §12.
|
||||
- `adminGate(users, gateOnboarded(h))` — the standard admin gate (10+ admin routes; e.g. `handlers.go:567-598`). Just wire the new routes through it.
|
||||
|
||||
### 1.2 What's NOT shipped
|
||||
|
||||
- **No `WriteOrg`** in `ExportService`. No `orgSheetQueries(...)` registry. The org scope is declared in the `ExportScope*` constants but no handler/writer references it yet.
|
||||
- **No `PALIAD_EXPORT_DIR`** env var. No `/var/lib/paliad/exports/` mount on Dokploy. Greenfield on storage.
|
||||
- **No Supabase Storage Go client** in `go.mod`. No `STORAGE_*` env vars. The Go side talks Postgres-only today; Supabase Storage would be a new HTTP-client dependency.
|
||||
- **No backup scheduler.** The only in-process scheduler is `ReminderService` (`internal/services/reminder_service.go`) — top-of-hour aligned ticker with startup catch-up. Same shape works for backups.
|
||||
- **No `paliad.backups`** table. The system_audit_log row alone is too sparse to back a "list past backups + click to download" UI; a dedicated catalog table is wanted.
|
||||
|
||||
### 1.3 New `paliad.*` tables since t-paliad-214 design (mig 103-120)
|
||||
|
||||
The org-scope sheet registry must enumerate these or fall behind reality. Mostly straightforward additions:
|
||||
|
||||
- `approval_suggest_changes` (103)
|
||||
- `user_dashboard_layouts` (109)
|
||||
- `user_checklists`, `checklist_shares`, `checklist_versioning_*` (114-116)
|
||||
- `firm_dashboard_default` (117)
|
||||
- `paliadin_aichat_conversation` (118) — sensitive, treat like `paliadin_turns`
|
||||
- `submission_drafts` (119/120) — content potentially sensitive (draft pleading text)
|
||||
|
||||
The coder enumerates `information_schema.tables WHERE table_schema='paliad'` at registry-build time and reconciles with the static list, raising a warning when the live schema introduces a table the registry doesn't know about (forces a registry update on every new migration touching paliad data).
|
||||
|
||||
### 1.4 Live row counts (informational)
|
||||
|
||||
Per the t-paliad-214 design's premise check (2026-05-19), full-org content was **< 600 user-content rows** + ~1000 reference rows. Total xlsx after compression: ~100 KB. A daily snapshot at this size is cheap to store, cheap to transfer, and well below any storage-lifecycle threshold.
|
||||
|
||||
### 1.5 Conclusion
|
||||
|
||||
**No new service abstraction is needed.** `ExportService` is the right home; extend it with `WriteOrg` and a small handler + scheduler + catalog table around it.
|
||||
|
||||
---
|
||||
|
||||
## 2. m's decisions (post head → m round-trip)
|
||||
|
||||
Pending. To be filled in once head returns m's answers on §11. Drafted with R defaults so the slice plan stays coherent in the meantime.
|
||||
|
||||
---
|
||||
|
||||
## 3. Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ Trigger surfaces │
|
||||
│ │
|
||||
│ (a) on-demand: POST /api/admin/backups/run │
|
||||
│ ─ admin only via adminGate(users, ...) │
|
||||
│ ─ enqueues a backup job, returns 202 + {job_id, audit_id} │
|
||||
│ │
|
||||
│ (b) scheduled: BackupScheduler.Start(ctx) │
|
||||
│ ─ top-of-hour ticker, fires at PALIAD_BACKUP_HOUR_UTC=3 │
|
||||
│ ─ runs the same code path as (a) under a synthetic admin id │
|
||||
│ ("system" user, FIRM_NAME), kind='scheduled' │
|
||||
└─────────────────────────┬────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ BackupRunner (the job body, shared by (a) + (b)) │
|
||||
│ │
|
||||
│ 1. INSERT paliad.backups (status='running', kind, requested_by) │
|
||||
│ INSERT paliad.system_audit_log │
|
||||
│ (event_type='backup_created', scope='org', metadata=…) │
|
||||
│ │
|
||||
│ 2. BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ │
|
||||
│ (snapshot consistency for the dump — see §3.3) │
|
||||
│ │
|
||||
│ 3. ExportService.WriteOrg(ctx, &buf, spec) │
|
||||
│ ─ orgSheetQueries(): registry of every paliad.* sheet │
|
||||
│ ─ uses the existing writeBundle() to produce a .zip OR │
|
||||
│ writes a bare .xlsx if §11 Q2 lands on xlsx-only │
|
||||
│ ─ no can_see_project predicate (org scope bypasses RLS via │
|
||||
│ the service-role DB handle paliad already uses) │
|
||||
│ │
|
||||
│ 4. ROLLBACK (read-only tx) │
|
||||
│ │
|
||||
│ 5. ArtifactStore.Put(ctx, key, body) — see §3.2 │
|
||||
│ │
|
||||
│ 6. UPDATE paliad.backups SET status='done', storage_uri=…, │
|
||||
│ size_bytes=…, row_counts=…, finished_at=now() │
|
||||
│ UPDATE paliad.system_audit_log metadata │
|
||||
│ (row_counts, file_size_bytes, storage_uri) │
|
||||
│ │
|
||||
│ On failure at any step ≥ 3: │
|
||||
│ UPDATE paliad.backups SET status='failed', error=… │
|
||||
│ INSERT paliad.system_audit_log │
|
||||
│ (event_type='backup_failed', scope='org', metadata={error}) │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────┐
|
||||
│ Catalog + download │
|
||||
│ │
|
||||
│ GET /admin/backups — page (chronological list) │
|
||||
│ GET /api/admin/backups — JSON list of paliad.backups │
|
||||
│ GET /api/admin/backups/{id} — single row + download URL │
|
||||
│ GET /api/admin/backups/{id}/file — streams the artifact; │
|
||||
│ writes event_type= │
|
||||
│ 'backup_downloaded' to audit │
|
||||
└──────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 3.1 Why extend `ExportService` and not introduce `BackupService`
|
||||
|
||||
m's issue text uses the noun "BackupService" once. That naming pulls a separate code path into the tree. But functionally, a "backup" is exactly what t-paliad-214 called an `org`-scope export — same sheets, same writer, same audit table, same xlsx library. Splitting into two services creates two parallel registries that drift apart on every new migration (the patterns we just fixed in §1.3).
|
||||
|
||||
**Pick: extend `ExportService`.** New methods:
|
||||
- `WriteOrg(ctx, w, spec) (ExportMeta, error)` — mirror of `WritePersonal`/`WriteProject`.
|
||||
- `orgSheetQueries() []sheetQuery` — the full-schema registry (§7).
|
||||
|
||||
The "backup" branding lives at the **handler + UI + scheduler + catalog table** layer. The data-generation layer stays unified.
|
||||
|
||||
### 3.2 Storage abstraction (`ArtifactStore`)
|
||||
|
||||
To survive the §11 Q1 decision without rewriting the handler, isolate storage behind a small interface:
|
||||
|
||||
```go
|
||||
type ArtifactStore interface {
|
||||
Put(ctx context.Context, key string, body io.Reader, size int64) (uri string, err error)
|
||||
Get(ctx context.Context, uri string) (io.ReadCloser, int64, error)
|
||||
Delete(ctx context.Context, uri string) error
|
||||
// List is only needed for the cleanup goroutine; can scan via the
|
||||
// catalog table instead, so this is optional.
|
||||
}
|
||||
```
|
||||
|
||||
Two implementations:
|
||||
- **`LocalDiskStore`** — writes to `$PALIAD_EXPORT_DIR/{backup_id}.{ext}`. `uri` = `file://...`. The bind-mount lives on the Dokploy host (Hostinger VPS disk encryption at-rest). Configure via `PALIAD_EXPORT_DIR`.
|
||||
- **`SupabaseStorageStore`** — uploads via Supabase Storage REST (`POST /storage/v1/object/paliad-backups/{key}`). `uri` = `supabase://paliad-backups/{key}` (resolved to a signed URL at download time). Configure via `SUPABASE_URL` + `SUPABASE_SERVICE_ROLE_KEY` (env var only used by the backend; never sent to clients) + `PALIAD_BACKUP_BUCKET` (default `paliad-backups`).
|
||||
|
||||
Pick **one** at boot via `PALIAD_BACKUP_STORAGE ∈ {local, supabase}`. Default = whichever §11 Q1 lands on. The handler and scheduler are oblivious — they just call `ArtifactStore.Put(...)`.
|
||||
|
||||
The interface also keeps the door open for a future S3/MinIO store without touching the runner.
|
||||
|
||||
### 3.3 Snapshot consistency (REPEATABLE READ)
|
||||
|
||||
Without a snapshot tx, a backup that runs while users are editing produces an internally inconsistent workbook (e.g. a `deadlines` row references a `project_id` that the `projects` sheet just deleted). At paliad's data shape today the window is tiny, but the failure mode is silent — and the whole point of a backup is that you can trust it.
|
||||
|
||||
**Pick: wrap the entire read pass in `BEGIN TRANSACTION ISOLATION LEVEL REPEATABLE READ; ... ROLLBACK;`.** No data is mutated by the backup, so the rollback is just bookkeeping. Postgres holds a consistent snapshot for the duration; concurrent writers see no lock contention (snapshot is per-tx).
|
||||
|
||||
Cost: an open read-only tx for ~1-2s at firm-scale, ~minutes at thousands-of-projects scale. Acceptable.
|
||||
|
||||
### 3.4 What runs the scheduled backup as
|
||||
|
||||
The scheduled backup needs an `actor_id` for the audit row. We don't have a "system" user today. Two options:
|
||||
|
||||
- **Use the `FIRM_NAME` env value as `actor_email`** with `actor_id = NULL` (the FK is `ON DELETE SET NULL`, NULL is allowed at write too). The audit row reads `system / HLC`.
|
||||
- **Seed a `paliad.users` row** with `email='system@<firm-domain>'`, `display_name='Paliad Backup System'`, `global_role='global_admin'`, never bound to Supabase Auth.
|
||||
|
||||
**Pick: option 1 (NULL `actor_id`, `actor_email='system@paliad'`).** Avoids polluting the user list with a phantom; the audit row stays auditable. The `paliad.backups` table mirrors this with a nullable `requested_by` and a `kind='scheduled'` discriminator that any UI can use to render "system" instead of an empty user pill.
|
||||
|
||||
---
|
||||
|
||||
## 4. Schema additions
|
||||
|
||||
One new table, one migration (mig 121, the next free slot per `internal/db/migrations/120_*` being the latest).
|
||||
|
||||
```sql
|
||||
-- 121_backups.up.sql
|
||||
SELECT set_config(
|
||||
'paliad.audit_reason',
|
||||
'mig 121: add paliad.backups catalog table for Backup Mode (t-paliad-246)',
|
||||
true);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS paliad.backups (
|
||||
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
kind text NOT NULL CHECK (kind IN ('scheduled', 'on_demand')),
|
||||
status text NOT NULL CHECK (status IN ('running', 'done', 'failed')),
|
||||
requested_by uuid REFERENCES paliad.users(id) ON DELETE SET NULL,
|
||||
-- requested_by_email captured at write time so the row survives user-deletion
|
||||
requested_by_email text NOT NULL,
|
||||
-- Pointer back into system_audit_log for cross-reference. Nullable so a
|
||||
-- backup row can be inserted even if audit write somehow fails first.
|
||||
audit_id uuid REFERENCES paliad.system_audit_log(id) ON DELETE SET NULL,
|
||||
storage_uri text, -- NULL until status='done'
|
||||
size_bytes bigint, -- NULL until status='done'
|
||||
row_counts jsonb NOT NULL DEFAULT '{}'::jsonb,
|
||||
sheet_count int, -- NULL until status='done'
|
||||
warnings jsonb NOT NULL DEFAULT '[]'::jsonb,
|
||||
error text, -- NULL unless status='failed'
|
||||
started_at timestamptz NOT NULL DEFAULT now(),
|
||||
finished_at timestamptz,
|
||||
-- deleted_at marks artifacts the lifecycle cleanup removed from storage
|
||||
-- (the catalog row stays forever — it's part of the audit chain). Without
|
||||
-- this column we can't distinguish "still on disk" from "expired".
|
||||
deleted_at timestamptz
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS backups_started_at_desc_idx
|
||||
ON paliad.backups (started_at DESC);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS backups_kind_status_idx
|
||||
ON paliad.backups (kind, status);
|
||||
|
||||
ALTER TABLE paliad.backups ENABLE ROW LEVEL SECURITY;
|
||||
|
||||
-- Admin-only read (consistent with system_audit_log_select_admin).
|
||||
DROP POLICY IF EXISTS backups_select_admin ON paliad.backups;
|
||||
CREATE POLICY backups_select_admin ON paliad.backups
|
||||
FOR SELECT USING (
|
||||
EXISTS (
|
||||
SELECT 1 FROM paliad.users u
|
||||
WHERE u.id = auth.uid() AND u.global_role = 'global_admin'
|
||||
)
|
||||
);
|
||||
-- No INSERT/UPDATE/DELETE policies: all writes go through the Go service path
|
||||
-- under the migration-runner role (the same role that writes to
|
||||
-- system_audit_log).
|
||||
|
||||
COMMENT ON TABLE paliad.backups IS
|
||||
'Catalog of org-scope backup runs. One row per scheduled or on-demand backup. status transitions: running → done | failed. storage_uri resolves through the ArtifactStore interface (local file:// or supabase://). audit_id links to system_audit_log; the catalog row is duplicate-with-richer-shape, the audit row is the trust signal.';
|
||||
```
|
||||
|
||||
**Why a separate catalog and not just the audit row?**
|
||||
|
||||
- The `/admin/backups` UI needs row-count / size / status as columns, not as nested `metadata` JSON. Reading `system_audit_log` and parsing JSON for every list-row is the wrong shape.
|
||||
- The catalog has a distinct `kind` discriminator (scheduled vs on_demand) — the audit doesn't model that today, and bolting it onto generic `event_type` is fragile.
|
||||
- The catalog's `deleted_at` separates artifact-lifecycle from audit-retention cleanly. Audit rows are eternal; catalog rows mark when the artifact disappeared.
|
||||
|
||||
**No other migrations.** `system_audit_log` already has the right shape; we just write new `event_type` values into it.
|
||||
|
||||
---
|
||||
|
||||
## 5. Service layer
|
||||
|
||||
### 5.1 `ExportService.WriteOrg`
|
||||
|
||||
```go
|
||||
// WriteOrg streams the full-schema org-scope bundle into w. Returns the
|
||||
// meta (incl. row_counts) for the catalog row + audit-row patching.
|
||||
//
|
||||
// Bypasses paliad.can_see_project — this is admin-only and runs under the
|
||||
// service-role DB handle. The handler/scheduler is responsible for the
|
||||
// admin gate before calling this method.
|
||||
//
|
||||
// Wraps the entire read pass in a REPEATABLE READ transaction (see
|
||||
// design-backup-mode-2026-05-25.md §3.3).
|
||||
func (s *ExportService) WriteOrg(ctx context.Context, w io.Writer, spec ExportSpec) (ExportMeta, error) {
|
||||
if spec.Scope != ExportScopeOrg {
|
||||
return ExportMeta{}, fmt.Errorf("WriteOrg: wrong scope %q", spec.Scope)
|
||||
}
|
||||
tx, err := s.db.BeginTxx(ctx, &sql.TxOptions{
|
||||
Isolation: sql.LevelRepeatableRead,
|
||||
ReadOnly: true,
|
||||
})
|
||||
if err != nil { return ExportMeta{}, fmt.Errorf("backup tx: %w", err) }
|
||||
defer tx.Rollback()
|
||||
|
||||
sheets := orgSheetQueries()
|
||||
meta, err := s.writeBundleWithTx(ctx, tx, w, sheets, spec) // mirrors writeBundle
|
||||
if err != nil { return ExportMeta{}, err }
|
||||
return meta, nil
|
||||
}
|
||||
```
|
||||
|
||||
The existing `writeBundle` takes a `*sqlx.DB`; we factor a `runSheetQueryWithTx` variant that takes `*sqlx.Tx`. The xlsx/JSON/CSV writers are pure (no DB) so they're untouched.
|
||||
|
||||
### 5.2 `orgSheetQueries()`
|
||||
|
||||
Returns the full registry. Mirrors `personalSheetQueries` and `projectSheetQueries` in shape. Sheet ordering: entity sheets alphabetical, then `ref__*` reference sheets alphabetical, then `__meta` (handled separately by writeBundle).
|
||||
|
||||
Concrete sheet list (post-§11 Q3 resolution; default EXCLUDE paliadin_turns):
|
||||
|
||||
```
|
||||
appointments
|
||||
approval_policies
|
||||
approval_requests
|
||||
approval_suggest_changes (new since t-paliad-214)
|
||||
backups (the catalog itself — self-reflexive, low rows, useful for backup-of-backups)
|
||||
caldav_sync_log
|
||||
checklist_instances
|
||||
checklist_shares (new)
|
||||
checklist_versions (new)
|
||||
deadlines
|
||||
documents (metadata only, ai_extracted dropped)
|
||||
email_broadcasts
|
||||
email_templates
|
||||
email_template_versions
|
||||
firm_dashboard_default (new)
|
||||
invitations (without raw tokens — Q7 from t-paliad-214)
|
||||
notes
|
||||
parties
|
||||
partner_unit_events
|
||||
partner_unit_members
|
||||
partner_units
|
||||
project_events
|
||||
project_partner_units
|
||||
project_teams
|
||||
projects
|
||||
reminder_log
|
||||
submission_drafts (new — see §6 redaction note if §11 Q3 picks redact)
|
||||
system_audit_log
|
||||
user_caldav_config (without encrypted_password — covered by piiColumnDenyRegex)
|
||||
user_card_layouts
|
||||
user_checklists (new)
|
||||
user_dashboard_layouts (new)
|
||||
user_pinned_projects
|
||||
user_preferences
|
||||
user_views
|
||||
users
|
||||
-- ref:
|
||||
ref__countries
|
||||
ref__courts
|
||||
ref__deadline_concept_event_types
|
||||
ref__deadline_concepts
|
||||
ref__deadline_event_types
|
||||
ref__deadline_rules
|
||||
ref__event_categories
|
||||
ref__event_category_concepts
|
||||
ref__event_types
|
||||
ref__holidays
|
||||
ref__proceeding_types
|
||||
ref__trigger_events
|
||||
```
|
||||
|
||||
**Excluded unconditionally:**
|
||||
- `paliadin_turns`, `paliadin_aichat_conversation` — per §11 Q3 default (prior m decision).
|
||||
- `auth.*` — not ours.
|
||||
- `paliad.paliad_schema_migrations` — operational, no business meaning.
|
||||
- Any `*_pre_NNN` shadow / pre-migration tables — duplicates of live tables.
|
||||
- Anything matched by the existing `piiColumnDenyRegex` (passwords/tokens/secrets/api keys/private keys) at column-discovery time.
|
||||
|
||||
### 5.3 `BackupRunner`
|
||||
|
||||
A new file `internal/services/backup_service.go` holds the orchestration the handler + scheduler share:
|
||||
|
||||
```go
|
||||
// BackupRunner orchestrates one backup run. Used by both the on-demand
|
||||
// handler and the scheduled goroutine.
|
||||
type BackupRunner struct {
|
||||
db *sqlx.DB
|
||||
export *ExportService
|
||||
store ArtifactStore
|
||||
}
|
||||
|
||||
// Run performs one backup. Writes catalog + audit rows; uploads to storage;
|
||||
// returns the catalog row id on success.
|
||||
//
|
||||
// kind discriminates 'scheduled' vs 'on_demand'.
|
||||
// actor: for on_demand, the calling admin's id+email; for scheduled, NULL+system.
|
||||
func (r *BackupRunner) Run(ctx context.Context, kind string, actor Actor) (uuid.UUID, error)
|
||||
|
||||
// Actor is the (caller-id, email, label) tuple for audit + catalog writes.
|
||||
type Actor struct {
|
||||
ID *uuid.UUID
|
||||
Email string
|
||||
Label string
|
||||
}
|
||||
```
|
||||
|
||||
### 5.4 `BackupScheduler`
|
||||
|
||||
```go
|
||||
// BackupScheduler fires a scheduled org backup once per day at the
|
||||
// configured UTC hour. Modelled on ReminderService: top-of-hour aligned
|
||||
// ticker + startup catch-up + dedup via paliad.backups.
|
||||
type BackupScheduler struct {
|
||||
runner *BackupRunner
|
||||
hour int // 0-23, from PALIAD_BACKUP_HOUR_UTC (default 3)
|
||||
clock func() time.Time
|
||||
}
|
||||
|
||||
func (s *BackupScheduler) Start(ctx context.Context) // spawns goroutine
|
||||
|
||||
// On every tick: if local hour == s.hour AND no 'done'/'running' backup
|
||||
// exists for today's UTC date AND kind='scheduled' → spawn r.Run.
|
||||
// Startup catch-up: same check, fires immediately if today's slot has
|
||||
// already passed but no row exists yet (covers redeploys).
|
||||
```
|
||||
|
||||
Reuses the `nextTopOfHour` + `loop` shape from `ReminderService`. Default hour = 3 (UTC).
|
||||
|
||||
### 5.5 `ArtifactStore` implementations
|
||||
|
||||
`internal/services/artifact_store.go` defines the interface + the two impls. ~150 LoC each.
|
||||
|
||||
- `LocalDiskStore` is straightforward: `os.WriteFile`, `os.Open`, `os.Remove`. The directory is created at boot (`os.MkdirAll($PALIAD_EXPORT_DIR, 0700)`).
|
||||
- `SupabaseStorageStore` uses `net/http` against `${SUPABASE_URL}/storage/v1/object/${bucket}/${key}` with the service-role key in the `Authorization` header. No SDK dependency — plain REST. The download path issues a signed URL via `POST /storage/v1/object/sign/${bucket}/${key}` so the browser fetches directly from Storage without proxying through paliad. Signed-URL TTL = 5 min (single-shot click-to-download).
|
||||
|
||||
---
|
||||
|
||||
## 6. UI surface
|
||||
|
||||
### 6.1 `/admin/backups` page
|
||||
|
||||
New page registered alongside the existing admin routes (`handlers.go:567-598` is the admin block):
|
||||
|
||||
```
|
||||
protected.HandleFunc("GET /admin/backups", adminGate(users, gateOnboarded(handleAdminBackupsPage)))
|
||||
protected.HandleFunc("POST /api/admin/backups/run", adminGate(users, handleAdminRunBackup))
|
||||
protected.HandleFunc("GET /api/admin/backups", adminGate(users, handleAdminListBackups))
|
||||
protected.HandleFunc("GET /api/admin/backups/{id}", adminGate(users, handleAdminGetBackup))
|
||||
protected.HandleFunc("GET /api/admin/backups/{id}/file", adminGate(users, handleAdminDownloadBackup))
|
||||
```
|
||||
|
||||
Page layout (rough):
|
||||
|
||||
```
|
||||
Admin · Backups [+ Backup jetzt erstellen]
|
||||
|
||||
Aktuelle Backups
|
||||
┌──────────────┬────────────┬───────────┬─────────┬────────────┬──────────┐
|
||||
│ Erstellt │ Kind │ Status │ Größe │ Zeilen │ Aktion │
|
||||
├──────────────┼────────────┼───────────┼─────────┼────────────┼──────────┤
|
||||
│ 2026-05-25 │ Geplant │ ✓ Fertig │ 142 KB │ 1’842 │ Download │
|
||||
│ 03:00 UTC │ │ │ │ │ │
|
||||
├──────────────┼────────────┼───────────┼─────────┼────────────┼──────────┤
|
||||
│ 2026-05-24 │ Manuell │ ✓ Fertig │ 138 KB │ 1’793 │ Download │
|
||||
│ 14:22 UTC │ │ │ │ (m@hl…) │ │
|
||||
├──────────────┼────────────┼───────────┼─────────┼────────────┼──────────┤
|
||||
│ 2026-05-24 │ Geplant │ ✓ Fertig │ 137 KB │ 1’791 │ Download │
|
||||
│ 03:00 UTC │ │ │ │ │ │
|
||||
└──────────────┴────────────┴───────────┴─────────┴────────────┴──────────┘
|
||||
|
||||
Footer: "Geplante Backups laufen täglich um 03:00 UTC. Aufbewahrung: 90 Tage."
|
||||
```
|
||||
|
||||
Frontend lives in `frontend/src/client/admin-backups.ts` + a `frontend/src/admin/backups.tsx` page entry. Reuses the existing `.entity-table` pattern (deadlines / projects detail). Per `.claude/CLAUDE.md` frontend conventions: row-click handler navigating to a detail view if we want one (deferred — v1 just has the action column).
|
||||
|
||||
The "Backup jetzt erstellen" button POSTs `/api/admin/backups/run`, then polls `/api/admin/backups/{id}` every 2s until `status != 'running'`, then refreshes the list. (At firm-scale this resolves in <2s; the polling pattern still degrades gracefully if a backup grows.)
|
||||
|
||||
### 6.2 Sidebar entry
|
||||
|
||||
The admin sidebar group (per `client/sidebar.ts initAdminGroup`, revealed after `/api/me` confirms `global_role='global_admin'`) gets a "Backups" entry pointing at `/admin/backups`.
|
||||
|
||||
### 6.3 i18n
|
||||
|
||||
New keys under `admin.backups.*`:
|
||||
|
||||
```
|
||||
admin.backups.title "Backups" / "Backups"
|
||||
admin.backups.run_now "Backup jetzt erstellen" / "Run backup now"
|
||||
admin.backups.kind.scheduled "Geplant" / "Scheduled"
|
||||
admin.backups.kind.on_demand "Manuell" / "Manual"
|
||||
admin.backups.status.running "Läuft …" / "Running …"
|
||||
admin.backups.status.done "✓ Fertig" / "✓ Done"
|
||||
admin.backups.status.failed "✗ Fehlgeschlagen" / "✗ Failed"
|
||||
admin.backups.download "Download" / "Download"
|
||||
admin.backups.empty "Noch keine Backups vorhanden." / "No backups yet."
|
||||
admin.backups.footer.note "Geplante Backups laufen täglich um {hour}:00 UTC. Aufbewahrung: {days} Tage."
|
||||
/ "Scheduled backups run daily at {hour}:00 UTC. Retention: {days} days."
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Workbook layout
|
||||
|
||||
### 7.1 Inside the bundle
|
||||
|
||||
Inherits the t-paliad-214 layout verbatim, with one addition:
|
||||
|
||||
```
|
||||
paliad-backup-2026-05-25T0300Z.zip
|
||||
├── README.txt # human-readable: what this is, how to read it
|
||||
├── paliad-export.xlsx # canonical workbook
|
||||
├── paliad-export.json # JSON twin (machine-readable)
|
||||
├── csv/
|
||||
│ ├── projects.csv
|
||||
│ ├── deadlines.csv
|
||||
│ ├── ...
|
||||
│ └── ref/
|
||||
│ ├── proceeding_types.csv
|
||||
│ └── ...
|
||||
└── __meta.json # standalone meta (same content as __meta sheet)
|
||||
```
|
||||
|
||||
If §11 Q2 picks xlsx-only, the bundle collapses to a single `paliad-backup-{timestamp}.xlsx` and the JSON/CSV twins disappear. (Inventor recommends against — see Q2 reasoning.)
|
||||
|
||||
### 7.2 Filename convention
|
||||
|
||||
```
|
||||
paliad-backup-{timestamp}.zip (or .xlsx if Q2 = single-file)
|
||||
timestamp = YYYY-MM-DDTHHMMZ # UTC, no colons (Windows-safe)
|
||||
```
|
||||
|
||||
Note the prefix is `paliad-backup-` (not `paliad-export-org-`) so the audience reads it as a backup, not a generic export. Both still produced by `ExportService`; the filename is the disambiguator.
|
||||
|
||||
### 7.3 The xlsx sheet list
|
||||
|
||||
Per the registry in §5.2. Sheet 1 = `__meta` (frozen header + key-value pairs from `ExportMeta`). The next several sheets are entity tables (alphabetical), then `ref__*` reference sheets, then a final `__lookup` sheet pairing every FK UUID to a human-readable label (project title, user email) so the workbook is self-joining in Power Query / Excel pivot tables.
|
||||
|
||||
Inherits all formatting decisions from t-paliad-214 §3.1: ISO 8601 strings for dates, `TRUE`/`FALSE` for booleans, semicolon-joined arrays, snake_case sheet names, frozen header row, column 1 always `id`.
|
||||
|
||||
### 7.4 The README.txt
|
||||
|
||||
A backup-specific variant of the t-paliad-214 README:
|
||||
|
||||
```
|
||||
Paliad Backup — {firm_name}
|
||||
Erzeugt am {generated_at} ({kind}: {scheduled | on-demand by {actor_email}})
|
||||
Paliad-Version: {git_sha}
|
||||
|
||||
Dieses Archiv enthält einen vollständigen Snapshot der paliad-Datenbank
|
||||
zum Zeitpunkt der Erstellung. Es ist auf den Admin-Bereich beschränkt
|
||||
und enthält potenziell vertrauliche Mandantendaten.
|
||||
|
||||
Inhalt
|
||||
- paliad-export.xlsx — kanonisches Excel-Workbook (eine Sheet pro Tabelle)
|
||||
- paliad-export.json — JSON-Variante für maschinelle Re-Ingestion
|
||||
- csv/ — Pro-Tabelle CSV-Dateien (UTF-8 mit BOM)
|
||||
- __meta.json — Snapshot-Metadaten
|
||||
|
||||
Sheet-Übersicht
|
||||
{row_counts as YAML}
|
||||
|
||||
Aufbewahrung
|
||||
Dieser Snapshot wird automatisch nach {retention_days} Tagen aus dem
|
||||
Speicherort entfernt. Der Audit-Eintrag in paliad.system_audit_log
|
||||
bleibt dauerhaft erhalten.
|
||||
|
||||
Weitergabe
|
||||
Die Weitergabe dieses Backups an Dritte erfolgt in eigener Verantwortung
|
||||
des Empfängers. Watermarks oder DRM sind nicht enthalten.
|
||||
```
|
||||
|
||||
(English mirror appended below.)
|
||||
|
||||
---
|
||||
|
||||
## 8. Permissions + audit
|
||||
|
||||
### 8.1 Permissions
|
||||
|
||||
- All `/admin/backups*` routes go through `adminGate(users, gateOnboarded(h))`. Pattern is set; no new middleware.
|
||||
- Service-role DB handle is the same one paliad uses today (it's just `*sqlx.DB` connected via `DATABASE_URL`). No RLS bypass machinery — the writer just runs queries; RLS isn't applied because `auth.uid()` is unset at this connection layer.
|
||||
- `paliad.backups` has admin-only SELECT RLS as a defense-in-depth (no end-user write surface; INSERT/UPDATE happen via the Go service path under the migration-runner role, same as `system_audit_log`).
|
||||
- The Supabase Storage bucket `paliad-backups` (if §11 Q1 picks Supabase): bucket policy restricts to service-role only; no anon read. Signed URLs are issued at download time by the backend with 5-min TTL.
|
||||
|
||||
### 8.2 Audit chain
|
||||
|
||||
Three event types on `paliad.system_audit_log`:
|
||||
|
||||
| event_type | when | metadata.* fields |
|
||||
|----------------------|-----------------------------------------|-------------------|
|
||||
| `backup_created` | catalog row INSERT (status='running') | `{kind, paliad_version, requested_by_email}` |
|
||||
| (patched on success) | catalog row UPDATE (status='done') | `+ {row_counts, file_size_bytes, sheet_count, storage_uri}` |
|
||||
| `backup_failed` | catalog row UPDATE (status='failed') | `+ {error, partial_row_counts}` |
|
||||
| `backup_downloaded` | per download click on /admin/backups | `{backup_id, downloaded_by_email}` |
|
||||
|
||||
`backup_created` reuses the same row across success/failure (UPDATE the metadata), matching the `data_export` pattern in `ExportService.WriteAuditRow` + `PatchAuditRowSuccess`/`PatchAuditRowFailure`. `backup_downloaded` is a new row per click.
|
||||
|
||||
Audit rows persist forever — the artifact lifecycle (90-day cleanup of the file on Storage) does **not** touch them. The chain is the trust signal.
|
||||
|
||||
---
|
||||
|
||||
## 9. Slice plan
|
||||
|
||||
Tracer-bullet: each slice ships and is reviewable alone. v1 = A + B + C.
|
||||
|
||||
### Slice A — On-demand backup + workbook generator (MVP path)
|
||||
|
||||
- `paliad.backups` migration (mig 121).
|
||||
- `ExportService.WriteOrg` + `orgSheetQueries()`.
|
||||
- `BackupRunner` struct + `Run(ctx, kind, actor)` method.
|
||||
- `LocalDiskStore` only (defer Supabase Store to Slice B even if §11 Q1 picks Supabase — Slice A still works against `/var/lib/paliad/exports/` and lets the rest of the pipeline ship).
|
||||
- Handlers: `POST /api/admin/backups/run`, `GET /api/admin/backups`, `GET /api/admin/backups/{id}`, `GET /api/admin/backups/{id}/file`.
|
||||
- Tests: unit on the registry shape (one entity sheet per known table), integration on the runner (insert → tx → upload → patch), handler-level test on the admin gate.
|
||||
- **Ships** an admin-only "click to generate a full backup, download it" flow against local disk. No scheduler, no Supabase Storage. Already meets the issue's on-demand requirement.
|
||||
|
||||
Estimated LoC: ~800 (most is the registry + WriteOrg, which mirrors the existing WritePersonal).
|
||||
|
||||
### Slice B — Scheduler + final storage backend
|
||||
|
||||
- `BackupScheduler` (modelled on `ReminderService`).
|
||||
- `SupabaseStorageStore` (only if §11 Q1 picks Supabase; otherwise reuse `LocalDiskStore`).
|
||||
- Cleanup goroutine: daily scan of `paliad.backups WHERE finished_at < now() - INTERVAL '90 days' AND deleted_at IS NULL`; calls `store.Delete`; sets `deleted_at`. (For Supabase Storage, this is redundant with bucket-side lifecycle rules — but having paliad assert ownership keeps the catalog accurate even if a bucket rule is misconfigured.)
|
||||
- Env wiring: `PALIAD_BACKUP_STORAGE`, `PALIAD_EXPORT_DIR` (local), `PALIAD_BACKUP_BUCKET` (supabase), `PALIAD_BACKUP_HOUR_UTC`, `PALIAD_BACKUP_RETENTION_DAYS`. All documented in CLAUDE.md "Environment variables" table.
|
||||
- Tests: scheduler dedup (don't fire twice in the same UTC day), storage round-trip (Put → Get → Delete), cleanup goroutine (mark-deleted logic).
|
||||
- **Ships** the daily snapshot to the chosen store + lifecycle.
|
||||
|
||||
Estimated LoC: ~500.
|
||||
|
||||
### Slice C — Admin UI polish
|
||||
|
||||
- `/admin/backups` page TSX + client TS bundle.
|
||||
- "Backup jetzt erstellen" button with polling for the running status.
|
||||
- Empty state + error-state rendering.
|
||||
- Sidebar entry.
|
||||
- i18n keys.
|
||||
- e2e (Playwright) on the page: load → click run → poll → see new row → click download.
|
||||
- **Ships** the catalog UI; before this slice, admins use the API directly.
|
||||
|
||||
Estimated LoC: ~400.
|
||||
|
||||
---
|
||||
|
||||
## 10. Out of scope
|
||||
|
||||
- **Restore-from-backup** tooling — separate phase per issue.
|
||||
- **Per-firm / multi-tenant** separation — paliad is single-firm; not relevant until that changes.
|
||||
- **App-layer encryption at rest** — Supabase Storage encrypts at the bucket layer (or Hostinger disk encrypts at the host layer for local-disk). v1 does not add per-file encryption.
|
||||
- **Differential / incremental backups** — v1 is full-snapshot only. Differentials are appealing only at firm-scale that we're nowhere near (>1M rows).
|
||||
- **Backup retention by count** (e.g. "keep last 30") — v1 is age-based only via the retention env var.
|
||||
- **Export-as-email-attachment** — would require attaching ~150 KB to a daily mail; out of v1.
|
||||
- **Public signed URLs that bypass cookie auth** — the download endpoint requires the admin cookie; no anon download surface in v1.
|
||||
- **A second audit chain table** — `paliad.system_audit_log` is the only chain. The `paliad.backups` catalog is operational metadata, not audit.
|
||||
|
||||
---
|
||||
|
||||
## 11. Material picks for head → m
|
||||
|
||||
Most of the issue's R answers are non-material and the design defaults to them silently. Four are material and get escalated. The design's draft picks (above) reflect the inventor's best guess; m's calls supersede.
|
||||
|
||||
### Q1 — Storage backend: Supabase Storage (issue R) vs. local disk (prior design pick)?
|
||||
|
||||
**Inventor draft pick: Supabase Storage** (matches issue R; m named it explicitly).
|
||||
|
||||
Trade-offs:
|
||||
- **Supabase Storage** — new HTTP-client dep (~200 LoC), but no provisioning (the youpc Supabase already has Storage available). Lifecycle rules are bucket-side. Signed-URL downloads bypass paliad → less bandwidth on the Dokploy host. Backups survive a Dokploy host loss.
|
||||
- **Local disk** — zero external dep, simpler code. Tied to the Dokploy compose volume; backup is lost if the host is lost (which is exactly the disaster a backup defends against). Cleanup is Go-side, slightly more code.
|
||||
|
||||
Material because: the artifact-store choice has a real disaster-recovery implication (the backup of paliad should not live on the same host as paliad). And the prior t-paliad-214 design's "no MinIO provisioning" rationale doesn't apply — the youpc Supabase is already provisioned.
|
||||
|
||||
### Q2 — Bundle format: single `.xlsx` (issue R) vs. `.zip` (shipped infra)?
|
||||
|
||||
**Inventor draft pick: `.zip` bundle** (xlsx + JSON + CSV + README).
|
||||
|
||||
Trade-offs:
|
||||
- **Single `.xlsx`** — matches m's "Admin Excel" framing literally. One file, double-click to open. Loses the JSON twin (Excel-independent re-ingest) — for org scope this re-introduces the lock-in problem the personal/project exports were designed to avoid.
|
||||
- **`.zip` bundle** — preserves no-lock-in. Slight UX friction: admins unzip first to get the xlsx. The .xlsx inside still satisfies "Admin Excel" — m gets the workbook by clicking through one zip layer.
|
||||
|
||||
Material because: the no-lock-in promise was a load-bearing rationale in t-paliad-214 §1. Cutting it for the admin scope only is a real product decision, not a default.
|
||||
|
||||
### Q3 — `paliadin_turns` (+ `paliadin_aichat_conversation`) in the backup: include (issue R) or exclude (prior m decision)?
|
||||
|
||||
**Inventor draft pick: EXCLUDE** (matches prior m decision from t-paliad-214 Q5 addendum).
|
||||
|
||||
Reasoning: the prior m decision was structural ("the `paliadin_turns` row drops from the org-scope sheet table entirely — no `?include=paliadin_turns` query param") with a precedent-setting rationale ("the *moment* Paliadin opens beyond owner-only, the AI conversation history per user is the most sensitive personal data we carry"). That rationale still holds — `paliadin_aichat_conversation` (mig 118) extends the conversation history surface.
|
||||
|
||||
Issue R says "nothing for the admin role; this is an internal backup, not a sharable artifact." That argument is reasonable but undercuts the prior precedent. The conflict is real; flagging it.
|
||||
|
||||
Two flavours of "include":
|
||||
- **All AI conversation tables included as-is** — full backup. Convenient. Most leaky.
|
||||
- **Tables included but with `assistant_response` / message-body columns dropped** — keeps the metadata (who talked to Paliadin when), drops the content. Middle ground.
|
||||
|
||||
### Q4 — Scheduled backup time: nightly 03:00 UTC (draft) or a different hour / cadence?
|
||||
|
||||
**Inventor draft pick: nightly 03:00 UTC.**
|
||||
|
||||
Reasoning: 03:00 UTC = 04:00/05:00 Europe/Berlin (winter/summer), well outside business hours for every paliad office (Munich, Düsseldorf, Amsterdam, London, Paris, Milan, Hamburg). Low write traffic → snapshot-tx contention is minimal. Daily cadence is the issue's stated cadence.
|
||||
|
||||
Material if m wants weekly + monthly + on-status-change (the t-paliad-214 §5.2 sketch). Defaulting to daily keeps Slice B small; multi-cadence belongs in a follow-up.
|
||||
|
||||
The hour is `PALIAD_BACKUP_HOUR_UTC` (default 3), m-tunable per deploy.
|
||||
|
||||
---
|
||||
|
||||
## 11. Open questions for m (FLAG only — no escalation)
|
||||
|
||||
These are coder-time choices that don't move the design. Listed for the coder shift to resolve in implementation chat with m.
|
||||
|
||||
- Exact bucket name if Q1=supabase (`paliad-backups`? `paliad-{firm-slug}-backups`?).
|
||||
- Whether the on-demand button rate-limits (e.g. "only one running backup at a time"). Default: yes, the runner rejects on overlap with a friendly 409.
|
||||
- Whether the `/admin/backups` row-click navigates to a backup-detail page (showing per-sheet row counts, warnings, the originating audit row). Default: no detail page in v1; the table columns are sufficient.
|
||||
- Whether the README.txt also embeds a one-line list of every sheet name. Default: yes (it's two lines of code and very useful for "what tables were in scope on this date").
|
||||
- Whether `submission_drafts.body_md` (the actual draft pleading text) is included verbatim or dropped. Default: included — admins are by definition allowed to see this. Flag to m so the precedent is recorded.
|
||||
|
||||
---
|
||||
|
||||
## 12. References
|
||||
|
||||
- `docs/design-paliad-data-export-2026-05-19.md` — the prior design this extends (Slices 1-2 SHIPPED, Slice 3-4 deferred until this design).
|
||||
- `internal/services/export_service.go` — the writer abstraction we extend with `WriteOrg`.
|
||||
- `internal/services/reminder_service.go` — the in-process scheduler template for `BackupScheduler`.
|
||||
- `internal/db/migrations/102_system_audit_log.up.sql` — the audit chain we reuse.
|
||||
- `internal/handlers/handlers.go:567-598` — the existing admin-gate block; new `/admin/backups*` routes register alongside.
|
||||
- m/paliad#77 — the originating ask.
|
||||
- m/paliad#214 — the prior data-export task (Slices 1-2 shipped).
|
||||
|
||||
---
|
||||
|
||||
**END OF DESIGN. Status: READY FOR REVIEW pending §11 Q1-Q4 material picks.**
|
||||
|
||||
Inventor parks here. Head's `mai-head` skill gates the coder shift; this draft uses the issue's R defaults everywhere except where they conflict with prior shipped m decisions (Q1, Q2, Q3) or where the issue text named no default (Q4).
|
||||
Reference in New Issue
Block a user