paliad/docs/plans/prd-docforge-2026-05-29.md

PRD — docforge: a modular document-generator engine

Task: t-paliad-349 (m/paliad#157) · Author: leibniz (inventor) · Date: 2026-05-29 Status: DESIGN — awaiting head's go/no-go on the coder shift. Supersedes nothing. Extends and re-homes the submission generator designed in docs/design-submission-generator-2026-05-19.md, …-v2-2026-05-26.md, and docs/design-submission-page-2026-05-22.md.

---

§0 Premises

0.1 What this is

m wants the paliad "doc generator" pulled apart into a clean, reusable engine. Verbatim direction (2026-05-29):

I want to be able to create and modify word documents, using variables inside the documents, "editing them live" and preview the results, export in the end. We should have all that modular to keep it clean. The editor is something else than the importing, exporting, variable exchange, data fetching etc.

Currently I can't upload the base document to insert variables into to create a template — and then later I want to fill the template using data, modifying it manually where necessary, then exporting.

Two distinct user surfaces fall out of that:

• Authoring — upload a base .docx → place variable slots into it → save as a reusable template. This is the gap that does not exist today.
• Generation — pick a template → bind variables to project data → manually edit where needed (live editor + preview) → export .docx.

0.2 Today's state (audited 2026-05-29, verified against the live tree)

The current submission generator is ~250 KB of Go plus a 115 KB editor bundle:

• internal/services/submission_vars.go — variable resolution across 7 namespaces (firm.*, today.*, user.*, project.*, parties.*, procedural_event.*
  • rule.* legacy aliases, deadline.*). Resolution is a push model: each namespace is a hardcoded addXxxVars(bag PlaceholderMap, …) function mutating a shared map[string]string. There is no interface and no registry — adding a namespace means hand-editing Build to call a new function.
• internal/services/submission_merge.go — placeholder substitution. The regex (line 95, verified) is \{\{\s*([A-Za-z][A-Za-z0-9_.]*)\s*\}\}. Two-pass: single-run replace inside each <w:t>, then cross-run merge for fragmented placeholders. HTML preview wraps (key,value) in Private-Use-Area sentinels so emitTextWithDraftVars can reconstruct <span class="draft-var" data-var="key">…</span> for click-to-jump.
• internal/services/submission_md.go — Markdown → OOXML runs. parseInlineSpans (lines 393–446) tokenises bold/italic and preserves {{…}} verbatim.
• internal/services/submission_compose.go — assembles the final .docx: unzip base, render each included section's Markdown to OOXML, splice between {{#section:KEY}}…{{/section:KEY}} anchors, patch hyperlink rels, repack, then run the placeholder pass.
• internal/services/submission_{draft,section,building_block,base}_service.go — the draft/section/building-block/base data model + CRUD.
• internal/handlers/submission_{drafts,sections,building_blocks,bases}.go — the HTTP wire (the 53 KB submission_drafts.go is the bulk).
• frontend/src/client/submission-draft.ts — the editor UI (one .ts bundle; there is no submission-draft.tsx — the brief was wrong on this point).

OOXML approach (verified): pure archive/zip + string manipulation of word/document.xml. No third-party docx library — go.mod has none. lukasjarosch/go-docx appears only in a comment (submission_merge.go:13) documenting why it was rejected (it refuses sibling placeholders in one run). The base stays byte-for-byte identical outside the regions we touch.

Reference model: pkg/litigationplanner/ (t-paliad-292). The package owns its types and exposes interfaces for stateful inputs (Catalog, HolidayCalendar, CourtRegistry); paliad implements them against Postgres, youpc.org against an embedded JSON snapshot. doc.go is the package doc; types_wire_test.go locks the JSON contract. docforge mirrors this packaging discipline exactly.

0.3 Premise correction (load-bearing)

The brief lists two consumers in scope: paliad + upc-commentary. Verified against the live repo: UPCommentary/upc-kommentar is Bun + SvelteKit + TypeScript + PLpgSQL — zero Go. A SvelteKit app cannot import a Go pkg/. m's resolution (2026-05-29): upc-kommentar is out of scope as a live consumer for now. docforge is a pure Go package; paliad imports it in-process like litigationplanner. The interfaces are designed so an HTTP veneer (for a future TS consumer) is addable later without rework — but none is built now. See §4 D-P1 and §8.

0.4 Locked constraints (m, confirmed)

• One Go module: pkg/docforge. Same packaging model as pkg/litigationplanner.
• docforge owns no database tables — data flows in via interfaces.
• .docx first; engine designed format-pluggable for .pdf/.html/.md later.
• Authoring and Generation are distinct pages, but share the engine + the generic editor plumbing.
• Generation must support minor manual content edits (live editor, not just data-binding).
• Editor stays per-consumer; the generic UX plumbing is extracted into a reusable UI package now.
• The neutral model must be lossless for our own .docx (the uploaded base is an opaque carrier, preserved byte-for-byte outside touched regions).

0.5 Contracts that MUST survive the refactor

These are invariants. The migration (§6) protects each by moving it with its file and its test, unchanged:

1. placeholderRegex = `\{\{\s*([A-Za-z][A-Za-z0-9_.]*)\s*\}\}` — underscores and dots legal in keys; whitespace inside braces trimmed; case-sensitive.
2. Last night's underscore fix (commit b78a984): parseInlineSpans short-circuits the inline scanner on {{ and copies the placeholder literally to }}, so {{project.case_number}} is never mangled to {{project.casenumber}}.
3. data-var contract — data-var="<key>" on both .draft-var preview spans and .submission-draft-var-input sidebar inputs; the click-to-jump and focus-highlight are bijective across repaints.
4. Missing-value markers — [KEIN WERT: key] (DE) / [NO VALUE: key] (EN) render inline, never an error.
5. Legacy aliases — procedural_event.X ≡ rule.X resolve identically (submission_vars_aliases_test.go); party variables emit comma-joined, indexed, and flat-legacy forms (submission_vars_parties_test.go).
6. Section anchor syntax — {{#section:KEY}}…{{/section:KEY}}, KEY matched against [A-Za-z0-9_]+.
7. No binary retention — exported .docx is regenerable from inputs; only audit rows persist (system_audit_log submission.exported + project_events).
8. V1 fallback path — pre-Composer drafts (base_id IS NULL, no section rows) render via the pure-placeholder path. No auto-upgrade.
9. {{…}} pass-through — the Markdown walker emits placeholders verbatim; the merge pass substitutes them afterward. Order is load-bearing (substitution runs inside compose, after section splicing).

---

§1 Goals

G1. Extract the format-neutral document machinery (Markdown→OOXML walker, OOXML merge/compose, placeholder engine, .dotm→.docx) into pkg/docforge with a clean public surface and zero behavior change at the extraction step.

G2. Introduce a neutral document/template model so importers produce it, the engine binds variables on it, and exporters render it out — with .docx as the first importer+exporter pair, not the universe. Lossless for our own .docx.

G3. Replace the hardcoded addXxxVars push with a VariableResolver interface per namespace + a ResolverSet that composes them, preserves aliases, and exposes the key catalogue (label + group) so the frontend variable form/palette becomes data-driven instead of hardcoded in TS.

G4. Build the Authoring surface: upload .docx → WYSIWYG render → click/select → insert {{slot}} → save template. Closes the gap m named.

G5. Refactor Generation onto docforge + uploaded templates, preserving the live editor, preview, manual-edit, and export — and every contract in §0.5.

G6. Extract the generic editor UX into frontend/src/lib/docforge-editor/, consumed by both the generation and authoring shells.

Non-goals (this PRD): implementation, migration SQL, code. Formats beyond .docx (interface only). Live upc-kommentar integration. Multi-user concurrent editing of one draft. An HTTP service veneer.

---

§2 User journeys

2.1 Authoring (new)

1. m opens /admin/templates (or /templates/new) and uploads a base .docx (firm letterhead with caption layout, signature block, etc.).
2. docforge's .docx importer parses the upload into a carrier (opaque OOXML kept intact) + a renderable preview. The page shows a WYSIWYG-ish render of the document.
3. m highlights a piece of text — e.g. Az. 4c O 12/23 — and a variable palette (sourced from the ResolverSet.Keys() catalogue, grouped DE/EN) lets him pick project.case_number. The selection is replaced with a {{project.case_number}} slot; a template_slots row records the slot key + its anchor position.
4. He repeats for every variable region, saves, and the template becomes pickable in Generation. (Editing the template later creates a new version — see §4 D-A3.)

Scope guard: v1 authoring places text-level slots in body paragraphs. Slots in headers/footers/tables/text-boxes are a flagged follow-up (§7 note), because the click→OOXML-run mapping there is materially harder.

2.2 Generation (refactor of today)

1. Lawyer picks a template (uploaded template or a legacy Gitea base — both supported during transition) for a submission code, optionally project-scoped.
2. A draft is created. Its template structure is snapshotted at create (§4 D-A3) so later template edits don't shift an in-flight draft.
3. The sidebar shows the variable form (data-driven from ResolverSet.Keys()); the resolved bag is merged with the lawyer's overrides; the live preview renders with data-var click-to-jump; manual prose edits autosave (500 ms debounce).
4. Export → docforge binds the model + carrier + resolved variables → .docx bytes stream as a download. Audit rows written. No binary retained.

2.3 upc-kommentar parallel journey (deferred — validates the abstractions)

Not built now, but the abstractions are sized for it: upc-kommentar authors work in Markdown (and want to import foreign doc/docx as input — m, 2026-05-29 Q4). When it becomes a consumer, it would: implement its own VariableResolver(s) over its Postgres (commentary metadata), feed Markdown through docforge's markdown importer into the neutral model, edit live in its own Svelte shell (reusing the wire contract, not Go code), and export. The Go engine is reached over an HTTP veneer added at that point. This journey is the litmus test for §3's seams: a new consumer adds resolvers + a transport, touches no engine internals.

---

§3 Module shape

3.1 Package tree

pkg/docforge/
  doc.go              // package doc (litigationplanner-style)
  model.go            // neutral model: Document, Block, InlineSpan, Slot
  template.go         // Template, TemplateSlot, Carrier
  variables.go        // VariableResolver interface, VariableKey, ResolverSet, alias registry
  bind.go             // binding engine: walk model, resolve slots, apply missing-marker policy
  render.go           // RenderHTML (preview w/ data-var spans) — format-neutral entry
  importer.go         // Importer interface
  exporter.go         // Exporter interface
  store.go            // TemplateStore interface (carrier bytes + slot persistence contract)
  errors.go           // sentinel errors (ErrUnknownTemplate, ErrUnboundSlot, …)
  placeholder.go      // placeholderRegex + substitution primitives (THE locked grammar)
  types_wire_test.go  // locks the JSON wire shape consumed by the TS editor
  docx/               // the .docx adapter — first importer + exporter
    importer.go       // DocxImporter: parse .docx -> Carrier + detect/locate slots
    exporter.go       // DocxExporter: (model + carrier + vars) -> .docx bytes  [today's compose+merge]
    ooxml.go          // archive/zip + document.xml manipulation [today's submission_merge/compose internals]
    md_to_ooxml.go    // Markdown -> OOXML runs [today's submission_md walker + the b78a984 fix]
    dotm.go           // ConvertDotmToDocx [today's pre-pass]
  markdown/           // markdown importer (input content; foreign-docx import is a later sibling)
    importer.go       // parse Markdown -> neutral blocks

What lives in docforge vs paliad:

Concern	Home	Why
Neutral model, binding, preview-render	docforge	format-neutral core
VariableResolver interface + ResolverSet	docforge	the seam m wants clean
Placeholder grammar + substitution	docforge	shared invariant (§0.5.1)
.docx importer + exporter, MD→OOXML walker	docforge/docx	first format adapter (ships inside the pkg, like litigationplanner's embedded snapshot)
Markdown importer	docforge/markdown	input-format adapter
Concrete resolvers (project, parties, firm, user, today, deadline, procedural_event)	paliad internal/…	they read paliad's DB/services
TemplateStore impl (Postgres bytea)	paliad	docforge owns no tables
Section / building-block model, submission codes	paliad	consumer-specific composition concepts
HTTP handlers, editor UI, authoring page	paliad	wire + per-consumer UI

3.2 The neutral model + the carrier (resolving "intermediate, but lossless docx")

// A Document is the format-neutral content model importers produce and exporters consume.
type Document struct {
    Blocks []Block
}
type Block struct {
    Kind   BlockKind      // paragraph | heading | list_item | blockquote | section_marker
    Style  string         // logical style key (mapped to a base stylemap on export)
    Spans  []InlineSpan   // text runs (bold/italic/link) + Slots
    // …list level, section key, etc.
}
type InlineSpan struct {
    Text   string
    Bold   bool
    Italic bool
    Link   string
    Slot   *Slot          // non-nil => this span is a variable slot, not literal text
}
type Slot struct {
    Key string             // e.g. "project.case_number" — the placeholder grammar key
}

The carrier keeps the lossless guarantee. The uploaded .docx chrome (letterhead, styles, caption, signature) is never round-tripped through Document. It is held as an opaque Carrier (the original OOXML), and the exporter splices the rendered neutral content into the carrier's named anchors, then substitutes slots — exactly today's compose mechanism, now formalised:

type Carrier struct {
    Format string   // "docx"
    Bytes  []byte   // original upload, preserved byte-for-byte outside anchor regions
    Anchors []Anchor // {{#section:KEY}}…{{/section:KEY}} positions + slot positions
}

So two layers: editable content = Document (neutral, format-pluggable); base chrome = Carrier (opaque, lossless). Foreign-docx import as input content (Q4) does parse into Document and is inherently lossy — flagged as a boundary (§8), distinct from the lossless export of our templates.

3.3 The variable resolver seam (G3)

// VariableResolver answers keys within one dotted namespace.
type VariableResolver interface {
    Namespace() string                         // e.g. "project"
    Resolve(key string) (value string, ok bool)// ok=false => unknown key => missing marker
    Keys() []VariableKey                        // catalogue for the palette + sidebar form
}
type VariableKey struct {
    Key, LabelDE, LabelEN, Group string
}

// ResolverSet composes namespaced resolvers, registers canonical<->legacy aliases,
// and offers BOTH a pull path (Resolve, used during binding) and a push path
// (BuildBag, preserving today's resolved_bag/merged_bag wire).
type ResolverSet struct{ /* … */ }
func (s *ResolverSet) Resolve(key string) (string, bool)
func (s *ResolverSet) BuildBag() map[string]string     // == today's PlaceholderMap
func (s *ResolverSet) Catalogue() []VariableKey         // drives the data-driven form/palette
func (s *ResolverSet) RegisterAlias(canonical, legacy string)

paliad's seven addXxxVars functions become seven resolver types implementing this interface. BuildBag() reproduces today's flat map exactly (alias parity tests pin it). Catalogue() kills the hardcoded VARIABLE_GROUPS/VARIABLE_LABELS in the TS bundle. Resolver model = hybrid (pull-capable interface, push-driven BuildBag default — inventor pick, §4 D-I1).

3.4 Wire contract (Go ↔ TS) — preserved, locked by test

The editor wire stays as-is; types_wire_test.go pins it:

• GET draft → { draft, resolved_bag, merged_bag, preview_html, rule, parties, sections }
• preview HTML carries <span class="draft-var" data-var="<key>">…</span> (built by docforge's RenderHTML, today's emitTextWithDraftVars).
• PATCH draft ← { variables: PlaceholderMap, … } (presence-tracked optional fields).
• export/preview endpoints unchanged.
• New (authoring): POST /api/templates (upload), GET /api/templates/:id (carrier preview + slots), POST /api/templates/:id/slots (place slot), GET /api/docforge/variables (the Catalogue()).

---

§4 Decisions (m's picks, 2026-05-29)

Prose-grill resolutions (core metaphor)

#	Question	m's decision	Note
P1	Cross-language sharing model	Go pkg only; upc-kommentar out of scope for now, "reuse later somehow"	Interfaces sized so an HTTP veneer is addable without rework. No service built.
P2	Intermediate model?	Yes — but lossless for our .docx	→ carrier (opaque OOXML) + neutral Document (editable content). §3.2.
P3	Authoring slot mechanic	(b) click-to-insert	Upload → render → click/select → inject {{…}}.
P4	Input formats	Markdown primary; foreign doc/docx import later	Markdown importer first; foreign-docx import is lossy (§8).
P5	Editor sharing	Build paliad's UI; extract generic UX into a UI package	frontend/src/lib/docforge-editor/.

Structured decisions

#	Decision	m's pick	Rationale / divergence
A1	Authoring UX	WYSIWYG inline	Matches "insert variables into the document". Hardest part — render fidelity + click→run mapping — flagged §7.
A2	Template storage	Postgres bytea (interface-backed)	m leans (1); flagged Supabase Storage as viable. Resolved: behind a TemplateStore interface, bytea impl now, Supabase Storage a one-impl swap later. No schema churn either way.
A3	Versioning of existing drafts	Snapshot at draft-create	Lawyer's in-flight draft won't shift under them; matches today's section-seeding.
A4	Migration strategy	Extract-in-place, then extend	Lowest risk to the recent fixes — they move with their files + tests; behavior identical at each step.
B1	Package name	docforge	—
B2	Schema scope	New generic tables (templates, template_slots, template_versions)	Authoring is domain-neutral; submission_bases (Gitea/section_spec) stays for legacy bases with a converge path.
B3	UI package extraction	Extract now	Authoring reuses it this cycle — earns its keep, not speculative.
B4	Exporter pluggability	Interface now, docx-only impl	Cheap insurance; matches "pluggable for later".

Inventor picks (m delegated — "whatever works best")

#	Pick	Reasoning
I1	VariableResolver = pull-capable interface, push BuildBag() default	Preserves today's flat-map wire while enabling on-demand resolution + the Catalogue() that data-drives the form.
I2	.docx adapter ships inside pkg/docforge/docx	Mirrors litigationplanner shipping its embedded snapshot in-package; keeps the first adapter co-located with the engine it proves.
I3	Carrier-vs-Document split (§3.2)	Only way to satisfy "intermediate model" AND "lossless our .docx" simultaneously.

---

§5 Data model deltas (paliad-side — docforge owns none)

New tables (additive; SQL drafted by the coder, not here):

• paliad.templates — id, slug, name_de/en, kind ('submission' | generic), source_format ('docx'), firm, is_active, created/updated_by, timestamps, current_version_id FK.
• paliad.template_versions — immutable snapshots: id, template_id FK, version int, carrier_blob bytea (the .docx; or storage ref via TemplateStore), created_at, created_by. Editing a template inserts a new version row.
• paliad.template_slots — id, template_version_id FK, slot_key (the variable key, e.g. project.case_number), anchor (position encoding — see flag below), label, order_index. Versioned alongside the carrier.

Snapshot semantics (A3): a draft pins template_version_id. Template edits create a new version; existing drafts keep their pinned version. (Flag for coder: pin template_version_id on the draft vs. copy a template_snapshot jsonb onto the draft — both satisfy A3; the version-table approach is preferred for auditability but the coder picks based on query ergonomics.)

Touched existing tables:

• submission_drafts — add nullable template_version_id for uploaded-template drafts; legacy base_id path preserved (extract-in-place ⇒ no data migration of the 11 existing drafts; §0.5.8 fallback intact).
• submission_bases, submission_sections, submission_building_blocks — unchanged. They remain paliad consumer-specific concepts that map onto docforge's neutral model at render time. submission_bases (Gitea-backed) coexists with the new uploaded-template tables during transition; convergence is a later, separate task.

Slot anchor encoding (flag for coder): how a template_slots.anchor records where in the carrier OOXML the slot sits (run index + offset, vs. a stable sentinel token injected into the carrier at authoring time). The sentinel-token approach is likely simpler and reuses the existing cross-run substitution machinery — resolve in implementation chat.

---

§6 Migration plan (protects working code + the recent fixes)

Principle: extract-in-place (A4). Each step compiles, passes the moved tests, and leaves observable behavior identical. The recent fixes travel with their files:

• The b78a984 underscore fix → pkg/docforge/docx/md_to_ooxml.go (was submission_md.go parseInlineSpans), submission_md_test.go moves alongside.
• placeholderRegex → pkg/docforge/placeholder.go; its tests move.
• data-var / emitTextWithDraftVars → pkg/docforge/render.go (RenderHTML); wire test moves and is pinned in types_wire_test.go.
• Cross-run merge, .dotm→.docx, anchor splicing → pkg/docforge/docx/; tests move.
• Building-block + section model, submission codes, the 7 concrete resolvers stay in internal/ (consumer-specific) — now calling into docforge.

Safety rails per step: (1) go build ./... green; (2) the moved test files green; (3) a golden-export check — generate a known draft before and after the step, assert byte-equal .docx; (4) the live preview HTML for a fixture draft is string-equal (the data-var contract). No step ships until all four hold.

What is explicitly NOT migrated: the 11 pre-Composer drafts (base_id IS NULL) keep the v1 fallback render path; no auto-upgrade (§0.5.8).

---

§7 Slice train

Tracer-bullet vertical slices, each independently shippable. Slices 1–3 are pure behavior-preserving refactors (the risky-to-working-code part, front-loaded under golden checks); 4–7 build the new capability; 8 sets up the future.

1. Extract the docx engine — move MD→OOXML walker, OOXML merge/compose, placeholder grammar, .dotm→.docx into pkg/docforge/{placeholder.go, render.go, docx/}. paliad's submission_* services become thin adapters. Golden-export + preview checks green. Protects b78a984, the regex, the data-var contract.
2. Neutral model + binding — introduce Document/Block/Slot/Carrier + bind.go; refactor the docx exporter to consume the neutral model (sections → blocks → OOXML spliced into carrier). Behavior identical (golden checks).
3. VariableResolver interface — refactor the 7 addXxxVars into resolver types + ResolverSet; BuildBag() reproduces today's map (alias-parity tests pin it); Catalogue() exposed. Frontend form switched to consume Catalogue() (kills hardcoded VARIABLE_GROUPS).
4. Template store + schema — templates/template_versions/template_slots + Postgres-bytea TemplateStore impl. No UI yet. Additive migrations.
5. UI package extraction — pull generic plumbing (debounced autosave, data-var wiring, preview/export round-trip, focus preservation, sticky collapse) into frontend/src/lib/docforge-editor/; submission editor consumes it. Refactor, behavior identical.
6. Authoring page — upload .docx → docforge docx-importer → WYSIWYG render → select text → pick variable from Catalogue() palette → inject slot (writes template_slots + new template_version). Reuses the UI package + docforge importer. (v1: body-paragraph text slots only.)
7. Generation on uploaded templates — generation page picks an uploaded template (template_version_id path) alongside legacy bases; snapshot-at-create; data-bind + manual edit + export via docforge. Legacy base path still works.
8. Markdown importer + exporter-interface finalisation — docforge/markdown importer as input; Exporter interface locked (docx-only impl). Sets up future formats + eventual upc-kommentar reuse.

Flagged follow-ups (post-train, separate tasks): slots in headers/footers/tables; foreign-docx import fidelity; the HTTP veneer + a TS consumer; submission_bases → templates convergence; auto-upgrade of pre-Composer drafts.

---

§8 Out of scope

• Implementation, migration SQL, code. PRD only.
• upc-kommentar as a live consumer — deferred; abstractions sized for it, nothing built.
• An HTTP service veneer — addable later without engine rework; not now.
• Formats beyond .docx — Exporter interface defined (B4), only the docx impl built.
• Lossless import of foreign .docx — our own templates export losslessly via the carrier; importing an arbitrary third-party Word doc as input content is best-effort and inherently lossy. Distinct guarantee.
• Multi-user concurrent editing of one draft.
• Re-proposing the current submission_*.go shape — the point is to extract + clean it.
• Slots outside body paragraphs (headers/footers/tables/text-boxes) in authoring v1.

---

Appendix — open flags for the coder (resolve in implementation chat)

1. Slot anchor encoding — run-index+offset vs. injected sentinel token (§5). Lean sentinel.
2. Snapshot mechanism — pinned template_version_id vs. template_snapshot jsonb on the draft (§5). Lean version-pin.
3. Authoring render fidelity — reuse the existing lossy docXMLToHTML preview for the WYSIWYG surface, or invest in higher fidelity. Lean reuse for v1, accept that complex layouts render approximately while slots still anchor correctly.
4. Storage backend — Postgres bytea now; Supabase Storage is a clean TemplateStore swap if template volume/size grows.