m/paliad
1
0
Fork 0
Code Issues 152 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

prod 500s after B1+B2 deploy — /api/builder/scenarios + /api/events?type=deadline failing #154

New Issue
Open
opened 2026-05-27 22:36:32 +00:00 by mAi · 2 comments
mAi commented 2026-05-27 22:36:32 +00:00
Collaborator
Copy Link

Regression on main 88c03e9 (post-B1+B2 deploy)

m reported live on paliad.de right after the B1+B2 deploy:

Failed to load resource: the server responded with a status of 500 ()
procedures.js:101 Uncaught (in promise) TypeError:
    Cannot read properties of null (reading 'filter')
        at D (procedures.js:101:761)
        at a (procedures.js:101:6050)
        at async M8 (procedures.js:101:7515)

Followed by:

GET https://paliad.de/api/events?type=deadline&status=pending  500
GET https://paliad.de/api/events?type=deadline&status=overdue  500

m's words: "It does not really work, cant set anything... you should fix it with a tester. Or is it not done yet?!"

What broke

  • Builder UI unusable: /tools/procedures loads the chrome (page header / scenario picker / matter picker / search / overview / event / from-matter / side panel), but interaction dies on the null.filter JS error. Likely first fetch (GET /api/builder/scenarios?status=active) returns 500 → fetchJSON swallows to null → downstream state.active.proceedings.filter(...) throws.
  • Dashboard deadline endpoints (existing feature) also 500: /api/events?type=deadline&status=pending and ?status=overdue. These are NOT new code from B0/B1/B2 — they were working before tonight's deploy. This points at mig 157 (B0) breaking something at the schema / RLS layer, or a service-init regression from the new wirings.

Train context

Tonight's session shipped (in order):

main SHA What
1844df3 B0 Scenario DB foundation (newton) — mig 157 + scenario_builder service + minimal API
1b4b2e4 submission-md placeholder underscores fix (head)
0c85702 Engine semantics — suppress optional + respect trigger_event_id (pasteur)
6c1d8cc B1 commit (galileo) — Litigation Builder shell + cold-open
46dc4ec B2 commit (galileo) — multi-triplet + spawn + per-event state
88c03e9 Current main (B1+B2 merged)

The /api/events regression is the loud signal — that endpoint isn't touched by any of the new code. mig 157's paliad.projects.origin_scenario_id column / RLS changes are the most likely culprit, but needs log investigation.

Assigned fixer

brunel running on t-paliad-344 — investigation order: pull Dokploy logs first (look at the SQL panic / error message for the /api/events?type=deadline&status=pending path), then verify mig 157 applied cleanly, then service init, then frontend null-guards. Verification via mai-tester (Playwright) before commit. Do NOT revert main without head approval.

Acceptance

  • /api/events?type=deadline&status=pending returns 200 with the deadline list (m's dashboard works)
  • /api/events?type=deadline&status=overdue returns 200
  • /api/builder/scenarios?status=active returns 200 with an array (possibly empty)
  • /tools/procedures loads without JS errors and the New Scenario CTA works through to a saved scratch scenario
## Regression on main `88c03e9` (post-B1+B2 deploy) m reported live on paliad.de right after the B1+B2 deploy: ``` Failed to load resource: the server responded with a status of 500 () procedures.js:101 Uncaught (in promise) TypeError: Cannot read properties of null (reading 'filter') at D (procedures.js:101:761) at a (procedures.js:101:6050) at async M8 (procedures.js:101:7515) ``` Followed by: ``` GET https://paliad.de/api/events?type=deadline&status=pending 500 GET https://paliad.de/api/events?type=deadline&status=overdue 500 ``` m's words: "It does not really work, cant set anything... you should fix it with a tester. Or is it not done yet?!" ## What broke - **Builder UI unusable**: `/tools/procedures` loads the chrome (page header / scenario picker / matter picker / search / overview / event / from-matter / side panel), but interaction dies on the `null.filter` JS error. Likely first fetch (`GET /api/builder/scenarios?status=active`) returns 500 → `fetchJSON` swallows to `null` → downstream `state.active.proceedings.filter(...)` throws. - **Dashboard deadline endpoints (existing feature) also 500**: `/api/events?type=deadline&status=pending` and `?status=overdue`. These are NOT new code from B0/B1/B2 — they were working before tonight's deploy. This points at mig 157 (B0) breaking something at the schema / RLS layer, or a service-init regression from the new wirings. ## Train context Tonight's session shipped (in order): | main SHA | What | |---|---| | `1844df3` | B0 Scenario DB foundation (newton) — mig 157 + scenario_builder service + minimal API | | `1b4b2e4` | submission-md placeholder underscores fix (head) | | `0c85702` | Engine semantics — suppress optional + respect trigger_event_id (pasteur) | | `6c1d8cc` | B1 commit (galileo) — Litigation Builder shell + cold-open | | `46dc4ec` | B2 commit (galileo) — multi-triplet + spawn + per-event state | | `88c03e9` | Current main (B1+B2 merged) | The `/api/events` regression is the loud signal — that endpoint isn't touched by any of the new code. mig 157's `paliad.projects.origin_scenario_id` column / RLS changes are the most likely culprit, but needs log investigation. ## Assigned fixer brunel running on t-paliad-344 — investigation order: pull Dokploy logs first (look at the SQL panic / error message for the `/api/events?type=deadline&status=pending` path), then verify mig 157 applied cleanly, then service init, then frontend null-guards. Verification via mai-tester (Playwright) before commit. Do NOT revert main without head approval. ## Acceptance - `/api/events?type=deadline&status=pending` returns 200 with the deadline list (m's dashboard works) - `/api/events?type=deadline&status=overdue` returns 200 - `/api/builder/scenarios?status=active` returns 200 with an array (possibly empty) - `/tools/procedures` loads without JS errors and the New Scenario CTA works through to a saved scratch scenario
mAi self-assigned this 2026-05-27 22:36:32 +00:00
mAi commented 2026-07-29 14:03:24 +00:00
Author
Collaborator
Copy Link

Measured 2026-07-29: does not reproduce at migration head 211

Not closing this — recording what was measured, because two months of silence is not the same as an answer.

Why it sat

All three endpoints are behind the auth gate, so an anonymous probe only ever returns 401. Nobody could establish whether the 500s still happened. cmd/server/http_smoke_enforcing_test.go (merged today) boots the real server binary against a prod-shape harness and mints a signature-valid session, which is what made this answerable.

What was run

Real binary, prod-shape harness at applied_migrations head 211, NOBYPASSRLS runtime role, prod's ownership map, both a global_admin and a staffed-member session:

endpoint status body
/api/events?type=deadline&status=pending 200 the fixture's deadline, non-empty
/api/events?type=deadline&status=overdue 200 []
/api/builder/scenarios?status=active 200 []

No 500 on any of them, for either session.

What is genuinely covered, and what is not

Covered. Schema-level and RLS-level breakage on these three paths. The pending path returns a real row, so it is not answering over an empty table. overdue and the scenario list return empty lists — the SQL still executes, and a broken statement 500s whether or not it matches, so the does it 500 question is answered. It does not prove those filters select the right rows.

Not covered. A failure that needs production's data — a NULL where the fixture has none, or a plan-dependent error at scale. The fixture carries 1 deadline, 1 appointment, 0 scenarios.

Not covered, and this is the one that matters here. The original deploy was mid-crash-loop (this issue names mig 157, and that era carried its own hotfixes). A partially-applied schema is not reproducible on a harness that applies every migration cleanly. If that was the cause, "it works here" is the expected result and says nothing.

No fix commit is identifiable

internal/services/event_service.go changed only through the db.Querier sweep and the ActingUserDB wiring. Nothing between the report and now says it fixed this. So the honest verdict is split:

  • resolved for the schema/RLS class — mig 195/197 and the scenario_* ownership reassignment (2026-07-27) are real changes to exactly that surface;
  • unreproducible by construction for the partial-migration class.

The client half of the report

The trace named procedures.js:101 and Cannot read properties of null (reading 'filter'). That file is now 15 lines and holds no fetch and no .filter — the builder moved to client/builder.ts, whose list callers normalise with Array.isArray(out) ? out : []. Server-side, both the list and the deep scenario endpoints initialise their slices, so they answer [] and never null. The specific crash in the trace cannot occur in the current code.

What changed

The three endpoints are now a permanent step in the HTTP smoke, so they can never again be unanswerable: 4c97c35. It asserts the status, that the body is not JSON null (a 200 alone does not rule out the null.filter shape), and a row count on the pending path so it cannot pass on an endpoint that returns [] for everything.

What would settle the remaining difference

One authenticated GET of those three URLs against production. That needs a session credential, which is m's or the head's to provide — it is the only thing that distinguishes fixed from not reproducible here.

## Measured 2026-07-29: does not reproduce at migration head 211 **Not closing this** — recording what was measured, because two months of silence is not the same as an answer. ### Why it sat All three endpoints are behind the auth gate, so an anonymous probe only ever returns 401. Nobody could establish whether the 500s still happened. `cmd/server/http_smoke_enforcing_test.go` (merged today) boots the real server binary against a prod-shape harness and mints a signature-valid session, which is what made this answerable. ### What was run Real binary, prod-shape harness at `applied_migrations` head **211**, NOBYPASSRLS runtime role, prod's ownership map, both a `global_admin` and a staffed-member session: | endpoint | status | body | |---|---|---| | `/api/events?type=deadline&status=pending` | **200** | the fixture's deadline, non-empty | | `/api/events?type=deadline&status=overdue` | **200** | `[]` | | `/api/builder/scenarios?status=active` | **200** | `[]` | No 500 on any of them, for either session. ### What is genuinely covered, and what is not **Covered.** Schema-level and RLS-level breakage on these three paths. The `pending` path returns a real row, so it is not answering over an empty table. `overdue` and the scenario list return empty lists — the SQL still executes, and a broken statement 500s whether or not it matches, so the *does it 500* question is answered. It does **not** prove those filters select the right rows. **Not covered.** A failure that needs production's *data* — a NULL where the fixture has none, or a plan-dependent error at scale. The fixture carries 1 deadline, 1 appointment, 0 scenarios. **Not covered, and this is the one that matters here.** The original deploy was mid-crash-loop (this issue names mig 157, and that era carried its own hotfixes). **A partially-applied schema is not reproducible on a harness that applies every migration cleanly.** If that was the cause, "it works here" is the expected result and says nothing. ### No fix commit is identifiable `internal/services/event_service.go` changed only through the `db.Querier` sweep and the `ActingUserDB` wiring. Nothing between the report and now says it fixed this. So the honest verdict is split: - **resolved** for the schema/RLS class — mig 195/197 and the `scenario_*` ownership reassignment (2026-07-27) are real changes to exactly that surface; - **unreproducible by construction** for the partial-migration class. ### The client half of the report The trace named `procedures.js:101` and `Cannot read properties of null (reading 'filter')`. That file is now 15 lines and holds no fetch and no `.filter` — the builder moved to `client/builder.ts`, whose list callers normalise with `Array.isArray(out) ? out : []`. Server-side, both the list and the deep scenario endpoints initialise their slices, so they answer `[]` and never `null`. The specific crash in the trace cannot occur in the current code. ### What changed The three endpoints are now a permanent step in the HTTP smoke, so they can never again be unanswerable: [4c97c35](https://mgit.msbls.de/m/paliad/commit/4c97c35). It asserts the status, that the body is not JSON `null` (a 200 alone does not rule out the `null.filter` shape), and a row count on the `pending` path so it cannot pass on an endpoint that returns `[]` for everything. ### What would settle the remaining difference One authenticated GET of those three URLs against production. That needs a session credential, which is m's or the head's to provide — it is the only thing that distinguishes *fixed* from *not reproducible here*.
mAi commented 2026-07-31 09:47:45 +00:00
Author
Collaborator
Copy Link

Re-measured 2026-07-31. Server half still does not reproduce, now at prod's exact applied set. The client half has one real gap — and prod cannot be asked at all.

Continuing the 2026-07-29 comment rather than repeating it. That comment measured the three endpoints at the time and named two things it could not reach: production-side evidence, and an authenticated GET against prod. I went after the first. It cannot be obtained, and the reason is worth more than the count would have been.

1. Server half — re-run against prod's applied set exactly

TestHTTPSmoke_EnforcingRole on main c107154, real server binary, prod-shape harness, NOBYPASSRLS runtime role.

The harness finished at paliad.applied_migrations max 216, 211 rows — byte-identical to production, which I read as max 216, 211 rows in the same session. Not "a recent head": the same one. (The five-number gap is the m/paliad#44 design, not a fault.)

--- PASS: TestHTTPSmoke_EnforcingRole/issue154_endpoints_do_not_500 (0.02s)
--- PASS: TestHTTPSmoke_EnforcingRole (1.48s)
ok   mgit.msbls.de/m/paliad/cmd/server  7.911s

That subtest asserts, per URL: status 200, body is not JSON null, and — on status=pending — a row count taken from the fixture so it cannot pass on an endpoint that answers [] for everything. Status-only would have called a 200 null green; it does not.

Still no 500 on any of the three. The 2026-07-29 caveat stands unchanged and I am not weakening it: a deploy that was mid-crash-loop on a partially applied schema is not reproducible on a harness that applies every migration cleanly. For that class, "it works here" is the expected result and proves nothing.

2. Production-side evidence — the count cannot be produced, by construction

The ask was a count of 500s on that route in prod. There is no way to get one. Two independent reasons, both worth knowing:

paliad emits no HTTP request log. The running container's entire log is 14 lines, all from boot:

2026/07/31 08:43:53 applying database migrations…
2026/07/31 08:43:53 database migrations applied
2026/07/31 08:43:53 RLS mode: permissive — connected as "postgres" (bypassrls=true, superuser=false)
2026/07/31 08:43:53 paliad server starting on :8080

There is no logging middleware on the mux. A 500 in production leaves no trace at all unless a specific handler logs it, and handleListEvents does not — it routes errors through writeServiceError and writes nothing.

Traefik's access log cannot cover it either. It exists (/etc/dokploy/traefik/dynamic/access.log, JSON), but:

  • it holds 2026-07-30T23:34Z → now — about ten hours. It does not reach 2026-05-27 and never will.
  • its config carries filters: { retryAttempts: true, minDuration: 10ms }, so a request that is fast and not retried is not written. A quick 500 is exactly that shape.
  • in that whole window, paliad.de produced 7 requests — six /assets/* and one /logout. Zero to /api/events. Zero 5xx on any paliad host.

So the honest reading of "no 500s in the prod log" is not "no 500s happened". It is "prod cannot answer this question". Anyone citing the log as evidence of health here would be citing an absence they never had the ability to observe.

3. Client half — audited independently, and one caller has no guard

Three client modules fetch /api/events. They do not behave the same way.

caller page guarded?
client/builder.ts /tools/procedures yes
client/projects-detail.ts matter detail yes — (await resp.json()) ?? []
client/events.ts the deadlines/events page no

The procedures client is fixed. procedures.ts is now 15 lines and holds no fetch and no .filter; the work moved to builder.ts, whose list fetches end in Array.isArray(out) ? out : [] (builder.ts:221, :226, and two more at :251/:256) and whose loadScenario normalises all four deep arrays before anything reads them (builder.ts:1268-1271). The trace in this issue — procedures.js:101, null.filter — cannot occur in the current code.

client/events.ts is the caller of the two URLs m actually reported, and it has no null guard.

// client/events.ts:423
const data: EventListItem[] = await resp.json();
allItems = data;
loadedOK = true;
render();
...
// client/events.ts:456
if (allItems.length === 0) {

A 200 carrying JSON null sets allItems = null and throws Cannot read properties of null (reading 'length') — the same class as the reported crash, one property along. Its 500 handling is fine: !resp.ok shows the unavailable banner and returns, no crash.

Is it reachable today? No — and only because the server happens not to do it. handleListEvents writes whatever EventService.ListVisibleForUser returns, and that function opens with out := make([]EventListItem, 0, 64) (event_service.go:137) and returns out, nil (:215). Non-nil on every success path, so the wire value is [], never null.

That makes it latent, not live — and the whole guarantee sits in one make call in a service, with nothing on the client and nothing pinning the coupling. Any future handler on that route that returns a nil slice turns this into m's blank page again. builder.ts carries a comment saying it normalises client-side precisely so a server regression cannot crash it; events.ts does not, and it is the page m was on.

4. Verdict

  • Endpoints: do not 500 and do not return null, at prod's exact applied set. Unchanged from 2026-07-29, now with the migration state matched rather than approximated. One correction to that comment: the issue154 subtest exercises the admin session only (http_smoke_enforcing_test.go:562-597). The member session appears in step 4, against /api/deadlines, a different route. So "for either session" is not what is covered on these three URLs.
  • Original client crash: cannot occur — that code is gone.
  • New finding: client/events.ts dereferences a fetch result with no null guard, unlike the other two callers of the same endpoint. Latent today, and the one thing here that is a specific unbuilt fix rather than a measurement.
  • Prod: unobservable. Not healthy, not unhealthy — unobservable. A request log would change that, and its absence is a bigger gap than this issue.

Two things would settle what is left, neither of which is mine to do: one authenticated GET of the three URLs against production, and a decision on whether paliad should log requests at all.

Not closing. Nothing written to prod, no code changed.

paliad/ritchie, t-paliad-issue-154-verify-500s.

## Re-measured 2026-07-31. Server half still does not reproduce, now at prod's exact applied set. The client half has one real gap — and prod cannot be asked at all. Continuing the 2026-07-29 comment rather than repeating it. That comment measured the three endpoints at the time and named two things it could not reach: production-side evidence, and an authenticated GET against prod. I went after the first. **It cannot be obtained, and the reason is worth more than the count would have been.** --- ### 1. Server half — re-run against prod's applied set exactly `TestHTTPSmoke_EnforcingRole` on `main` `c107154`, real server binary, prod-shape harness, NOBYPASSRLS runtime role. The harness finished at `paliad.applied_migrations` **max 216, 211 rows** — byte-identical to production, which I read as **max 216, 211 rows** in the same session. Not "a recent head": the same one. (The five-number gap is the m/paliad#44 design, not a fault.) ``` --- PASS: TestHTTPSmoke_EnforcingRole/issue154_endpoints_do_not_500 (0.02s) --- PASS: TestHTTPSmoke_EnforcingRole (1.48s) ok mgit.msbls.de/m/paliad/cmd/server 7.911s ``` That subtest asserts, per URL: status 200, body is not JSON `null`, and — on `status=pending` — a row count taken from the fixture so it cannot pass on an endpoint that answers `[]` for everything. Status-only would have called a `200 null` green; it does not. Still no 500 on any of the three. The 2026-07-29 caveat stands unchanged and I am not weakening it: **a deploy that was mid-crash-loop on a partially applied schema is not reproducible on a harness that applies every migration cleanly.** For that class, "it works here" is the expected result and proves nothing. ### 2. Production-side evidence — **the count cannot be produced, by construction** The ask was a count of 500s on that route in prod. There is no way to get one. Two independent reasons, both worth knowing: **paliad emits no HTTP request log.** The running container's entire log is **14 lines**, all from boot: ``` 2026/07/31 08:43:53 applying database migrations… 2026/07/31 08:43:53 database migrations applied 2026/07/31 08:43:53 RLS mode: permissive — connected as "postgres" (bypassrls=true, superuser=false) 2026/07/31 08:43:53 paliad server starting on :8080 ``` There is no logging middleware on the mux. A 500 in production leaves **no trace at all** unless a specific handler logs it, and `handleListEvents` does not — it routes errors through `writeServiceError` and writes nothing. **Traefik's access log cannot cover it either.** It exists (`/etc/dokploy/traefik/dynamic/access.log`, JSON), but: - it holds **2026-07-30T23:34Z → now** — about ten hours. It does not reach 2026-05-27 and never will. - its config carries `filters: { retryAttempts: true, minDuration: 10ms }`, so a request that is fast and not retried is **not written**. A quick 500 is exactly that shape. - in that whole window, `paliad.de` produced **7 requests** — six `/assets/*` and one `/logout`. **Zero** to `/api/events`. **Zero** 5xx on any paliad host. So the honest reading of "no 500s in the prod log" is **not** "no 500s happened". It is **"prod cannot answer this question"**. Anyone citing the log as evidence of health here would be citing an absence they never had the ability to observe. ### 3. Client half — audited independently, and one caller has no guard Three client modules fetch `/api/events`. They do not behave the same way. | caller | page | guarded? | |---|---|---| | `client/builder.ts` | `/tools/procedures` | **yes** | | `client/projects-detail.ts` | matter detail | **yes** — `(await resp.json()) ?? []` | | `client/events.ts` | the deadlines/events page | **no** | **The procedures client is fixed.** `procedures.ts` is now 15 lines and holds no fetch and no `.filter`; the work moved to `builder.ts`, whose list fetches end in `Array.isArray(out) ? out : []` (`builder.ts:221`, `:226`, and two more at `:251`/`:256`) and whose `loadScenario` normalises all four deep arrays before anything reads them (`builder.ts:1268-1271`). The trace in this issue — `procedures.js:101`, `null.filter` — cannot occur in the current code. **`client/events.ts` is the caller of the two URLs m actually reported, and it has no null guard.** ```ts // client/events.ts:423 const data: EventListItem[] = await resp.json(); allItems = data; loadedOK = true; render(); ... // client/events.ts:456 if (allItems.length === 0) { ``` A 200 carrying JSON `null` sets `allItems = null` and throws `Cannot read properties of null (reading 'length')` — the same class as the reported crash, one property along. Its 500 handling is fine: `!resp.ok` shows the unavailable banner and returns, no crash. **Is it reachable today? No — and only because the server happens not to do it.** `handleListEvents` writes whatever `EventService.ListVisibleForUser` returns, and that function opens with `out := make([]EventListItem, 0, 64)` (`event_service.go:137`) and returns `out, nil` (`:215`). Non-nil on every success path, so the wire value is `[]`, never `null`. That makes it **latent, not live** — and the whole guarantee sits in one `make` call in a service, with nothing on the client and nothing pinning the coupling. Any future handler on that route that returns a nil slice turns this into m's blank page again. `builder.ts` carries a comment saying it normalises client-side precisely so a server regression cannot crash it; `events.ts` does not, and it is the page m was on. ### 4. Verdict - **Endpoints:** do not 500 and do not return `null`, at prod's exact applied set. Unchanged from 2026-07-29, now with the migration state matched rather than approximated. One correction to that comment: the `issue154` subtest exercises the **admin** session only (`http_smoke_enforcing_test.go:562-597`). The `member` session appears in step 4, against `/api/deadlines`, a different route. So "for either session" is not what is covered on these three URLs. - **Original client crash:** cannot occur — that code is gone. - **New finding:** `client/events.ts` dereferences a fetch result with no null guard, unlike the other two callers of the same endpoint. Latent today, and the one thing here that is a specific unbuilt fix rather than a measurement. - **Prod:** unobservable. Not healthy, not unhealthy — unobservable. A request log would change that, and its absence is a bigger gap than this issue. Two things would settle what is left, neither of which is mine to do: one authenticated GET of the three URLs against production, and a decision on whether paliad should log requests at all. Not closing. Nothing written to prod, no code changed. *paliad/ritchie, t-paliad-issue-154-verify-500s.*
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
deferred
Parked — not pursuing now, may un-park later
done
Work completed by an agent; awaiting m to close
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
mAi m
No Assignees mAi
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: m/paliad#154
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?