mDMS/infra/mdms-mover/README.md

# mdms-mover — age-gated inbox → toprocess promoter + blank-page stripper

Two jobs in one user-systemd timer:

1. **Stability gate** (otto#438): solves the chunk-write race between the
   Canon MB5100 (SMB scans land in `/mnt/mdms/inbox/` in pieces) and
   Paperless (polls `/mnt/mdms/toprocess/` every 60s and consumes
   anything it sees). A file is only promoted when **both**:
   - `mtime > 3 minutes` ago, and
   - file size is unchanged since the previous run.
2. **Blank-page strip** (mDMS#2): duplex scans through patch-T separators
   leave a blank backside (the unprinted reverse of the separator sheet)
   at the front of every subsequent document. PDF files are passed
   through `strip_blank_pages.py` before promotion. Pages with no
   embedded text AND >97% near-white pixels are dropped.

## Layout on mDock

```
/home/m/mdms-mover/mover.sh                  # script, deployed copy
/home/m/mdms-mover/strip_blank_pages.py      # blank-page detector
/home/m/.config/systemd/user/mdms-mover.service  # oneshot service
/home/m/.config/systemd/user/mdms-mover.timer    # OnUnitActiveSec=1min
/home/m/.local/state/mdms-mover/state.tsv    # last-seen size per file
/home/m/.local/bin/uv                        # uv runner for the strip script
```

Runs as user `m` under user-systemd. mDock has `Linger=yes` for user
`m`, so the timer keeps firing across reboots and logout sessions.

## Why systemd, not cron

The original spec (otto#438) called for `/etc/cron.d/mdms-mover`. mDock
runs Ubuntu 24.04 server which ships with systemd-timers and no `cron`
package. Installing cron only to honour the spec wording would add a
package we don't otherwise need; a user-systemd timer is the canonical
Ubuntu 24.04 approach and gives better observability
(`systemctl --user status mdms-mover.timer`, `journalctl --user -u mdms-mover`).

User-mode (not system-mode) keeps the entire install in `m`'s home — no
sudo at deploy or maintenance time, no `/var/lib/...` directories to
chown, the service can read/write the NFS mount because `m` owns it.

## Configuration

```
| var                    | default                                       | meaning                                            |
|------------------------|-----------------------------------------------|----------------------------------------------------|
| MDMS_INBOX             | /mnt/mdms/inbox                               | source — scanner SMB target                        |
| MDMS_TOPROCESS         | /mnt/mdms/toprocess                           | destination — Paperless consume                    |
| MDMS_STATE             | $HOME/.local/state/mdms-mover/state.tsv       | per-file size memory                               |
| MDMS_MIN_AGE_MIN       | 3                                             | minimum mtime age in minutes                       |
| MDMS_STRIP_BLANK       | true                                          | run blank-page strip on PDFs (set to "false" to disable) |
| MDMS_STRIP_SCRIPT      | <mover dir>/strip_blank_pages.py              | path override for the strip script                 |
| MDMS_BLANK_THRESHOLD   | 0.97                                          | near-white pixel ratio to call a page blank (read by strip script) |
| MDMS_BLANK_NEAR_WHITE  | 240                                           | grayscale cutoff (0-255) for "near white" pixels (read by strip script) |
| MDMS_BLANK_DPI         | 50                                            | thumbnail render DPI (read by strip script)        |
```

To override at runtime, drop into
`~/.config/systemd/user/mdms-mover.service.d/override.conf`:

```ini
[Service]
Environment=MDMS_MIN_AGE_MIN=5
Environment=MDMS_BLANK_THRESHOLD=0.99
```

then `systemctl --user daemon-reload && systemctl --user restart mdms-mover.timer`.

## Blank-page detection — what gets dropped

A page is dropped iff BOTH:

1. embedded text is empty / whitespace-only (image-only scans always
   pass this — they have no embedded text), AND
2. the rendered thumbnail is ≥ `MDMS_BLANK_THRESHOLD` near-white pixels
   (0.97 by default → >97% of pixels brighter than grayscale 240).

The threshold is conservative on purpose: a false-negative (keeping a
blank page we should have dropped) is recoverable via Paperless's UI; a
false-positive (dropping a real page) silently loses data. If real
pages get dropped in practice, **raise** `MDMS_BLANK_THRESHOLD` toward
0.99 — that makes the strip step pickier and keeps more pages.

Edge cases handled inside `strip_blank_pages.py`:

- **1-page input:** strip is skipped entirely (single-page docs never
  have separator-backside artefacts).
- **All pages would drop:** the script exits with code `2` and writes no
  output. The mover keeps the file in the inbox and logs
  `WARNING: <name> appears all-blank, kept in inbox`. m can inspect via
  `journalctl --user -u mdms-mover`.
- **strip_blank_pages.py errors out:** mover falls back to a plain `mv`
  (unstripped) so a transient problem in the detector never blocks a
  scan from reaching Paperless.

The script is a uv-inline-deps single file (PyMuPDF for both rendering
and text extraction — one wheel, no `poppler-utils` apt install on
mdock). Mirrors the pattern from `infra/paperless/generate_separator.py`.

## Deploy / sync

The live files on mDock must match this directory byte-for-byte (md5,
same convention as `infra/samba-canon/`).

```bash
ssh mdock 'mkdir -p ~/mdms-mover ~/.config/systemd/user ~/.local/state/mdms-mover ~/.local/bin'

# uv binary (single static binary, user-space — no apt, no sudo)
rsync -av ~/.local/bin/uv mdock:/home/m/.local/bin/uv

# mover + strip script
scp infra/mdms-mover/mover.sh             mdock:/home/m/mdms-mover/mover.sh
scp infra/mdms-mover/strip_blank_pages.py mdock:/home/m/mdms-mover/strip_blank_pages.py
scp infra/mdms-mover/mdms-mover.service   mdock:/home/m/.config/systemd/user/
scp infra/mdms-mover/mdms-mover.timer     mdock:/home/m/.config/systemd/user/

ssh mdock 'chmod +x ~/mdms-mover/mover.sh ~/mdms-mover/strip_blank_pages.py && \
           systemctl --user daemon-reload && \
           systemctl --user enable --now mdms-mover.timer'
```

The first time the strip script runs, `uv` downloads python + PyMuPDF
into `~/.cache/uv/` (~30 MB). Subsequent runs reuse the cache.

## Verify

```bash
ssh mdock 'systemctl --user list-timers mdms-mover.timer'
ssh mdock 'journalctl --user -u mdms-mover -n 20 --no-pager'
ssh mdock 'cat ~/.local/state/mdms-mover/state.tsv'
ssh mdock 'journalctl -t mdms-mover -n 20 --no-pager'
```

## Emergency disable

Stop the timer entirely:

```bash
ssh mdock 'systemctl --user stop mdms-mover.timer && \
           systemctl --user disable mdms-mover.timer'
```

Or just disable the strip step while keeping the stability gate:

```bash
mkdir -p ~/.config/systemd/user/mdms-mover.service.d
cat > ~/.config/systemd/user/mdms-mover.service.d/override.conf <<EOF
[Service]
Environment=MDMS_STRIP_BLANK=false
EOF
systemctl --user daemon-reload
```

Re-enable the timer with `systemctl --user enable --now mdms-mover.timer`.

If you need to drain the inbox manually while disabled, files older
than a few minutes are safe to `mv` into `toprocess/` by hand —
Paperless will pick them up on its next poll.

## Logs

Service logs land in the user journal under unit `mdms-mover`, and
moved-file events also go through `logger -t mdms-mover` so they appear
under that tag in the system journal too:

```bash
ssh mdock 'journalctl --user -u mdms-mover -f'   # service execution
ssh mdock 'journalctl -t mdms-mover -f'          # moved-file events
```

## Refs

- mDMS#2 — blank-page strip (this README)
- otto#438 — original scheduler / staging-folder design
- otto#429 — original Paperless pipeline setup
- otto#431 — samba-canon bridge container (upstream of this mover)
- `docs/strategy.md` — overall mDMS dataset layout
- `infra/paperless/generate_separator.py` — sibling uv-inline-deps script