Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7164bff8e3 | ||
|
|
793cefa5ac | ||
|
|
a84b375f5d | ||
|
|
52358c5bf9 | ||
|
|
c1e68c27c5 | ||
|
|
d4edff9059 | ||
|
|
80febc9993 | ||
|
|
c500c2fae7 | ||
|
|
0a1b4ec173 | ||
|
|
b8d8f028c9 | ||
|
|
cd56367295 | ||
|
|
40a2d2844f | ||
|
|
f79968f336 | ||
|
|
df43e09eca | ||
|
|
cb0f335e8a | ||
|
|
38d06d3104 | ||
|
|
fe33378333 | ||
|
|
5944cd106b | ||
|
|
fd81bf4cf8 | ||
|
|
d23318cfa6 | ||
|
|
142544c9af | ||
|
|
99a5f2a1bc | ||
|
|
863cad22bd | ||
|
|
c06657c8f9 | ||
|
|
532a80211f | ||
|
|
d06a367b87 | ||
|
|
ce6abe6e23 | ||
|
|
ece3562dc9 | ||
|
|
84d4455682 | ||
|
|
b9a7f2083b | ||
|
|
b36f3e0875 | ||
|
|
e4379184b5 | ||
|
|
5ccb56da2d | ||
|
|
16f589448d | ||
|
|
75ea3c9a1f | ||
|
|
7529190587 | ||
|
|
b9f9f84907 | ||
|
|
68ec176719 | ||
|
|
d17c3f4fec | ||
|
|
783a51950f | ||
|
|
51d3d27fd3 | ||
|
|
ccfd22b2a9 | ||
|
|
a5ee329c36 | ||
|
|
2091874050 | ||
|
|
22e8a82fa1 | ||
|
|
01751bf2f0 | ||
|
|
119eb1f5ad | ||
|
|
d3638d786d | ||
|
|
e5a258a648 | ||
|
|
c64369e56c | ||
|
|
f29500c138 | ||
|
|
9d11918bd8 | ||
|
|
8d2bb14f99 | ||
|
|
b53aaa7d03 | ||
|
|
3fa0f30d43 | ||
|
|
9ea848570f | ||
|
|
d6061eb0aa | ||
|
|
1759e0bdd5 | ||
|
|
514b020f74 | ||
|
|
18b0b19a67 | ||
|
|
a8bda9240c | ||
|
|
0692058602 | ||
|
|
8150d16b6f | ||
|
|
c8f238ae60 | ||
|
|
8df49ac053 | ||
|
|
328cd8c664 | ||
|
|
f703a67817 | ||
|
|
105af51100 | ||
|
|
57bf17d348 | ||
|
|
11df30b914 | ||
|
|
43162c40ad | ||
|
|
be57f11747 | ||
|
|
8ba8f69252 | ||
|
|
be97268a7c | ||
|
|
b95b7c53c7 | ||
|
|
a9641bf583 | ||
|
|
9e7661eedd | ||
|
|
ba9eeddf59 | ||
|
|
a49b2275e7 | ||
|
|
a660aed964 | ||
|
|
1077c4f134 | ||
|
|
e468ed5e65 | ||
|
|
bad9e9ac61 | ||
|
|
4da5b4b7eb | ||
|
|
26d835eda6 | ||
|
|
cfd6d19886 | ||
|
|
7bcaeb69c5 | ||
|
|
c168e5a658 | ||
|
|
313f614ec2 | ||
|
|
b0109e174b | ||
|
|
38180550a3 |
@@ -0,0 +1,97 @@
|
||||
---
|
||||
name: feature-walkthrough
|
||||
description: >-
|
||||
Explain the full logic and process of the current branch end-to-end so someone
|
||||
with no prior knowledge of the task can understand, review, and reproduce it.
|
||||
Scopes the change from the branch diff, traces the flow across every layer it
|
||||
touches (frontend tool/hook/component, Java controller/service/endpoint, Python
|
||||
engine, config, i18n, tests), and produces a self-contained walkthrough document
|
||||
with Mermaid diagrams (sequence/flow/architecture), annotated file map with
|
||||
clickable references, before/after behavior, screenshots where a UI is involved,
|
||||
a "try it locally" section, and edge cases/risks. Use when asked for a feature or
|
||||
branch walkthrough, "explain what this branch does", a design/logic writeup, PR
|
||||
reviewer onboarding, or a hand-off doc. Pass --html to also emit a rendered HTML
|
||||
version; --no-screens to skip screenshots.
|
||||
argument-hint: "[branch-or-area] [--html] [--no-screens]"
|
||||
allowed-tools: Read, Write, Edit, Glob, Grep, Bash
|
||||
---
|
||||
|
||||
# Feature / Branch Walkthrough
|
||||
|
||||
Turn the current branch into a walkthrough a newcomer can follow. Audience:
|
||||
**someone who has never seen this task**. Explain the *why*, the *flow*, and *how to
|
||||
try it* - not just a diff summary.
|
||||
|
||||
`$ARGUMENTS` may name a branch or area to focus on; default is the current branch
|
||||
vs `main`. Flags: `--html` (also emit a rendered HTML twin), `--no-screens`.
|
||||
|
||||
## Process
|
||||
|
||||
### 1. Scope the change
|
||||
- `git log --oneline main..HEAD` and `git diff --stat main...HEAD` for the shape.
|
||||
- Read the PR description / commit messages for stated intent. Do **not** invent
|
||||
history or motivation that isn't evidenced (state current behavior in present tense).
|
||||
- Classify touched files by layer:
|
||||
- **Frontend**: tools (`frontend/editor/src/core/components/tools/*` or `.../core/tools/*`),
|
||||
hooks (`core/hooks/tools/*`, `useToolOperation`), contexts, routes, i18n
|
||||
(`public/locales/en-US`).
|
||||
- **Java backend**: controllers (`.../controller/api/...`), services, models, config.
|
||||
- **Engine**: `engine/src/stirling/{agents,contracts,api,services}`.
|
||||
- **Config / build / docker / tests.**
|
||||
|
||||
### 2. Trace the flow end-to-end
|
||||
Follow one real path from user action to result. For a typical PDF tool that's:
|
||||
UI control → `useToolOperation` hook → `POST /api/v1/...` → Spring controller →
|
||||
service (PDFBox / LibreOffice / engine call) → response → review panel → download.
|
||||
Read the actual files so the narrative is true to the code, and collect the exact
|
||||
file:line anchors you'll cite.
|
||||
|
||||
### 3. Draw the diagrams (Mermaid)
|
||||
Pick what fits; usually 2-3 of:
|
||||
- **Sequence diagram** - request/response across frontend → backend → engine.
|
||||
- **Flowchart** - the core decision/branching logic of the feature.
|
||||
- **Architecture/component** - new pieces and how they wire to existing ones.
|
||||
- **State** - if the feature has modes/steps.
|
||||
Keep nodes labeled in plain language. Validate the Mermaid parses before shipping.
|
||||
|
||||
### 4. Screenshots (unless --no-screens)
|
||||
If a UI is involved, capture key states with the stubbed Playwright harness
|
||||
(see the **ui-walkthrough** skill and `files-page-screenshots.spec.ts` for the
|
||||
pattern) or, for before/after, capture `main` then the branch. Drop PNGs in
|
||||
`walkthrough/<feature>/` and reference them from the doc. For backend-only
|
||||
changes, show request/response examples (curl + JSON) instead.
|
||||
|
||||
### 5. Write the walkthrough
|
||||
Create `walkthrough/<feature>/FEATURE-WALKTHROUGH.md` with:
|
||||
1. **TL;DR** - what the branch does and who it's for, in 3-4 sentences.
|
||||
2. **Problem & approach** - what wasn't possible before; the chosen solution.
|
||||
3. **Architecture diagram** + 1-paragraph orientation.
|
||||
4. **End-to-end flow** - the sequence diagram + a numbered walk of each step,
|
||||
each citing the real file (clickable `path:line`).
|
||||
5. **Key files** - annotated map (path → one line on its role).
|
||||
6. **Logic deep-dive** - the flowchart + prose for the non-obvious decisions.
|
||||
7. **Behavior** - before vs after; screenshots or request/response examples.
|
||||
8. **Try it locally** - exact steps (`task dev` / `task dev:all`, the route to
|
||||
open or the curl to run, any env like `DOCKER_ENABLE_SECURITY` or a test
|
||||
license key). Make it copy-pasteable.
|
||||
9. **Edge cases, risks, follow-ups** - what's untested, known limits, gotchas.
|
||||
|
||||
Markdown is the primary deliverable - it renders with diagrams in GitHub PRs and
|
||||
IDEs, no build step, ideal for review.
|
||||
|
||||
### 6. If `--html`
|
||||
Also emit `walkthrough/<feature>/walkthrough.html`: the same content with Mermaid
|
||||
rendered via `mermaid.initialize({startOnLoad:true})` (script from CDN; note in
|
||||
the file that rendering diagrams needs network, the `.md` is the offline copy) and
|
||||
screenshots inline. Keep it self-contained otherwise.
|
||||
|
||||
### 7. Deliver
|
||||
Give the doc path and a short chat summary. Offer to `SendUserFile` it.
|
||||
|
||||
## Principles
|
||||
- **True to the code.** Every claim traces to a file you read; cite `path:line`.
|
||||
No fabricated migration/version history.
|
||||
- **Newcomer-first.** Define repo-specific terms (FileContext, `useToolOperation`,
|
||||
the `@app/*` layer cascade, stubbed vs live tests) on first use.
|
||||
- **Show, don't assert.** Prefer a diagram + a real example over adjectives.
|
||||
- Don't commit the `walkthrough/` output unless asked.
|
||||
@@ -0,0 +1,122 @@
|
||||
---
|
||||
name: ui-before-after
|
||||
description: >-
|
||||
Analyse a branch or PR and automatically capture before/after screenshots of
|
||||
every UI surface its changes touch, then pixel-diff the pairs to surface what
|
||||
actually changed and assemble PR-ready before/after montage images. Generic and
|
||||
diff-driven: it derives the capture targets from the diff (changed tools/routes →
|
||||
URLs) instead of hand-listing screens, captures "before" from the base branch and
|
||||
"after" from the head, then keeps only the views that visually differ. Each
|
||||
comparison is auto-cropped to the region that actually changed (the bounding box of
|
||||
differing pixels), falling back to the full page only when the change spans most of
|
||||
it. Use for before/after shots, a visual diff of a branch/PR, "screenshots for the
|
||||
PR description", "show what changed in the UI", or a side-by-side of UI changes.
|
||||
Takes a PR number/URL (resolved via gh) or a branch; defaults to the current branch
|
||||
vs its base. Flags: --scope <selector>, --base <ref|merge-base>, --theme
|
||||
light|dark|both, --all (capture every route, not just changed), --no-autocrop,
|
||||
--pagewide <n>, --threshold <n>.
|
||||
argument-hint: "[PR# | PR-url | branch] [--scope <sel>] [--base <ref>] [--theme both] [--all] [--no-autocrop]"
|
||||
allowed-tools: Read, Write, Edit, Glob, Grep, Bash
|
||||
---
|
||||
|
||||
# UI Before / After (generic visual diff)
|
||||
|
||||
Point it at a branch or PR; it figures out which UI changed, screenshots every
|
||||
affected surface **before** (base) and **after** (head), pixel-diffs the pairs, and
|
||||
montages the ones that actually changed into images for the PR description.
|
||||
|
||||
`$ARGUMENTS`: a PR number/URL, a branch, or nothing (current branch vs base).
|
||||
By default it captures the full viewport and auto-crops each comparison to the region
|
||||
that changed. Flags: `--scope <css>` (narrow the *capture* to a container, e.g.
|
||||
`[data-sidebar="tool-panel"]`, when you already know where the change is),
|
||||
`--no-autocrop` (keep full frames), `--pagewide <fraction>` (above this share of the
|
||||
page, skip cropping; default 0.6), `--base <ref|merge-base>`,
|
||||
`--theme light|dark|both`, `--all` (walk every route, not just changed),
|
||||
`--threshold <fraction>` (diff sensitivity, default 0.001).
|
||||
|
||||
Shares the capture harness with **ui-walkthrough** - read its SKILL.md for the
|
||||
stubbed-Playwright setup, worktree node_modules + `generate-icons`, the
|
||||
stale-`:5173` gotcha, and the dark-mode init-script. Bundled helpers:
|
||||
[capture-spec.template.ts](capture-spec.template.ts), [diff-shots.mjs](diff-shots.mjs),
|
||||
[montage-template.html](montage-template.html), [shoot-sections.mjs](shoot-sections.mjs).
|
||||
|
||||
## Process
|
||||
|
||||
### 1. Resolve target + base
|
||||
```
|
||||
gh pr view <pr> --json number,title,headRefName,baseRefName,url,files # PR
|
||||
# or branch: base = merge-base(main, HEAD); head = HEAD
|
||||
gh pr diff <pr> --name-only # or: git diff --name-only <base>...HEAD
|
||||
```
|
||||
|
||||
### 2. Derive capture targets from the diff (the "analyse" step - no hand-listing)
|
||||
Map changed frontend files to URLs generically:
|
||||
- **Tools**: a changed `components/tools/<toolDir>/…` or `hooks/tools/<tool>/…` →
|
||||
toolId → URL via the repo's own rule `getToolUrlPath` in
|
||||
[toolsTaxonomy.ts:200](frontend/editor/src/core/data/toolsTaxonomy.ts): `/` + the
|
||||
id kebab-cased (`addPageNumbers` → `/add-page-numbers`).
|
||||
- **Pages/routes**: changed `filesPage/*` → `/files`, etc.
|
||||
- `--all`: enumerate every tool in the registry instead of just changed ones.
|
||||
Write `frontend/editor/screenshots/ui-diff/targets.json` =
|
||||
`[{ "id":"compress", "url":"/compress", "name":"Compress" }]`. This is what makes
|
||||
it generic - the spec never names a tool.
|
||||
|
||||
### 3. Capture AFTER (head) then BEFORE (base)
|
||||
Copy [capture-spec.template.ts](capture-spec.template.ts) →
|
||||
`src/core/tests/stubbed/ui-before-after.spec.ts` (it loops `targets.json`, seeds a
|
||||
sample PDF so file-dependent panels render, navigates to each URL, and screenshots
|
||||
the full viewport - or the `--scope` container if given). Ensure the harness is ready
|
||||
(node_modules + icons).
|
||||
```
|
||||
# after = current head
|
||||
cd frontend/editor && PR_SHOT_SIDE=after PR_SHOT_THEME=light \
|
||||
npx playwright test --project=stubbed ui-before-after.spec.ts
|
||||
# before = base, in an isolated worktree (copy the spec + targets.json in)
|
||||
git worktree add ../ba-base origin/<baseRefName> # or the merge-base
|
||||
# set up its frontend, copy spec + screenshots/ui-diff/targets.json across, then:
|
||||
cd ../ba-base/frontend/editor && PR_SHOT_SIDE=before PR_SHOT_THEME=light \
|
||||
npx playwright test --project=stubbed ui-before-after.spec.ts
|
||||
# copy its screenshots/ui-diff/before/ back next to after/. Repeat with
|
||||
# PR_SHOT_THEME=dark if --theme includes dark. Remove worktree when done.
|
||||
```
|
||||
|
||||
### 4. Auto-diff (surface what changed)
|
||||
```
|
||||
cd frontend/editor && node <skill>/diff-shots.mjs \
|
||||
screenshots/ui-diff/before screenshots/ui-diff/after screenshots/ui-diff
|
||||
```
|
||||
Produces `diff-report.json` classifying each view `unchanged | changed | added |
|
||||
removed`. For each changed view it computes the bounding box of differing pixels and
|
||||
writes cropped `__before_crop.png` / `__after_crop.png` / `__diff.png` to that region
|
||||
(+ padding) - **unless** the change covers more than `--pagewide` of the frame, where
|
||||
it keeps the full frame (`pageWide:true`). Drop `unchanged` - that's the noise the
|
||||
user doesn't want.
|
||||
|
||||
### 5. Montage the changes
|
||||
Build the manifest from the non-unchanged entries (group by tab/tool; each becomes a
|
||||
state row with before/after). For changed views use the cropped `cropBefore` /
|
||||
`cropAfter` from `diff-report.json` (tight on the affected region; full frame when
|
||||
`pageWide`); `added`/`removed` render the "not present" placeholder. Fill
|
||||
[montage-template.html](montage-template.html) (replace the `window.__BA__` data
|
||||
block; base64-inline the PNGs for portability), then render one PNG per section with
|
||||
[shoot-sections.mjs](shoot-sections.mjs). Optionally include the `__diff.png` overlay
|
||||
as a third column.
|
||||
|
||||
### 6. Deliver
|
||||
Output the `montage_<tab>.png` files + a short summary (N changed / added / removed,
|
||||
M unchanged skipped) and a paste-ready Markdown block. GitHub has no PR-body image
|
||||
API, so tell the user to drag the PNGs into the description. Do **not** post to the
|
||||
PR.
|
||||
|
||||
## Gotchas
|
||||
- Two installs (base worktree + head); junction main's node_modules only if its deps
|
||||
match that ref, else `npm ci` (see ui-walkthrough's stale-dep note).
|
||||
- A view that errors on one side (refactored/removed) → that side is missing; the
|
||||
diff marks it added/removed rather than failing the run.
|
||||
- Pixel diff needs equal dimensions, so capture at a fixed viewport (the template
|
||||
does); a view whose size changed is reported as "changed (dimensions differ)",
|
||||
uncropped.
|
||||
- Auto-crop uses a single bounding box, so two far-apart changes give one large crop
|
||||
(or trip `--pagewide`); narrow with `--scope` if that happens.
|
||||
- `getToolUrlPath` is the source of truth for tool URLs - use it, don't guess slugs.
|
||||
- Don't commit `screenshots/`, the throwaway spec, or the base worktree.
|
||||
@@ -0,0 +1,67 @@
|
||||
// Generic before/after capturer. NOT app-specific: it walks a targets.json that
|
||||
// the ui-before-after skill generates from the branch/PR diff, so nothing here is
|
||||
// hand-listed. Copy to src/core/tests/stubbed/ui-before-after.spec.ts, then run
|
||||
// once per (side, theme):
|
||||
// PR_SHOT_SIDE=after PR_SHOT_THEME=light \
|
||||
// npx playwright test --project=stubbed ui-before-after.spec.ts
|
||||
//
|
||||
// targets.json shape: [{ "id":"compress", "url":"/compress", "name":"Compress",
|
||||
// "needsFile": true }]
|
||||
import { test } from "@app/tests/helpers/stub-test-base";
|
||||
import type { Page } from "@playwright/test";
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
const SIDE = process.env.PR_SHOT_SIDE ?? "after";
|
||||
const THEME = process.env.PR_SHOT_THEME ?? "light";
|
||||
// Capture the full viewport by default so the affected region is in frame
|
||||
// wherever it is; diff-shots.mjs crops each comparison to what actually changed.
|
||||
// Set PR_SHOT_SCOPE to a selector to narrow the capture to one container.
|
||||
const SCOPE = process.env.PR_SHOT_SCOPE ?? "";
|
||||
const ROOT = path.resolve(process.cwd(), "screenshots", "ui-diff");
|
||||
const OUT = path.join(ROOT, SIDE);
|
||||
// A tiny sample PDF so file-dependent tool panels render. Point at a real fixture.
|
||||
const SAMPLE_PDF = process.env.PR_SHOT_SAMPLE ?? "src/core/tests/test-fixtures/sample.pdf";
|
||||
|
||||
type Target = { id: string; url: string; name?: string; needsFile?: boolean };
|
||||
const targets: Target[] = JSON.parse(fs.readFileSync(path.join(ROOT, "targets.json"), "utf-8"));
|
||||
|
||||
test.use({ autoGoto: false, viewport: { width: 1600, height: 900 }, seedJwt: true });
|
||||
|
||||
async function applyTheme(page: Page): Promise<void> {
|
||||
if (THEME !== "dark") return;
|
||||
await page.addInitScript(() => {
|
||||
localStorage.setItem("mantine-color-scheme", "dark");
|
||||
localStorage.setItem("mantine-color-scheme-value", "dark");
|
||||
});
|
||||
await page.emulateMedia({ colorScheme: "dark" });
|
||||
}
|
||||
|
||||
async function seedFile(page: Page): Promise<void> {
|
||||
if (!fs.existsSync(SAMPLE_PDF)) return;
|
||||
await page.goto("/", { waitUntil: "domcontentloaded" });
|
||||
await page.getByTestId("files-button").click().catch(() => {});
|
||||
await page.locator('[data-testid="file-input"]').setInputFiles(SAMPLE_PDF).catch(() => {});
|
||||
await page.locator(".file-sidebar-file-item").first().isVisible({ timeout: 8_000 }).catch(() => {});
|
||||
}
|
||||
|
||||
for (const t of targets) {
|
||||
// One test per target so a single failure doesn't drop the rest.
|
||||
test(`${SIDE}/${THEME} ${t.id}`, async ({ page }) => {
|
||||
fs.mkdirSync(OUT, { recursive: true });
|
||||
await applyTheme(page);
|
||||
if (t.needsFile !== false) await seedFile(page);
|
||||
await page.goto(t.url, { waitUntil: "domcontentloaded" });
|
||||
await page.waitForTimeout(400); // settle Mantine portals/transitions
|
||||
const shot = path.join(OUT, `${t.id}__${THEME}.png`);
|
||||
if (SCOPE) {
|
||||
const scope = page.locator(SCOPE).first();
|
||||
if (await scope.isVisible({ timeout: 8_000 }).catch(() => false)) {
|
||||
await scope.screenshot({ path: shot });
|
||||
return;
|
||||
}
|
||||
}
|
||||
// Full viewport (fixed size → stable dimensions for pixel diffing).
|
||||
await page.screenshot({ path: shot });
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,106 @@
|
||||
// Auto-diff before/ vs after/ screenshots, classify each as
|
||||
// unchanged | changed | added | removed, and CROP each changed pair to the
|
||||
// affected region (bounding box of differing pixels + padding) - unless the
|
||||
// change spans most of the page, in which case the full frame is kept.
|
||||
// Run from frontend/editor (so deps resolve):
|
||||
// node <skill>/diff-shots.mjs <beforeDir> <afterDir> [outDir]
|
||||
// Env:
|
||||
// DIFF_THRESHOLD min fraction of differing pixels to count as changed (default 0.001)
|
||||
// DIFF_PAD padding px around the affected region (default 24)
|
||||
// DIFF_PAGEWIDE if affected bbox area / image area exceeds this, keep full frame (default 0.6)
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { createRequire } from "node:module";
|
||||
|
||||
const require = createRequire(path.join(process.cwd(), "noop.js"));
|
||||
const pm = require("pixelmatch");
|
||||
const pixelmatch = pm.default || pm;
|
||||
const { PNG } = require("pngjs");
|
||||
|
||||
const beforeDir = path.resolve(process.argv[2]);
|
||||
const afterDir = path.resolve(process.argv[3]);
|
||||
const outDir = path.resolve(process.argv[4] || afterDir);
|
||||
const THRESHOLD = Number(process.env.DIFF_THRESHOLD ?? "0.001");
|
||||
const PAD = Number(process.env.DIFF_PAD ?? "24");
|
||||
const PAGEWIDE = Number(process.env.DIFF_PAGEWIDE ?? "0.6");
|
||||
|
||||
const read = (p) => PNG.sync.read(fs.readFileSync(p));
|
||||
const isShot = (f) => f.endsWith(".png") && !/__(diff|before_crop|after_crop)\.png$/.test(f);
|
||||
const list = (d) => (fs.existsSync(d) ? fs.readdirSync(d).filter(isShot) : []);
|
||||
const names = [...new Set([...list(beforeDir), ...list(afterDir)])].sort();
|
||||
fs.mkdirSync(outDir, { recursive: true });
|
||||
|
||||
function cropPNG(src, x, y, w, h) {
|
||||
const out = new PNG({ width: w, height: h });
|
||||
PNG.bitblt(src, out, x, y, w, h, 0, 0);
|
||||
return out;
|
||||
}
|
||||
const writePNG = (p, png) => fs.writeFileSync(p, PNG.sync.write(png));
|
||||
|
||||
// Bounding box of differing pixels using a diff mask (alpha>0 where changed).
|
||||
function changedBBox(before, after, w, h) {
|
||||
const mask = new PNG({ width: w, height: h });
|
||||
pixelmatch(before.data, after.data, mask.data, w, h, { threshold: 0.1, diffMask: true });
|
||||
let minX = w, minY = h, maxX = -1, maxY = -1, count = 0;
|
||||
for (let y = 0; y < h; y++) {
|
||||
for (let x = 0; x < w; x++) {
|
||||
if (mask.data[(y * w + x) * 4 + 3] > 0) {
|
||||
count++;
|
||||
if (x < minX) minX = x; if (x > maxX) maxX = x;
|
||||
if (y < minY) minY = y; if (y > maxY) maxY = y;
|
||||
}
|
||||
}
|
||||
}
|
||||
return maxX < 0 ? null : { minX, minY, maxX, maxY, count };
|
||||
}
|
||||
|
||||
const report = [];
|
||||
for (const name of names) {
|
||||
const id = name.replace(/\.png$/, "");
|
||||
const bp = path.join(beforeDir, name), ap = path.join(afterDir, name);
|
||||
const hasB = fs.existsSync(bp), hasA = fs.existsSync(ap);
|
||||
if (hasB && !hasA) { report.push({ id, status: "removed", before: bp }); continue; }
|
||||
if (!hasB && hasA) { report.push({ id, status: "added", after: ap }); continue; }
|
||||
|
||||
const before = read(bp), after = read(ap);
|
||||
if (before.width !== after.width || before.height !== after.height) {
|
||||
report.push({ id, status: "changed", note: "dimensions differ", before: bp, after: ap });
|
||||
continue;
|
||||
}
|
||||
const w = after.width, h = after.height;
|
||||
const overlay = new PNG({ width: w, height: h });
|
||||
const px = pixelmatch(before.data, after.data, overlay.data, w, h, { threshold: 0.1 });
|
||||
const ratio = px / (w * h);
|
||||
if (ratio <= THRESHOLD) { report.push({ id, status: "unchanged", ratio: Number(ratio.toFixed(5)), before: bp, after: ap }); continue; }
|
||||
|
||||
const box = changedBBox(before, after, w, h);
|
||||
// Pad + clamp the affected region.
|
||||
const x = Math.max(0, box.minX - PAD), y = Math.max(0, box.minY - PAD);
|
||||
const x2 = Math.min(w, box.maxX + 1 + PAD), y2 = Math.min(h, box.maxY + 1 + PAD);
|
||||
const bw = x2 - x, bh = y2 - y;
|
||||
const pageWide = (bw * bh) / (w * h) > PAGEWIDE;
|
||||
|
||||
const entry = { id, status: "changed", ratio: Number(ratio.toFixed(5)), before: bp, after: ap, pageWide };
|
||||
if (pageWide) {
|
||||
// Change spans most of the page - keep the full frame, full overlay.
|
||||
const dp = path.join(outDir, `${id}__diff.png`); writePNG(dp, overlay);
|
||||
entry.diff = dp;
|
||||
} else {
|
||||
entry.bbox = { x, y, w: bw, h: bh };
|
||||
const cb = path.join(outDir, `${id}__before_crop.png`); writePNG(cb, cropPNG(before, x, y, bw, bh));
|
||||
const ca = path.join(outDir, `${id}__after_crop.png`); writePNG(ca, cropPNG(after, x, y, bw, bh));
|
||||
const dp = path.join(outDir, `${id}__diff.png`); writePNG(dp, cropPNG(overlay, x, y, bw, bh));
|
||||
entry.cropBefore = cb; entry.cropAfter = ca; entry.diff = dp;
|
||||
}
|
||||
report.push(entry);
|
||||
}
|
||||
|
||||
fs.writeFileSync(path.join(outDir, "diff-report.json"), JSON.stringify(report, null, 2));
|
||||
const changed = report.filter((r) => r.status !== "unchanged");
|
||||
console.log(`diffed ${report.length} view(s): ${changed.length} changed/added/removed, ${report.length - changed.length} unchanged`);
|
||||
for (const r of changed) {
|
||||
const tail = r.status !== "changed" ? ""
|
||||
: r.pageWide ? " (page-wide → full frame)"
|
||||
: ` (${(r.ratio * 100).toFixed(2)}%, cropped to ${r.bbox.w}×${r.bbox.h})`;
|
||||
console.log(` ${r.status.padEnd(9)} ${r.id}${tail}${r.note ? " - " + r.note : ""}`);
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
"""Build EXAMPLE.html from montage-template.html using REAL files-page shots as
|
||||
stand-in before/after pairs (layout demo, not an actual PR diff). Inlines PNGs as
|
||||
data URIs so the HTML is portable. Run: python make_example.py"""
|
||||
import base64
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
|
||||
HERE = pathlib.Path(__file__).parent
|
||||
SHOTS = pathlib.Path(
|
||||
r"C:\Users\systo\git\Stirling-PDFNew\.claude\worktrees\kind-faraday-522a30"
|
||||
r"\frontend\editor\screenshots\files-page"
|
||||
)
|
||||
|
||||
|
||||
def uri(fname):
|
||||
p = SHOTS / fname
|
||||
return "data:image/png;base64," + base64.b64encode(p.read_bytes()).decode() if p.exists() else None
|
||||
|
||||
|
||||
data = {
|
||||
"pr": "DEMO",
|
||||
"title": "EXAMPLE — before/after montage (layout demo, real Files-page shots; not a real PR diff)",
|
||||
"base": "main", "head": "demo-branch",
|
||||
"cropSelector": "[data-sidebar=\"tool-panel\"] (real runs crop to the side; these demo shots are full-page)",
|
||||
"tabs": [
|
||||
{"id": "files", "title": "Files page", "ctx": "Each row = one flow state; left = base branch, right = this PR.",
|
||||
"states": [
|
||||
{"name": "Empty folder", "before": uri("01_empty_state_ctas.png"), "after": uri("02_empty_state_storage_off.png")},
|
||||
{"name": "Files + details panel", "before": uri("03_subtoolbar_with_files.png"), "after": uri("06_details_panel_save_to_server.png")},
|
||||
{"name": "Delete folder confirm", "before": None, "after": uri("19_delete_folder_dialog.png"), "note": "New in this PR"},
|
||||
]},
|
||||
{"id": "move", "title": "Move-to-folder dialog",
|
||||
"states": [
|
||||
{"name": "Dialog opened", "before": uri("07_move_dialog_collapsed.png"), "after": uri("08_move_dialog_create_folder_expanded.png")},
|
||||
{"name": "After folder created", "before": None, "after": uri("08b_move_dialog_after_create_folder.png"), "note": "New flow"},
|
||||
]},
|
||||
],
|
||||
}
|
||||
|
||||
tpl = (HERE / "montage-template.html").read_text(encoding="utf-8")
|
||||
out = re.sub(
|
||||
r"/\*__DATA__\*/.*?/\*__END__\*/",
|
||||
lambda _m: "/*__DATA__*/" + json.dumps(data) + "/*__END__*/",
|
||||
tpl, count=1, flags=re.S,
|
||||
)
|
||||
(HERE / "EXAMPLE.html").write_text(out, encoding="utf-8")
|
||||
print("wrote", HERE / "EXAMPLE.html", "(", (HERE / "EXAMPLE.html").stat().st_size // 1024, "KB )")
|
||||
@@ -0,0 +1,106 @@
|
||||
<!doctype html>
|
||||
<!--
|
||||
Before/After montage for a PR description. The ui-before-after skill replaces
|
||||
the JSON in the window.__BA__ data block below with the captured manifest, then
|
||||
screenshots each .tab-section (id="section-<tabId>") into a PNG to drag into the
|
||||
PR description. Self-contained; images may be relative paths or data URIs.
|
||||
|
||||
Data shape:
|
||||
{
|
||||
"pr":"6552","title":"...","base":"main","head":"feat/x",
|
||||
"cropSelector":"[data-sidebar=\"tool-panel\"]",
|
||||
"tabs":[
|
||||
{ "id":"sign","title":"Sign tool","states":[
|
||||
{"name":"Initial","before":"before/sign__initial.png","after":"after/sign__initial.png"},
|
||||
{"name":"Cert selected","before":null,"after":"after/sign__cert.png","note":"New in this PR"}
|
||||
]}
|
||||
]
|
||||
}
|
||||
-->
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<title>Before / After</title>
|
||||
<style>
|
||||
:root { --bg:#ffffff; --ink:#0b0c0e; --muted:#6b7280; --line:#e5e7eb;
|
||||
--before:#6b7280; --after:#1f883d; --frame:#f3f4f6; --note:#b45309; }
|
||||
* { box-sizing: border-box; }
|
||||
body { margin:0; background:var(--bg); color:var(--ink);
|
||||
font:14px/1.5 -apple-system,"Segoe UI",Roboto,system-ui,sans-serif; }
|
||||
.wrap { max-width:1100px; margin:0 auto; padding:24px; }
|
||||
.doc-head { margin-bottom:8px; }
|
||||
.doc-head h1 { font-size:18px; margin:0 0 2px; }
|
||||
.doc-head .sub { color:var(--muted); font-size:12.5px; }
|
||||
.legend { display:flex; gap:14px; align-items:center; margin:10px 0 4px; font-size:12px; color:var(--muted); }
|
||||
.chip { font-size:10px; font-weight:700; letter-spacing:.04em; text-transform:uppercase;
|
||||
padding:2px 8px; border-radius:999px; color:#fff; }
|
||||
.chip.before { background:var(--before); } .chip.after { background:var(--after); }
|
||||
|
||||
.tab-section { border:1px solid var(--line); border-radius:14px; padding:18px 18px 8px;
|
||||
margin:18px 0; background:var(--bg); }
|
||||
.tab-section > h2 { font-size:16px; margin:0 0 2px; }
|
||||
.tab-section > .ctx { color:var(--muted); font-size:12px; margin-bottom:14px; }
|
||||
.state { margin-bottom:18px; }
|
||||
.state .name { font-weight:600; font-size:13.5px; margin-bottom:8px; display:flex; gap:8px; align-items:center; }
|
||||
.state .name .note { font-weight:500; color:var(--note); font-size:12px; }
|
||||
.pair { display:grid; grid-template-columns:1fr 1fr; gap:14px; align-items:start; }
|
||||
.cell { border:1px solid var(--line); border-radius:10px; overflow:hidden; background:var(--frame); }
|
||||
.cell .cap { display:flex; align-items:center; gap:8px; padding:7px 10px; border-bottom:1px solid var(--line);
|
||||
background:var(--bg); }
|
||||
.cell .cap .meta { color:var(--muted); font-size:11px; }
|
||||
.cell img { display:block; width:100%; height:auto; background:#fff; }
|
||||
.cell.empty .ph { display:flex; align-items:center; justify-content:center; height:160px; color:var(--muted);
|
||||
font-size:12.5px; text-align:center; padding:0 16px; }
|
||||
.single .pair { grid-template-columns:1fr; }
|
||||
.empty-doc { color:var(--muted); padding:40px; text-align:center; }
|
||||
@media (max-width:760px){ .pair{ grid-template-columns:1fr; } }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="wrap" id="root"></div>
|
||||
|
||||
<script id="data">
|
||||
window.__BA__ = /*__DATA__*/{"pr":"","title":"No data","base":"","head":"","cropSelector":"","tabs":[]}/*__END__*/;
|
||||
</script>
|
||||
<script>
|
||||
(function(){
|
||||
var D = window.__BA__ || { tabs: [] };
|
||||
var root = document.getElementById("root");
|
||||
function el(html){ var t=document.createElement("template"); t.innerHTML=html.trim(); return t.content.firstChild; }
|
||||
function esc(s){ return (s==null?"":String(s)).replace(/[&<>]/g, function(c){return {"&":"&","<":"<",">":">"}[c];}); }
|
||||
|
||||
function cell(kind, src){
|
||||
if (src) {
|
||||
return '<div class="cell"><div class="cap"><span class="chip '+kind+'">'+kind+'</span></div>'+
|
||||
'<img src="'+esc(src)+'" alt="'+kind+'"/></div>';
|
||||
}
|
||||
return '<div class="cell empty"><div class="cap"><span class="chip '+kind+'">'+kind+'</span>'+
|
||||
'<span class="meta">not present</span></div><div class="ph">No '+kind+' screenshot for this state</div></div>';
|
||||
}
|
||||
|
||||
var head = '<div class="doc-head"><h1>'+esc(D.title || ("PR #"+D.pr))+'</h1>'+
|
||||
'<div class="sub">Before / after · base <code>'+esc(D.base)+'</code> → head <code>'+esc(D.head)+'</code>'+
|
||||
(D.cropSelector ? ' · cropped to <code>'+esc(D.cropSelector)+'</code>' : '')+'</div></div>'+
|
||||
'<div class="legend"><span class="chip before">Before</span> base branch'+
|
||||
'<span class="chip after">After</span> this PR</div>';
|
||||
root.appendChild(el('<div>'+head+'</div>'));
|
||||
|
||||
if (!D.tabs || !D.tabs.length){ root.appendChild(el('<div class="empty-doc">No tabs captured yet.</div>')); return; }
|
||||
|
||||
D.tabs.forEach(function(tab){
|
||||
var states = (tab.states||[]).map(function(s){
|
||||
var onlyOne = (!s.before || !s.after);
|
||||
return '<div class="state'+(onlyOne?' ':'')+'">'+
|
||||
'<div class="name">'+esc(s.name)+(s.note?'<span class="note">'+esc(s.note)+'</span>':'')+'</div>'+
|
||||
'<div class="pair">'+cell("before", s.before)+cell("after", s.after)+'</div></div>';
|
||||
}).join("");
|
||||
var sec = '<section class="tab-section" id="section-'+esc(tab.id)+'">'+
|
||||
'<h2>'+esc(tab.title)+'</h2>'+
|
||||
(tab.ctx?'<div class="ctx">'+esc(tab.ctx)+'</div>':'')+
|
||||
states+'</section>';
|
||||
root.appendChild(el(sec));
|
||||
});
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,25 @@
|
||||
// Render each .tab-section of a montage HTML into its own PNG (the PR-ready image).
|
||||
// Run from frontend/editor (so @playwright/test resolves):
|
||||
// node <skill>/shoot-sections.mjs <montage.html> <outDir>
|
||||
import path from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { createRequire } from "node:module";
|
||||
|
||||
const require = createRequire(path.join(process.cwd(), "noop.js"));
|
||||
const { chromium } = require("@playwright/test");
|
||||
|
||||
const htmlPath = path.resolve(process.argv[2]);
|
||||
const outDir = path.resolve(process.argv[3] || path.dirname(htmlPath));
|
||||
|
||||
const browser = await chromium.launch();
|
||||
const page = await browser.newPage({ viewport: { width: 1200, height: 1200 }, deviceScaleFactor: 2 });
|
||||
await page.goto(pathToFileURL(htmlPath).href, { waitUntil: "load" });
|
||||
await page.waitForTimeout(250); // let images/fonts paint
|
||||
const ids = await page.$$eval(".tab-section", (els) => els.map((e) => e.id));
|
||||
if (!ids.length) { console.error("no .tab-section found"); process.exit(1); }
|
||||
for (const id of ids) {
|
||||
const name = id.replace(/^section-/, "");
|
||||
await page.locator("#" + id).screenshot({ path: path.join(outDir, `montage_${name}.png`) });
|
||||
console.log("wrote montage_" + name + ".png");
|
||||
}
|
||||
await browser.close();
|
||||
@@ -0,0 +1,120 @@
|
||||
---
|
||||
name: ui-walkthrough
|
||||
description: >-
|
||||
Full UI investigation of the current branch's feature. Enumerates every view
|
||||
and state (empty, populated, loading, error, each dialog/menu/panel, responsive
|
||||
breakpoints, light + dark + RTL), captures them with the stubbed Playwright
|
||||
harness, assembles a single-image HTML walkthrough with a global light/dark
|
||||
toggle slider, then runs two review passes: visual/consistency (alignment,
|
||||
spacing, professionalism, dark/light parity, contrast, truncation) and
|
||||
UX/ease-of-use (flow, discoverability, affordances, empty/error states,
|
||||
expectations). Use when asked for a UI walkthrough, screenshot review, design
|
||||
or QA pass, "find anywhere to make it easier/better for users", or before
|
||||
merging frontend work. Pass --fix to auto-apply safe frontend fixes and
|
||||
re-capture; --theme to limit themes; --no-rtl to skip RTL.
|
||||
argument-hint: "[feature/area] [--fix] [--theme light|dark|both] [--no-rtl] [--breakpoints]"
|
||||
allowed-tools: Read, Write, Edit, Glob, Grep, Bash
|
||||
---
|
||||
|
||||
# UI Walkthrough
|
||||
|
||||
Produce a reviewable HTML walkthrough of a feature's UI in every state and theme,
|
||||
then critique it. Optionally auto-fix and re-capture.
|
||||
|
||||
`$ARGUMENTS` may name the feature/area to focus on. If empty, scope from the
|
||||
current branch diff. Flags: `--fix`, `--theme light|dark|both` (default both),
|
||||
`--no-rtl`, `--breakpoints` (also capture phone/narrow widths).
|
||||
|
||||
## What this repo gives you (use it, don't reinvent)
|
||||
|
||||
- **Stubbed Playwright project** = backend-free screenshots via `page.route()` mocks.
|
||||
Reference implementation: `frontend/editor/src/core/tests/stubbed/files-page-screenshots.spec.ts`.
|
||||
It already shows the light / **dark** / **RTL** passes, JWT seeding, IndexedDB
|
||||
seeding, and dumping PNGs to a `screenshots/<area>/` folder. Copy its shape.
|
||||
- Helpers: `frontend/editor/src/core/tests/helpers/ui-helpers.ts`
|
||||
(`uploadFiles`, `openSettings`, `waitForModalOpen`, `dismissTourTooltip`, …)
|
||||
and the `stub-test-base` fixtures (`autoGoto`, `seedJwt`, `viewport`).
|
||||
- Config: `frontend/editor/playwright.config.ts` (run from `frontend/editor/`).
|
||||
- Report template: [report-template.html](report-template.html) - self-contained,
|
||||
one big image at a time, a global light/dark slider that flips every shot,
|
||||
thumbnail rail, prev/next + arrow keys, and a Findings tab.
|
||||
|
||||
## Process
|
||||
|
||||
### 1. Scope the feature
|
||||
- If `$ARGUMENTS` is empty: `git diff --name-only main...HEAD` and read the PR/commits.
|
||||
Identify changed pages, tools (`core/components/tools/<tool>` or `core/tools/<tool>`),
|
||||
dialogs, panels, and routes.
|
||||
- Enumerate **every view and state** to capture, e.g.:
|
||||
empty / populated / loading / error / disabled; each dialog, menu, popover, tooltip;
|
||||
each tab or step; selection + multi-select; success/result panel; and (if relevant)
|
||||
permission/role variants. Write the list down before capturing - it's the report's spine.
|
||||
|
||||
### 2. Prepare the harness (worktree-safe)
|
||||
Worktrees have no `node_modules` and no generated icons. From repo root:
|
||||
```
|
||||
cd frontend && npm ci # or junction main's node_modules (see memory)
|
||||
cd frontend/editor && node scripts/generate-icons.js
|
||||
```
|
||||
Kill any stale dev server first (it serves old modules):
|
||||
`Get-NetTCPConnection -LocalPort 5173 -State Listen | %{ Stop-Process -Id $_.OwningProcess -Force }`
|
||||
|
||||
### 3. Write the capture spec
|
||||
Create `frontend/editor/src/core/tests/stubbed/<feature>-walkthrough.spec.ts`,
|
||||
modeled on `files-page-screenshots.spec.ts`. For each enumerated view:
|
||||
- stub the APIs it needs, drive the UI to that state, wait on a real locator
|
||||
(not a fixed sleep), `await settle(page)` for Mantine portals, then
|
||||
`page.screenshot({ path: shotPath("NN_name_<theme>") })`.
|
||||
- Capture each view in **light and dark** (and RTL unless `--no-rtl`). Reuse the
|
||||
`enableDarkMode` / `enableRtl` init-script pattern from the reference spec
|
||||
(`localStorage["mantine-color-scheme"]="dark"` + `emulateMedia({colorScheme:"dark"})`).
|
||||
- Name shots `NN_<view>_<theme>.png` so light/dark pair up by suffix.
|
||||
- Prefer **stable test-ids** over translated accessible names (RTL/i18n breaks text locators).
|
||||
|
||||
Run it: `cd frontend/editor && npx playwright test --project=stubbed <feature>-walkthrough.spec.ts`.
|
||||
Add `--project=stubbed-firefox`/`-webkit` only if cross-browser layout matters.
|
||||
|
||||
### 4. Build the report
|
||||
- Copy `report-template.html` to `screenshots/<feature>/walkthrough.html` (so the
|
||||
relative `screenshots/...` image paths resolve, or rewrite paths to sit beside it).
|
||||
- Build the manifest and inject it: replace the JSON between the
|
||||
`/*__DATA__*/` … `/*__END__*/` markers with one `views[]` entry per view
|
||||
(`{id,title,light,dark,viewport,notes}`) and an empty `findings` object you'll
|
||||
fill in step 5. Keep `light`/`dark` as relative paths.
|
||||
- The toggle slider answers the "one big image + flip light/dark for all" request:
|
||||
it shows a single large screenshot, and switching the slider re-themes every view.
|
||||
|
||||
### 5. Review pass 1 - visual & consistency
|
||||
Open each screenshot (Read the PNG) and judge against the others:
|
||||
alignment & spacing rhythm, control placement, button hierarchy, typography,
|
||||
**light/dark parity** (contrast, invisible borders, washed-out text, wrong tokens),
|
||||
truncation/overflow, RTL mirroring, focus states, icon consistency, professional polish.
|
||||
Record each issue as a finding `{severity:high|med|low, view, title, detail, fix}`.
|
||||
|
||||
### 6. Review pass 2 - UX & ease of use
|
||||
Walk the flow as a first-time user: discoverability, number of steps, affordance
|
||||
clarity, empty-state guidance, error recovery, destructive-action confirmation,
|
||||
defaults, loading feedback, mobile reachability, accessible names, and whether the
|
||||
UI matches user expectations for this kind of tool. Record findings the same way.
|
||||
|
||||
Write both finding lists into the report's `findings.visual` / `findings.ux`,
|
||||
and add short per-view `notes`. Re-inject the manifest.
|
||||
|
||||
### 7. If `--fix`
|
||||
Only safe, self-contained frontend fixes (spacing, alignment, tokens, missing
|
||||
dark-mode colors, labels, aria, obvious copy). For each: edit the component/CSS,
|
||||
mark the finding `fixed:true` with what changed, then **re-run the spec** to
|
||||
re-capture the affected shots and regenerate the report. Run `task frontend:check`.
|
||||
Leave anything risky or ambiguous as a finding, not a change.
|
||||
|
||||
### 8. Deliver
|
||||
Tell the user the report path and give a tight chat summary: N views ×
|
||||
themes captured, top findings by severity, and (if `--fix`) what changed.
|
||||
Optionally `SendUserFile` the `walkthrough.html`.
|
||||
|
||||
## Gotchas
|
||||
- Stale `:5173` server serves old bundles - kill it before capturing (see step 2).
|
||||
- Missing `material-symbols-icons.json` → blank app → every shot times out. Run
|
||||
`generate-icons.js` first.
|
||||
- `await settle(page)` before shots or portals/transitions tear mid-capture.
|
||||
- Don't commit the generated `screenshots/` or the throwaway spec unless asked.
|
||||
@@ -0,0 +1,116 @@
|
||||
"""Build a self-contained EXAMPLE.html from report-template.html with mock
|
||||
light/dark screenshots, so the viewer + global theme slider can be demoed
|
||||
without a real capture run. Run: python make_example.py"""
|
||||
import base64
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
|
||||
HERE = pathlib.Path(__file__).parent
|
||||
|
||||
|
||||
def svg(bg, fg, panel, accent, muted, label, kind):
|
||||
"""A simple fake 'screen' SVG: title bar, sidebar, content varies by kind."""
|
||||
parts = [
|
||||
f'<svg xmlns="http://www.w3.org/2000/svg" width="1600" height="900" viewBox="0 0 1600 900">',
|
||||
f'<rect width="1600" height="900" fill="{bg}"/>',
|
||||
# top bar
|
||||
f'<rect width="1600" height="64" fill="{panel}"/>',
|
||||
f'<circle cx="40" cy="32" r="12" fill="{accent}"/>',
|
||||
f'<rect x="64" y="24" width="160" height="16" rx="6" fill="{muted}"/>',
|
||||
f'<rect x="1430" y="20" width="130" height="24" rx="12" fill="{accent}"/>',
|
||||
# left sidebar
|
||||
f'<rect x="0" y="64" width="220" height="836" fill="{panel}"/>',
|
||||
]
|
||||
for i in range(6):
|
||||
y = 100 + i * 56
|
||||
parts.append(f'<rect x="24" y="{y}" width="172" height="32" rx="8" fill="{bg}"/>')
|
||||
if kind == "empty":
|
||||
parts += [
|
||||
f'<rect x="700" y="360" width="200" height="120" rx="16" fill="none" stroke="{muted}" stroke-width="3" stroke-dasharray="10 8"/>',
|
||||
f'<rect x="690" y="510" width="220" height="44" rx="10" fill="{accent}"/>',
|
||||
f'<text x="800" y="600" fill="{muted}" font-family="sans-serif" font-size="26" text-anchor="middle">{label}</text>',
|
||||
]
|
||||
elif kind == "form":
|
||||
for i in range(4):
|
||||
y = 140 + i * 90
|
||||
parts.append(f'<rect x="280" y="{y}" width="160" height="16" rx="6" fill="{muted}"/>')
|
||||
parts.append(f'<rect x="280" y="{y+26}" width="900" height="44" rx="8" fill="{panel}" stroke="{muted}" stroke-width="1"/>')
|
||||
parts.append(f'<rect x="280" y="560" width="200" height="50" rx="10" fill="{accent}"/>')
|
||||
parts.append(f'<text x="800" y="850" fill="{muted}" font-family="sans-serif" font-size="24" text-anchor="middle">{label}</text>')
|
||||
else: # dialog
|
||||
parts += [
|
||||
f'<rect width="1600" height="900" fill="{fg}" opacity="0.45"/>',
|
||||
f'<rect x="520" y="280" width="560" height="360" rx="18" fill="{panel}"/>',
|
||||
f'<rect x="556" y="320" width="280" height="22" rx="8" fill="{fg}"/>',
|
||||
f'<rect x="556" y="372" width="488" height="14" rx="6" fill="{muted}"/>',
|
||||
f'<rect x="556" y="398" width="420" height="14" rx="6" fill="{muted}"/>',
|
||||
f'<rect x="820" y="560" width="110" height="44" rx="9" fill="{bg}" stroke="{muted}"/>',
|
||||
f'<rect x="946" y="560" width="98" height="44" rx="9" fill="{accent}"/>',
|
||||
f'<text x="800" y="700" fill="#fff" font-family="sans-serif" font-size="24" text-anchor="middle">{label}</text>',
|
||||
]
|
||||
parts.append("</svg>")
|
||||
return "".join(parts)
|
||||
|
||||
|
||||
def data_uri(s):
|
||||
return "data:image/svg+xml;base64," + base64.b64encode(s.encode()).decode()
|
||||
|
||||
|
||||
LIGHT = dict(bg="#ffffff", fg="#111418", panel="#f1f3f6", accent="#2f6fed", muted="#c2c8d0")
|
||||
DARK = dict(bg="#16181c", fg="#000000", panel="#1f232a", accent="#5b8cff", muted="#3a414b")
|
||||
|
||||
|
||||
def pair(kind, label):
|
||||
return (
|
||||
data_uri(svg(LIGHT["bg"], LIGHT["fg"], LIGHT["panel"], LIGHT["accent"], LIGHT["muted"], label, kind)),
|
||||
data_uri(svg(DARK["bg"], DARK["fg"], DARK["panel"], DARK["accent"], DARK["muted"], label, kind)),
|
||||
)
|
||||
|
||||
|
||||
views = []
|
||||
for idx, (kind, title, label) in enumerate([
|
||||
("empty", "Empty state", "Drop a PDF to start"),
|
||||
("form", "Tool options panel", "Compress options"),
|
||||
("dialog", "Confirm dialog", "Replace original file?"),
|
||||
], start=1):
|
||||
light, dark = pair(kind, label)
|
||||
views.append({
|
||||
"id": f"{idx:02d}_{kind}",
|
||||
"title": title,
|
||||
"light": light,
|
||||
"dark": dark,
|
||||
"viewport": "1600x900",
|
||||
"notes": ["This is mock data to demo the viewer."],
|
||||
})
|
||||
|
||||
data = {
|
||||
"feature": "EXAMPLE - Compress PDF (mock data)",
|
||||
"branch": "demo",
|
||||
"generated": "example",
|
||||
"views": views,
|
||||
"findings": {
|
||||
"visual": [
|
||||
{"severity": "high", "view": "03_dialog", "title": "Dialog buttons too close",
|
||||
"detail": "Cancel/Confirm have only 8px gap; easy to misclick.",
|
||||
"fix": "Increase gap to var(--mantine-spacing-md)."},
|
||||
{"severity": "low", "view": "02_form", "title": "Field labels low contrast in dark mode",
|
||||
"detail": "Muted token fails WCAG AA on the dark panel.",
|
||||
"fix": "Use --mantine-color-dimmed instead of a hard-coded grey."},
|
||||
],
|
||||
"ux": [
|
||||
{"severity": "med", "view": "01_empty", "title": "Primary CTA below the dropzone",
|
||||
"detail": "Users expect the action button adjacent to the dropzone.",
|
||||
"fix": "Move the button directly under the dashed zone."},
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
tpl = (HERE / "report-template.html").read_text(encoding="utf-8")
|
||||
out = re.sub(
|
||||
r"/\*__DATA__\*/.*?/\*__END__\*/",
|
||||
lambda _m: "/*__DATA__*/" + json.dumps(data) + "/*__END__*/",
|
||||
tpl, count=1, flags=re.S,
|
||||
)
|
||||
(HERE / "EXAMPLE.html").write_text(out, encoding="utf-8")
|
||||
print("wrote", (HERE / "EXAMPLE.html"))
|
||||
@@ -0,0 +1,298 @@
|
||||
<!doctype html>
|
||||
<!--
|
||||
UI Walkthrough report template (self-contained, works from file://).
|
||||
The ui-walkthrough skill replaces the JSON in the window.__WALKTHROUGH__ data
|
||||
block below with the captured manifest. Do not add external CDN deps - it must open offline.
|
||||
|
||||
Data shape:
|
||||
{
|
||||
"feature": "Compress PDF tool",
|
||||
"branch": "claude/...",
|
||||
"generated": "2026-06-21",
|
||||
"views": [
|
||||
{ "id": "01_empty", "title": "Empty state",
|
||||
"light": "screenshots/compress/01_empty_light.png",
|
||||
"dark": "screenshots/compress/01_empty_dark.png",
|
||||
"viewport": "1600x900",
|
||||
"notes": ["Heading is centered", "Primary CTA below the fold on mobile"] }
|
||||
],
|
||||
"findings": {
|
||||
"visual": [ { "severity":"high", "view":"01_empty", "title":"...", "detail":"...", "fix":"..." } ],
|
||||
"ux": [ { "severity":"med", "view":"03_dialog", "title":"...", "detail":"...", "fix":"..." } ]
|
||||
}
|
||||
}
|
||||
-->
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>UI Walkthrough</title>
|
||||
<style>
|
||||
:root {
|
||||
--bg: #f6f7f9; --panel: #ffffff; --panel-2: #f0f2f5; --text: #1a1b1e;
|
||||
--muted: #6b7280; --border: #e2e5ea; --accent: #2f6fed; --accent-weak: #e8f0fe;
|
||||
--shadow: 0 1px 3px rgba(0,0,0,.08), 0 8px 24px rgba(0,0,0,.06);
|
||||
--hi: #d92d20; --med: #d98e00; --low: #2f6fed; --stage: #0b0c0e;
|
||||
}
|
||||
html[data-theme="dark"] {
|
||||
--bg: #0d0e10; --panel: #16181c; --panel-2: #1d2024; --text: #e6e8eb;
|
||||
--muted: #9aa3ad; --border: #2a2e35; --accent: #5b8cff; --accent-weak: #1a2336;
|
||||
--shadow: 0 1px 3px rgba(0,0,0,.5), 0 8px 24px rgba(0,0,0,.4); --stage: #000;
|
||||
}
|
||||
* { box-sizing: border-box; }
|
||||
body { margin: 0; font: 14px/1.5 -apple-system, "Segoe UI", Roboto, system-ui, sans-serif;
|
||||
background: var(--bg); color: var(--text); }
|
||||
header { display: flex; align-items: center; gap: 16px; padding: 12px 20px;
|
||||
background: var(--panel); border-bottom: 1px solid var(--border); position: sticky; top: 0; z-index: 5; }
|
||||
header h1 { font-size: 15px; margin: 0; font-weight: 650; }
|
||||
header .sub { color: var(--muted); font-size: 12px; }
|
||||
.spacer { flex: 1; }
|
||||
.counter { color: var(--muted); font-variant-numeric: tabular-nums; font-size: 13px; }
|
||||
.tabs { display: flex; gap: 4px; }
|
||||
.tab { border: 1px solid var(--border); background: var(--panel-2); color: var(--text);
|
||||
padding: 6px 12px; border-radius: 8px; cursor: pointer; font-size: 13px; }
|
||||
.tab.active { background: var(--accent); color: #fff; border-color: var(--accent); }
|
||||
|
||||
/* Light/Dark slider */
|
||||
.theme-toggle { display: flex; align-items: center; gap: 9px; user-select: none; }
|
||||
.theme-toggle .lbl { font-size: 12px; color: var(--muted); }
|
||||
.theme-toggle .lbl.on { color: var(--text); font-weight: 600; }
|
||||
.switch { position: relative; width: 52px; height: 28px; }
|
||||
.switch input { opacity: 0; width: 0; height: 0; }
|
||||
.slider { position: absolute; inset: 0; cursor: pointer; background: var(--panel-2);
|
||||
border: 1px solid var(--border); border-radius: 999px; transition: .2s; }
|
||||
.slider:before { content: ""; position: absolute; height: 20px; width: 20px; left: 3px; top: 3px;
|
||||
background: #fbbf24; border-radius: 50%; transition: .2s; box-shadow: 0 1px 2px rgba(0,0,0,.3); }
|
||||
.switch input:checked + .slider { background: var(--accent); }
|
||||
.switch input:checked + .slider:before { transform: translateX(24px); background: #c7d2fe; }
|
||||
|
||||
main { display: grid; grid-template-columns: 240px 1fr; height: calc(100vh - 53px); }
|
||||
.rail { border-right: 1px solid var(--border); overflow-y: auto; background: var(--panel); padding: 8px; }
|
||||
.rail .group-label { font-size: 11px; text-transform: uppercase; letter-spacing: .05em;
|
||||
color: var(--muted); padding: 10px 8px 4px; }
|
||||
.thumb { display: flex; gap: 9px; align-items: center; padding: 7px; border-radius: 8px;
|
||||
cursor: pointer; border: 1px solid transparent; }
|
||||
.thumb:hover { background: var(--panel-2); }
|
||||
.thumb.active { background: var(--accent-weak); border-color: var(--accent); }
|
||||
.thumb img { width: 64px; height: 40px; object-fit: cover; border-radius: 4px; border: 1px solid var(--border); background: var(--stage); }
|
||||
.thumb .t { font-size: 12.5px; line-height: 1.3; }
|
||||
.thumb .badge { font-size: 10px; color: var(--muted); }
|
||||
.thumb .dot { width: 7px; height: 7px; border-radius: 50%; margin-left: auto; flex: none; }
|
||||
|
||||
.stagewrap { display: flex; flex-direction: column; min-width: 0; }
|
||||
.stage { flex: 1; display: flex; align-items: center; justify-content: center; padding: 22px;
|
||||
background: var(--stage); position: relative; min-height: 0; }
|
||||
.stage img { max-width: 100%; max-height: 100%; object-fit: contain; border-radius: 8px;
|
||||
box-shadow: 0 4px 30px rgba(0,0,0,.4); background: #fff; }
|
||||
html[data-theme="dark"] .stage img { background: #16181c; }
|
||||
.nav-btn { position: absolute; top: 50%; transform: translateY(-50%); width: 42px; height: 42px;
|
||||
border-radius: 50%; border: 1px solid var(--border); background: var(--panel);
|
||||
color: var(--text); cursor: pointer; font-size: 18px; opacity: .85; }
|
||||
.nav-btn:hover { opacity: 1; } .nav-btn.prev { left: 16px; } .nav-btn.next { right: 16px; }
|
||||
.nav-btn:disabled { opacity: .25; cursor: default; }
|
||||
.missing { color: var(--muted); font-size: 13px; text-align: center; }
|
||||
|
||||
.detail { border-top: 1px solid var(--border); background: var(--panel); padding: 14px 20px;
|
||||
max-height: 38vh; overflow-y: auto; }
|
||||
.detail h2 { margin: 0 0 4px; font-size: 15px; }
|
||||
.detail .meta { color: var(--muted); font-size: 12px; margin-bottom: 10px; }
|
||||
.notes { list-style: none; padding: 0; margin: 0; display: grid; gap: 6px; }
|
||||
.notes li { display: flex; gap: 8px; align-items: flex-start; }
|
||||
.sev { font-size: 10px; font-weight: 700; text-transform: uppercase; padding: 2px 7px; border-radius: 999px;
|
||||
color: #fff; flex: none; margin-top: 1px; }
|
||||
.sev.high { background: var(--hi); } .sev.med { background: var(--med); } .sev.low { background: var(--low); }
|
||||
.finding .fix { color: var(--muted); font-size: 12.5px; }
|
||||
.finding .fix b { color: var(--text); font-weight: 600; }
|
||||
|
||||
/* Summary tab */
|
||||
.summary { padding: 20px 28px; overflow-y: auto; }
|
||||
.summary h2 { font-size: 16px; margin: 22px 0 8px; }
|
||||
.summary .empty { color: var(--muted); }
|
||||
.card { background: var(--panel); border: 1px solid var(--border); border-radius: 10px;
|
||||
padding: 12px 14px; margin-bottom: 8px; box-shadow: var(--shadow); }
|
||||
.card .head { display: flex; gap: 8px; align-items: center; }
|
||||
.card a { color: var(--accent); text-decoration: none; cursor: pointer; }
|
||||
.hide { display: none !important; }
|
||||
kbd { font: 11px ui-monospace, monospace; background: var(--panel-2); border: 1px solid var(--border);
|
||||
border-radius: 4px; padding: 1px 5px; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header>
|
||||
<div>
|
||||
<h1 id="feature-title">UI Walkthrough</h1>
|
||||
<div class="sub" id="feature-sub"></div>
|
||||
</div>
|
||||
<div class="spacer"></div>
|
||||
<div class="tabs">
|
||||
<button class="tab active" data-tab="viewer">Walkthrough</button>
|
||||
<button class="tab" data-tab="summary">Findings</button>
|
||||
</div>
|
||||
<div class="counter" id="counter"></div>
|
||||
<label class="theme-toggle" title="Toggle light / dark for every screenshot">
|
||||
<span class="lbl" id="lbl-light">Light</span>
|
||||
<span class="switch"><input type="checkbox" id="theme-switch" /><span class="slider"></span></span>
|
||||
<span class="lbl" id="lbl-dark">Dark</span>
|
||||
</label>
|
||||
</header>
|
||||
|
||||
<main id="viewer-pane">
|
||||
<aside class="rail" id="rail"></aside>
|
||||
<section class="stagewrap">
|
||||
<div class="stage">
|
||||
<button class="nav-btn prev" id="prev" aria-label="Previous">‹</button>
|
||||
<img id="stage-img" alt="" />
|
||||
<div class="missing hide" id="missing"></div>
|
||||
<button class="nav-btn next" id="next" aria-label="Next">›</button>
|
||||
</div>
|
||||
<div class="detail">
|
||||
<h2 id="view-title"></h2>
|
||||
<div class="meta" id="view-meta"></div>
|
||||
<ul class="notes" id="view-notes"></ul>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
|
||||
<section class="summary hide" id="summary-pane"></section>
|
||||
|
||||
<script id="data">
|
||||
window.__WALKTHROUGH__ = /*__DATA__*/{"feature":"No data","branch":"","generated":"","views":[],"findings":{"visual":[],"ux":[]}}/*__END__*/;
|
||||
</script>
|
||||
<script>
|
||||
(function () {
|
||||
var D = window.__WALKTHROUGH__ || { views: [], findings: { visual: [], ux: [] } };
|
||||
var views = D.views || [];
|
||||
var state = { i: 0, theme: localStorage.getItem("ui-wt-theme") || "light", tab: "viewer" };
|
||||
|
||||
var $ = function (id) { return document.getElementById(id); };
|
||||
function sevClass(s) { return s === "high" ? "high" : s === "med" || s === "medium" ? "med" : "low"; }
|
||||
|
||||
function applyChrome() {
|
||||
document.documentElement.setAttribute("data-theme", state.theme);
|
||||
$("theme-switch").checked = state.theme === "dark";
|
||||
$("lbl-light").classList.toggle("on", state.theme === "light");
|
||||
$("lbl-dark").classList.toggle("on", state.theme === "dark");
|
||||
}
|
||||
|
||||
function srcFor(v) { return state.theme === "dark" ? (v.dark || v.light) : (v.light || v.dark); }
|
||||
|
||||
function findingsForView(id) {
|
||||
var all = (D.findings && D.findings.visual || []).concat(D.findings && D.findings.ux || []);
|
||||
return all.filter(function (f) { return f.view === id; });
|
||||
}
|
||||
|
||||
function renderRail() {
|
||||
var rail = $("rail");
|
||||
rail.innerHTML = "";
|
||||
if (!views.length) { rail.innerHTML = '<div class="group-label">No views captured</div>'; return; }
|
||||
views.forEach(function (v, idx) {
|
||||
var fs = findingsForView(v.id);
|
||||
var worst = fs.some(function (f){return sevClass(f.severity)==="high";}) ? "var(--hi)"
|
||||
: fs.some(function (f){return sevClass(f.severity)==="med";}) ? "var(--med)"
|
||||
: fs.length ? "var(--low)" : "transparent";
|
||||
var el = document.createElement("div");
|
||||
el.className = "thumb" + (idx === state.i ? " active" : "");
|
||||
el.innerHTML = '<img src="' + srcFor(v) + '" alt="" />' +
|
||||
'<div><div class="t">' + (v.title || v.id) + '</div>' +
|
||||
'<div class="badge">' + (v.viewport || "") + '</div></div>' +
|
||||
'<span class="dot" style="background:' + worst + '"></span>';
|
||||
el.onclick = function () { state.i = idx; render(); };
|
||||
rail.appendChild(el);
|
||||
});
|
||||
}
|
||||
|
||||
function render() {
|
||||
applyChrome();
|
||||
if (!views.length) {
|
||||
$("missing").classList.remove("hide"); $("stage-img").classList.add("hide");
|
||||
$("missing").textContent = "No screenshots in this report yet.";
|
||||
$("counter").textContent = ""; return;
|
||||
}
|
||||
var v = views[state.i];
|
||||
var src = srcFor(v);
|
||||
var img = $("stage-img");
|
||||
if (src) {
|
||||
img.classList.remove("hide"); $("missing").classList.add("hide");
|
||||
img.src = src; img.alt = v.title || v.id;
|
||||
} else {
|
||||
img.classList.add("hide"); $("missing").classList.remove("hide");
|
||||
$("missing").textContent = "No " + state.theme + " screenshot for this view.";
|
||||
}
|
||||
$("counter").textContent = (state.i + 1) + " / " + views.length;
|
||||
$("view-title").textContent = v.title || v.id;
|
||||
$("view-meta").textContent = [v.viewport, state.theme + " mode"].filter(Boolean).join(" · ");
|
||||
var notes = $("view-notes"); notes.innerHTML = "";
|
||||
var fs = findingsForView(v.id);
|
||||
(v.notes || []).forEach(function (n) {
|
||||
var li = document.createElement("li"); li.textContent = "· " + n; notes.appendChild(li);
|
||||
});
|
||||
fs.forEach(function (f) {
|
||||
var li = document.createElement("li"); li.className = "finding";
|
||||
li.innerHTML = '<span class="sev ' + sevClass(f.severity) + '">' + (f.severity || "note") + '</span>' +
|
||||
'<span><b>' + (f.title || "") + '</b> — ' + (f.detail || "") +
|
||||
(f.fix ? ' <span class="fix"><b>Fix:</b> ' + f.fix + '</span>' : '') + '</span>';
|
||||
notes.appendChild(li);
|
||||
});
|
||||
$("prev").disabled = state.i === 0;
|
||||
$("next").disabled = state.i === views.length - 1;
|
||||
renderRail();
|
||||
}
|
||||
|
||||
function renderSummary() {
|
||||
var pane = $("summary-pane");
|
||||
function block(title, arr) {
|
||||
var h = '<h2>' + title + ' (' + arr.length + ')</h2>';
|
||||
if (!arr.length) return h + '<div class="empty">None found.</div>';
|
||||
return h + arr.map(function (f) {
|
||||
return '<div class="card"><div class="head">' +
|
||||
'<span class="sev ' + sevClass(f.severity) + '">' + (f.severity || "note") + '</span>' +
|
||||
'<b>' + (f.title || "") + '</b>' +
|
||||
(f.view ? ' <a data-jump="' + f.view + '">' + f.view + '</a>' : '') + '</div>' +
|
||||
'<div style="margin-top:6px">' + (f.detail || "") + '</div>' +
|
||||
(f.fix ? '<div class="finding" style="margin-top:6px"><span class="fix"><b>Fix:</b> ' + f.fix + '</span></div>' : '') +
|
||||
'</div>';
|
||||
}).join("");
|
||||
}
|
||||
pane.innerHTML = block("Visual & consistency", (D.findings && D.findings.visual) || []) +
|
||||
block("UX & ease of use", (D.findings && D.findings.ux) || []);
|
||||
pane.querySelectorAll("[data-jump]").forEach(function (a) {
|
||||
a.onclick = function () {
|
||||
var id = a.getAttribute("data-jump");
|
||||
var idx = views.findIndex(function (v) { return v.id === id; });
|
||||
if (idx >= 0) { state.i = idx; setTab("viewer"); }
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
function setTab(t) {
|
||||
state.tab = t;
|
||||
document.querySelectorAll(".tab").forEach(function (b) { b.classList.toggle("active", b.dataset.tab === t); });
|
||||
$("viewer-pane").classList.toggle("hide", t !== "viewer");
|
||||
$("summary-pane").classList.toggle("hide", t !== "summary");
|
||||
if (t === "viewer") $("viewer-pane").style.display = "grid";
|
||||
if (t === "summary") renderSummary();
|
||||
}
|
||||
|
||||
// wiring
|
||||
$("feature-title").textContent = D.feature || "UI Walkthrough";
|
||||
$("feature-sub").textContent = [D.branch, D.generated].filter(Boolean).join(" · ");
|
||||
$("theme-switch").onchange = function () {
|
||||
state.theme = this.checked ? "dark" : "light";
|
||||
localStorage.setItem("ui-wt-theme", state.theme);
|
||||
render();
|
||||
};
|
||||
$("prev").onclick = function () { if (state.i > 0) { state.i--; render(); } };
|
||||
$("next").onclick = function () { if (state.i < views.length - 1) { state.i++; render(); } };
|
||||
document.addEventListener("keydown", function (e) {
|
||||
if (state.tab !== "viewer") return;
|
||||
if (e.key === "ArrowLeft") $("prev").click();
|
||||
if (e.key === "ArrowRight") $("next").click();
|
||||
if (e.key.toLowerCase() === "t") $("theme-switch").click();
|
||||
});
|
||||
document.querySelectorAll(".tab").forEach(function (b) { b.onclick = function () { setTab(b.dataset.tab); }; });
|
||||
|
||||
render();
|
||||
})();
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -87,6 +87,13 @@ stirling/
|
||||
customFiles/
|
||||
configs/
|
||||
|
||||
# Local dev runtime dirs created by bootRun under the module dir (hold the locked H2 DB,
|
||||
# downloaded models, logs); never build input. Root-level patterns above don't match these.
|
||||
**/configs/
|
||||
app/core/customFiles/
|
||||
app/core/logs/
|
||||
app/core/pipeline/
|
||||
|
||||
# Claude Code workspace
|
||||
.claude/
|
||||
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
build: &build
|
||||
- build.gradle
|
||||
- app/(common|core|proprietary)/build.gradle
|
||||
- app/(common|core|proprietary|saas)/build.gradle
|
||||
- Taskfile.yml
|
||||
- .taskfiles/backend.yml
|
||||
|
||||
openapi: &openapi
|
||||
- *build
|
||||
- app/(common|core|proprietary)/src/main/java/**
|
||||
- app/(common|core|proprietary|saas)/src/main/java/**
|
||||
|
||||
docker-base: &docker-base
|
||||
- docker/base/Dockerfile
|
||||
@@ -23,9 +23,9 @@ docker: &docker
|
||||
- *docker-base
|
||||
|
||||
project: &project
|
||||
- app/(common|core|proprietary)/src/(main|test)/java/**
|
||||
- app/(common|core|proprietary|saas)/src/(main|test)/java/**
|
||||
- *build
|
||||
- "app/(common|core|proprietary)/src/(main|test)/resources/**/!(messages_*.properties|*.md)*"
|
||||
- "app/(common|core|proprietary|saas)/src/(main|test)/resources/**/!(messages_*.properties|*.md)*"
|
||||
- exampleYmlFiles/**
|
||||
- gradle/**
|
||||
- libs/**
|
||||
@@ -82,7 +82,7 @@ tauri: &tauri
|
||||
# tool surfaces it generates models from.
|
||||
engine: &engine
|
||||
- engine/**
|
||||
- app/(common|core|proprietary)/src/main/java/**
|
||||
- app/(common|core|proprietary|saas)/src/main/java/**
|
||||
- .github/workflows/ai-engine.yml
|
||||
- Taskfile.yml
|
||||
- .taskfiles/engine.yml
|
||||
|
||||
@@ -63,6 +63,7 @@ labels:
|
||||
files:
|
||||
- 'app/core/src/main/resources/static/.*'
|
||||
- 'app/proprietary/src/main/resources/static/.*'
|
||||
- 'app/saas/src/main/resources/static/.*'
|
||||
- 'frontend/**'
|
||||
- 'frontend/.*'
|
||||
- 'frontend/**/.*'
|
||||
@@ -83,6 +84,7 @@ labels:
|
||||
- 'app/common/src/main/java/.*.java'
|
||||
- 'app/proprietary/src/main/java/.*.java'
|
||||
- 'app/core/src/main/java/.*.java'
|
||||
- 'app/saas/src/main/java/.*.java'
|
||||
|
||||
- label: 'Back End'
|
||||
files:
|
||||
@@ -90,6 +92,9 @@ labels:
|
||||
- 'app/core/src/main/java/stirling/software/SPDF/controller/.*'
|
||||
- 'app/core/src/main/resources/settings.yml.template'
|
||||
- 'app/core/src/main/resources/application.properties'
|
||||
- 'app/proprietary/src/main/resources/application-proprietary.properties'
|
||||
- 'app/saas/src/main/resources/application-dev.properties'
|
||||
- 'app/saas/src/main/resources/application-saas.properties'
|
||||
- 'app/core/src/main/resources/banner.txt'
|
||||
- 'app/core/src/main/resources/static/python/png_to_webp.py'
|
||||
- 'app/core/src/main/resources/static/python/split_photos.py'
|
||||
@@ -153,6 +158,7 @@ labels:
|
||||
- 'app/common/src/test/.*'
|
||||
- 'app/proprietary/src/test/.*'
|
||||
- 'app/core/src/test/.*'
|
||||
- 'app/saas/src/test/.*'
|
||||
- 'testing/.*'
|
||||
- '.github/workflows/scorecards.yml'
|
||||
- 'exampleYmlFiles/test_cicd.yml'
|
||||
@@ -171,3 +177,4 @@ labels:
|
||||
- 'app/common/build.gradle'
|
||||
- 'app/proprietary/build.gradle'
|
||||
- 'app/core/build.gradle'
|
||||
- 'app/saas/build.gradle'
|
||||
|
||||
@@ -116,6 +116,9 @@ jobs:
|
||||
env:
|
||||
USE_DEPOT: ${{ needs.pick.outputs.is_fork != 'true' }}
|
||||
DEPOT_TOKEN: ${{ secrets.DEPOT_TOKEN }}
|
||||
# Single source of truth for whether this preview embeds the admin portal:
|
||||
# drives the image build-arg and the deployment comment.
|
||||
BUILD_PORTAL: "true"
|
||||
|
||||
steps:
|
||||
- name: Harden Runner
|
||||
@@ -246,12 +249,14 @@ jobs:
|
||||
file: ./docker/embedded/Dockerfile
|
||||
push: true
|
||||
tags: ${{ secrets.DOCKER_HUB_USERNAME }}/test:v2-${{ steps.commit-hash.outputs.app_short }}
|
||||
build-args: VERSION_TAG=v2-alpha
|
||||
build-args: |
|
||||
VERSION_TAG=v2-alpha
|
||||
BUILD_PORTAL=${{ env.BUILD_PORTAL }}
|
||||
platforms: linux/amd64
|
||||
|
||||
- name: Build and push V2 image (Docker fork fallback)
|
||||
if: env.USE_DEPOT != 'true' && steps.check-image.outputs.exists == 'false'
|
||||
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
|
||||
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
|
||||
with:
|
||||
context: .
|
||||
file: ./docker/embedded/Dockerfile
|
||||
@@ -259,7 +264,9 @@ jobs:
|
||||
cache-from: type=gha,scope=stirling-pdf-latest
|
||||
cache-to: type=gha,mode=max,scope=stirling-pdf-latest
|
||||
tags: ${{ secrets.DOCKER_HUB_USERNAME }}/test:v2-${{ steps.commit-hash.outputs.app_short }}
|
||||
build-args: VERSION_TAG=v2-alpha
|
||||
build-args: |
|
||||
VERSION_TAG=v2-alpha
|
||||
BUILD_PORTAL=${{ env.BUILD_PORTAL }}
|
||||
platforms: linux/amd64
|
||||
|
||||
- name: Set up SSH
|
||||
@@ -290,6 +297,8 @@ jobs:
|
||||
- /stirling/V2-PR-${{ needs.check-pr.outputs.pr_number }}/storage:/storage:rw
|
||||
environment:
|
||||
DISABLE_ADDITIONAL_FEATURES: "false"
|
||||
POLICIES_ENABLED: "true"
|
||||
STIRLING_BILLING_ACCOUNT_LINK_ENABLED: "true"
|
||||
SECURITY_ENABLELOGIN: "true"
|
||||
SECURITY_INITIALLOGIN_USERNAME: "${{ secrets.TEST_LOGIN_USERNAME }}"
|
||||
SECURITY_INITIALLOGIN_PASSWORD: "${{ secrets.TEST_LOGIN_PASSWORD }}"
|
||||
@@ -333,9 +342,70 @@ jobs:
|
||||
# Set port for output
|
||||
echo "v2_port=${V2_PORT}" >> $GITHUB_OUTPUT
|
||||
|
||||
# ---- Storybook preview (only when this PR touches stories/.storybook) ----
|
||||
# Runs inside the same approved-contributor-gated deploy job, so it deploys
|
||||
# under the exact same access rules as the app preview.
|
||||
- name: Detect Storybook changes
|
||||
id: sb-changes
|
||||
uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v4.0.1
|
||||
with:
|
||||
list-files: json
|
||||
filters: |
|
||||
storybook:
|
||||
- 'frontend/**/*.stories.@(ts|tsx|mdx)'
|
||||
- 'frontend/**/*.mdx'
|
||||
- 'frontend/.storybook/**'
|
||||
|
||||
- name: Set up Node.js for Storybook
|
||||
if: steps.sb-changes.outputs.storybook == 'true'
|
||||
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: "22"
|
||||
cache: "npm"
|
||||
cache-dependency-path: frontend/package-lock.json
|
||||
|
||||
- name: Install Task for Storybook
|
||||
if: steps.sb-changes.outputs.storybook == 'true'
|
||||
uses: go-task/setup-task@01a4adf9db2d14c1de7a560f09170b6e0df736aa # v2.1.0
|
||||
|
||||
- name: Build and deploy Storybook
|
||||
id: storybook
|
||||
if: steps.sb-changes.outputs.storybook == 'true'
|
||||
env:
|
||||
VPS_HOST: ${{ secrets.NEW_VPS_HOST }}
|
||||
VPS_USER: ${{ secrets.NEW_VPS_USERNAME }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# `prepare` generates the icon set stories import (not committed).
|
||||
task frontend:prepare
|
||||
task frontend:storybook:build
|
||||
PR=${{ needs.check-pr.outputs.pr_number }}
|
||||
# Served at the ROOT of its own port so Storybook's global MSW worker
|
||||
# (/mockServiceWorker.js) resolves. Port = PR + 20000 (bijective, offset
|
||||
# from the app preview's bare-PR-number port).
|
||||
SB_PORT=$((PR + 20000))
|
||||
DIR=/stirling/SB-PR-$PR
|
||||
tar czf storybook.tgz -C frontend/storybook-static .
|
||||
scp -i ../private.key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
|
||||
storybook.tgz "$VPS_USER@$VPS_HOST:/tmp/storybook-$PR.tgz"
|
||||
ssh -i ../private.key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -T \
|
||||
"$VPS_USER@$VPS_HOST" << ENDSSH
|
||||
set -e
|
||||
rm -rf "$DIR" && mkdir -p "$DIR"
|
||||
tar xzf /tmp/storybook-$PR.tgz -C "$DIR"
|
||||
rm -f /tmp/storybook-$PR.tgz
|
||||
docker rm -f storybook-pr-$PR 2>/dev/null || true
|
||||
docker run -d --name storybook-pr-$PR --restart unless-stopped \
|
||||
-p $SB_PORT:80 -v "$DIR":/usr/share/nginx/html:ro nginx:alpine
|
||||
ENDSSH
|
||||
echo "url=http://$VPS_HOST:$SB_PORT/" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Post V2 deployment URL to PR
|
||||
if: success()
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
||||
env:
|
||||
SB_URL: ${{ steps.storybook.outputs.url }}
|
||||
SB_FILES: ${{ steps.sb-changes.outputs.storybook_files }}
|
||||
with:
|
||||
github-token: ${{ steps.setup-bot.outputs.token }}
|
||||
script: |
|
||||
@@ -359,12 +429,40 @@ jobs:
|
||||
}
|
||||
|
||||
const deploymentUrl = `http://${{ secrets.NEW_VPS_HOST }}:${v2Port}`;
|
||||
const httpsUrl = `https://${v2Port}.ssl.stirlingpdf.cloud`;
|
||||
|
||||
// Only mention the portal when this image actually embeds it.
|
||||
// Use the direct IP URL - the SSL hostname isn't supported yet.
|
||||
const withPortal = "${{ env.BUILD_PORTAL }}" === "true";
|
||||
const portalNote = withPortal
|
||||
? `🧩 **Admin portal** included - try it at [${deploymentUrl}/portal](${deploymentUrl}/portal).\n\n`
|
||||
: ``;
|
||||
|
||||
// Storybook preview: only present when this PR changed stories/config.
|
||||
const sbUrl = process.env.SB_URL;
|
||||
let storybookNote = "";
|
||||
if (sbUrl) {
|
||||
const files = JSON.parse(process.env.SB_FILES || "[]");
|
||||
const stories = files.filter((f) => /\.stories\.(ts|tsx|mdx)$/.test(f));
|
||||
const config = files.filter((f) => f.startsWith("frontend/.storybook/"));
|
||||
const shorten = (f) =>
|
||||
f.replace(/^frontend\/editor\/src\//, "").replace(/^frontend\//, "");
|
||||
const storyList = stories.map((f) => `- \`${shorten(f)}\``).join("\n");
|
||||
const configList = config.map((f) => `- \`${shorten(f)}\``).join("\n");
|
||||
const summary =
|
||||
`${stories.length} stor${stories.length === 1 ? "y" : "ies"} changed` +
|
||||
(config.length ? ` (+${config.length} config file${config.length === 1 ? "" : "s"})` : "");
|
||||
storybookNote =
|
||||
`📚 **Storybook:** [${sbUrl}](${sbUrl})\n\n` +
|
||||
`<details>\n<summary>${summary}</summary>\n\n` +
|
||||
(storyList ? `**Stories**\n${storyList}\n\n` : "") +
|
||||
(configList ? `**Config**\n${configList}\n` : "") +
|
||||
`</details>\n\n`;
|
||||
}
|
||||
|
||||
const commentBody = `## 🚀 V2 Auto-Deployment Complete!\n\n` +
|
||||
`Your V2 PR with embedded architecture has been deployed!\n\n` +
|
||||
`🔗 **Direct Test URL (non-SSL)** [${deploymentUrl}](${deploymentUrl})\n\n` +
|
||||
`🔐 **Secure HTTPS URL**: [${httpsUrl}](${httpsUrl})\n\n` +
|
||||
portalNote +
|
||||
storybookNote +
|
||||
`_This deployment will be automatically cleaned up when the PR is closed._\n\n` +
|
||||
`🔄 **Auto-deployed** for approved V2 contributors.`;
|
||||
|
||||
@@ -460,7 +558,11 @@ jobs:
|
||||
else
|
||||
echo "V2 PR directory not found, nothing to clean up"
|
||||
fi
|
||||
|
||||
|
||||
# Remove this PR's Storybook preview (container + files), if any.
|
||||
docker rm -f storybook-pr-${{ github.event.pull_request.number }} 2>/dev/null || true
|
||||
rm -rf /stirling/SB-PR-${{ github.event.pull_request.number }}
|
||||
|
||||
# Clean up old unused images (older than 2 weeks) but keep recent ones for reuse
|
||||
docker image prune -af --filter "until=336h" --filter "label!=keep=true" || true
|
||||
|
||||
|
||||
@@ -270,7 +270,7 @@ jobs:
|
||||
|
||||
- name: Build and push PR-specific image (Docker fork fallback)
|
||||
if: env.USE_DEPOT != 'true'
|
||||
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
|
||||
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
|
||||
with:
|
||||
context: .
|
||||
file: ./docker/embedded/Dockerfile
|
||||
@@ -296,7 +296,7 @@ jobs:
|
||||
|
||||
- name: Build and push engine image (Docker fork fallback)
|
||||
if: env.USE_DEPOT != 'true' && needs.check-comment.outputs.enable_prototypes == 'true'
|
||||
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
|
||||
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
|
||||
with:
|
||||
context: ./engine
|
||||
file: ./engine/Dockerfile
|
||||
|
||||
@@ -47,7 +47,7 @@ jobs:
|
||||
distribution: "temurin"
|
||||
|
||||
- name: Cache Gradle dependency artifacts
|
||||
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
|
||||
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
||||
with:
|
||||
path: |
|
||||
~/.gradle/wrapper
|
||||
|
||||
@@ -29,7 +29,7 @@ jobs:
|
||||
distribution: "temurin"
|
||||
|
||||
- name: Cache Gradle dependency artifacts
|
||||
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
|
||||
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
||||
with:
|
||||
path: |
|
||||
~/.gradle/wrapper
|
||||
|
||||
@@ -34,7 +34,7 @@ jobs:
|
||||
distribution: "temurin"
|
||||
|
||||
- name: Cache Gradle dependency artifacts
|
||||
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
|
||||
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
||||
with:
|
||||
path: |
|
||||
~/.gradle/wrapper
|
||||
|
||||
@@ -51,7 +51,7 @@ jobs:
|
||||
distribution: "temurin"
|
||||
|
||||
- name: Cache Gradle dependency artifacts
|
||||
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
|
||||
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
||||
with:
|
||||
path: |
|
||||
~/.gradle/wrapper
|
||||
|
||||
@@ -37,7 +37,7 @@ jobs:
|
||||
distribution: temurin
|
||||
|
||||
- name: Cache Gradle dependency artifacts
|
||||
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
|
||||
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
||||
with:
|
||||
path: |
|
||||
~/.gradle/wrapper
|
||||
|
||||
@@ -121,7 +121,7 @@ jobs:
|
||||
|
||||
- name: Build and push frontend image (Docker fork fallback)
|
||||
if: env.USE_DEPOT != 'true' && steps.check-frontend.outputs.exists == 'false'
|
||||
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
|
||||
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
|
||||
with:
|
||||
context: .
|
||||
file: ./docker/frontend/Dockerfile
|
||||
@@ -150,7 +150,7 @@ jobs:
|
||||
|
||||
- name: Build and push backend image (Docker fork fallback)
|
||||
if: env.USE_DEPOT != 'true' && steps.check-backend.outputs.exists == 'false'
|
||||
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
|
||||
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
|
||||
with:
|
||||
context: .
|
||||
file: ./docker/backend/Dockerfile
|
||||
|
||||
@@ -50,7 +50,7 @@ jobs:
|
||||
distribution: "temurin"
|
||||
|
||||
- name: Cache Gradle dependency artifacts
|
||||
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
|
||||
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
||||
with:
|
||||
path: |
|
||||
~/.gradle/wrapper
|
||||
|
||||
@@ -61,7 +61,7 @@ jobs:
|
||||
distribution: "temurin"
|
||||
|
||||
- name: Cache Gradle dependencies
|
||||
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
|
||||
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
||||
with:
|
||||
path: |
|
||||
~/.gradle/caches
|
||||
|
||||
@@ -85,7 +85,7 @@ jobs:
|
||||
|
||||
- name: Build and push base image
|
||||
id: build-push-base
|
||||
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
|
||||
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
|
||||
with:
|
||||
builder: ${{ steps.buildx.outputs.name }}
|
||||
context: docker/base
|
||||
|
||||
@@ -66,7 +66,7 @@ jobs:
|
||||
distribution: "temurin"
|
||||
|
||||
- name: Cache Gradle dependencies
|
||||
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
|
||||
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
||||
with:
|
||||
path: |
|
||||
~/.gradle/caches
|
||||
@@ -145,7 +145,7 @@ jobs:
|
||||
id: build-push-latest
|
||||
# Empty-tag guard: build-push-action errors when asked to push with no tags.
|
||||
if: env.RUN_MAIN_APP == 'true' && steps.meta.outputs.tags != ''
|
||||
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
|
||||
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
|
||||
with:
|
||||
builder: ${{ steps.buildx.outputs.name }}
|
||||
context: .
|
||||
@@ -192,7 +192,7 @@ jobs:
|
||||
|
||||
- name: Build and push Unified Dockerfile (fat variant)
|
||||
id: build-push-fat
|
||||
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
|
||||
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
|
||||
if: env.RUN_MAIN_APP == 'true' && github.ref != 'refs/heads/main' && github.ref != 'refs/heads/testMain' && steps.meta-fat.outputs.tags != ''
|
||||
with:
|
||||
builder: ${{ steps.buildx.outputs.name }}
|
||||
@@ -236,7 +236,7 @@ jobs:
|
||||
|
||||
- name: Build and push Unified Dockerfile (ultra-lite variant)
|
||||
id: build-push-lite
|
||||
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
|
||||
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
|
||||
if: env.RUN_MAIN_APP == 'true' && github.ref != 'refs/heads/main' && github.ref != 'refs/heads/testMain' && steps.meta-lite.outputs.tags != ''
|
||||
with:
|
||||
builder: ${{ steps.buildx.outputs.name }}
|
||||
@@ -365,7 +365,7 @@ jobs:
|
||||
- name: Build and push unoserver image
|
||||
id: build-push-unoserver
|
||||
if: env.RUN_UNOSERVER == 'true' && steps.unoserverDecision.outputs.mode != 'skip'
|
||||
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
|
||||
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
|
||||
with:
|
||||
builder: ${{ steps.buildx.outputs.name }}
|
||||
context: .
|
||||
|
||||
@@ -95,7 +95,7 @@ jobs:
|
||||
distribution: "temurin"
|
||||
|
||||
- name: Cache Gradle dependency artifacts
|
||||
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
|
||||
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
|
||||
with:
|
||||
path: |
|
||||
~/.gradle/wrapper
|
||||
@@ -186,7 +186,7 @@ jobs:
|
||||
# (multi-platform + gha cache) against the published base image.
|
||||
- name: Build ${{ matrix.docker-rev }} (Docker fork fallback)
|
||||
if: env.USE_DEPOT != 'true' && inputs.docker-base-changed != 'true'
|
||||
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
|
||||
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
|
||||
with:
|
||||
builder: ${{ steps.buildx.outputs.name }}
|
||||
context: .
|
||||
@@ -259,7 +259,7 @@ jobs:
|
||||
|
||||
- name: Build docker/unoserver/Dockerfile (Docker fork fallback)
|
||||
if: env.USE_DEPOT != 'true'
|
||||
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
|
||||
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
|
||||
with:
|
||||
builder: ${{ steps.buildx.outputs.name }}
|
||||
context: .
|
||||
|
||||
@@ -95,7 +95,7 @@ jobs:
|
||||
|
||||
- name: Build and push test image (Docker fork fallback)
|
||||
if: env.USE_DEPOT != 'true'
|
||||
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
|
||||
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0
|
||||
with:
|
||||
context: .
|
||||
file: ./docker/embedded/Dockerfile
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"ignoredFiles": [
|
||||
"frontend/editor/src-tauri/icons/icon.png"
|
||||
]
|
||||
}
|
||||
@@ -80,6 +80,12 @@ tasks:
|
||||
OPEN: '{{.OPEN | default ""}}'
|
||||
env:
|
||||
BACKEND_URL: '{{.BACKEND_URL}}'
|
||||
# Dev-only browser-tab label so concurrent worktrees are distinguishable.
|
||||
# Only the worktree folder basename (e.g. "wt1") is exposed — never the
|
||||
# full path, hostname, or user. Consumed at dev-serve time by vite.config
|
||||
# and dropped from production builds.
|
||||
STIRLING_DEV_LABEL:
|
||||
sh: basename "$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cmds:
|
||||
- npx vite editor --mode {{.MODE}} --port {{.PORT}}{{if .OPEN}} --open{{end}}
|
||||
|
||||
|
||||
@@ -453,6 +453,7 @@ The frontend is organized with a clear separation of concerns:
|
||||
|
||||
- **CRITICAL**: Always update translations in `en-US` only - all other languages (including `en-GB`) are handled separately
|
||||
- Translation files are located in `frontend/editor/public/locales/`
|
||||
- After changing any translation file, run `task pre-commit:fix`
|
||||
|
||||
## Important Notes
|
||||
|
||||
|
||||
@@ -536,6 +536,7 @@ public class EndpointConfiguration {
|
||||
addEndpointToGroup("Java", "pdf-to-epub");
|
||||
addEndpointToGroup("Java", "eml-to-pdf");
|
||||
addEndpointToGroup("Java", "handleData");
|
||||
addEndpointToGroup("Java", "form-detection");
|
||||
addEndpointToGroup("rar", "pdf-to-cbr");
|
||||
|
||||
// Javascript
|
||||
|
||||
@@ -40,6 +40,9 @@ public class RuntimePathConfig {
|
||||
// Tesseract data path
|
||||
private final String tessDataPath;
|
||||
|
||||
// Auto Form Detection model directory
|
||||
private final String formDetectionModelPath;
|
||||
|
||||
private final List<ApplicationProperties.ProcessExecutor.UnoServerEndpoint> unoServerEndpoints;
|
||||
|
||||
// Pipeline paths
|
||||
@@ -130,6 +133,22 @@ public class RuntimePathConfig {
|
||||
|
||||
log.info("Using Tesseract data path: {}", this.tessDataPath);
|
||||
|
||||
// Auto Form Detection model directory (kept under <configs> so it survives
|
||||
// restarts/updates)
|
||||
String configuredModelDir =
|
||||
properties.getFormDetection() != null
|
||||
? properties.getFormDetection().getModelDir()
|
||||
: null;
|
||||
this.formDetectionModelPath =
|
||||
StringUtils.isNotBlank(configuredModelDir)
|
||||
? configuredModelDir
|
||||
: Path.of(
|
||||
InstallationPathConfig.getConfigPath(),
|
||||
"models",
|
||||
"form-detection")
|
||||
.toString();
|
||||
log.info("Using Auto Form Detection model path: {}", this.formDetectionModelPath);
|
||||
|
||||
ApplicationProperties.ProcessExecutor processExecutor = properties.getProcessExecutor();
|
||||
int libreOfficeLimit = 1;
|
||||
if (processExecutor != null && processExecutor.getSessionLimit() != null) {
|
||||
|
||||
@@ -77,6 +77,7 @@ public class ApplicationProperties {
|
||||
private ProcessExecutor processExecutor = new ProcessExecutor();
|
||||
private PdfEditor pdfEditor = new PdfEditor();
|
||||
private AiEngine aiEngine = new AiEngine();
|
||||
private FormDetection formDetection = new FormDetection();
|
||||
private Mcp mcp = new Mcp();
|
||||
private InternalApi internalApi = new InternalApi();
|
||||
private Cluster cluster = new Cluster();
|
||||
@@ -246,6 +247,14 @@ public class ApplicationProperties {
|
||||
* and paused runs are kept regardless of age.
|
||||
*/
|
||||
private int runExpiryMinutes = 30;
|
||||
|
||||
/**
|
||||
* Whether a policy S3 source's custom endpoint may resolve to a loopback, link-local, or
|
||||
* private address. Off by default so a user-supplied endpoint cannot be pointed at internal
|
||||
* services (e.g. the cloud metadata address); enable for a self-hosted MinIO or other
|
||||
* in-network object store.
|
||||
*/
|
||||
private boolean allowPrivateS3Endpoints = false;
|
||||
}
|
||||
|
||||
@Data
|
||||
@@ -302,6 +311,37 @@ public class ApplicationProperties {
|
||||
private int longRunningTimeoutSeconds = 600;
|
||||
}
|
||||
|
||||
/**
|
||||
* Auto Form Detection settings. The model itself is downloaded on demand by an admin (see
|
||||
* {@code /api/v1/ai/form-detection-model/*}); only lightweight pointers are persisted here.
|
||||
*/
|
||||
@Data
|
||||
public static class FormDetection {
|
||||
/** Master on/off switch for the whole feature (admin-controlled). */
|
||||
private boolean enabled = true;
|
||||
|
||||
/**
|
||||
* Where detection runs: {@code auto} (browser first, server fallback), {@code browser}
|
||||
* (in-browser WASM only - the PDF never leaves the device), or {@code server} (backend
|
||||
* inference). Read by the frontend tool to choose its pipeline.
|
||||
*/
|
||||
private String executionMode = "auto";
|
||||
|
||||
/** Id of the installed model; blank means none installed. */
|
||||
private String activeModelId = "";
|
||||
|
||||
/** Optional override dir; blank uses {@code <configs>/models/form-detection}. */
|
||||
private String modelDir = "";
|
||||
|
||||
/**
|
||||
* Read-only dir of models baked into the image (e.g. the Docker server image pre-downloads
|
||||
* FFDNet-S here). On startup any {@code <catalogId>.onnx} found here is copied into the
|
||||
* writable model dir if not already present, and activated if no model is active - so the
|
||||
* feature works out-of-the-box. Blank (default) disables seeding.
|
||||
*/
|
||||
private String preinstalledModelDir = "";
|
||||
}
|
||||
|
||||
/**
|
||||
* Model Context Protocol (MCP) server configuration. All keys live under the top-level {@code
|
||||
* mcp.*} prefix. {@link #enabled} defaults to {@code false}: when off, no MCP beans are wired,
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
package stirling.software.common.service;
|
||||
|
||||
/**
|
||||
* Thread-scoped correlation id for one automation run — a single pipeline, policy, or AI-workflow
|
||||
* execution over its input file(s).
|
||||
*
|
||||
* <p>Automations dispatch each tool step as a separate internal loopback POST via {@link
|
||||
* InternalApiClient}. The orchestrator opens a run scope around its dispatch loop; {@code
|
||||
* InternalApiClient} reads {@link #current()} and stamps it on every sub-step request as {@link
|
||||
* #RUN_ID_HEADER}. The SaaS PAYG interceptor uses that header so all sub-steps of ONE run group
|
||||
* into a single charge, while two <em>separate</em> runs that happen to touch identical bytes stay
|
||||
* distinct charges (the old content+time-window grouping merged them).
|
||||
*
|
||||
* <p>Sub-steps dispatch synchronously on the orchestrator's own thread (loopback {@code
|
||||
* RestTemplate}), so this ThreadLocal is visible to {@code InternalApiClient}. The id then crosses
|
||||
* to the receiving request thread via the HTTP header — never via this ThreadLocal.
|
||||
*
|
||||
* <p>No-op when the id is absent (a standalone tool call): the interceptor treats a missing run id
|
||||
* as "its own charge", which is exactly what a one-off call should be.
|
||||
*/
|
||||
public final class AutomationRunContext {
|
||||
|
||||
/** Header carrying the run id on internal sub-step dispatches. */
|
||||
public static final String RUN_ID_HEADER = "X-Stirling-Run-Id";
|
||||
|
||||
private static final ThreadLocal<String> CURRENT = new ThreadLocal<>();
|
||||
|
||||
private AutomationRunContext() {}
|
||||
|
||||
/**
|
||||
* Opens a run scope on the current thread. Returns an {@link AutoCloseable} that restores the
|
||||
* previously-active id (nesting-safe) — use in try-with-resources around the dispatch loop.
|
||||
*/
|
||||
public static Scope open(String runId) {
|
||||
String previous = CURRENT.get();
|
||||
CURRENT.set(runId);
|
||||
return () -> {
|
||||
if (previous == null) {
|
||||
CURRENT.remove();
|
||||
} else {
|
||||
CURRENT.set(previous);
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
/** The run id active on this thread, or {@code null} when not inside a run scope. */
|
||||
public static String current() {
|
||||
return CURRENT.get();
|
||||
}
|
||||
|
||||
/** AutoCloseable whose {@link #close()} declares no checked exception. */
|
||||
public interface Scope extends AutoCloseable {
|
||||
@Override
|
||||
void close();
|
||||
}
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import java.nio.file.Files;
|
||||
import java.time.Duration;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
import org.slf4j.MDC;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.core.env.Environment;
|
||||
import org.springframework.core.io.FileSystemResource;
|
||||
@@ -60,6 +61,17 @@ public class InternalApiClient {
|
||||
*/
|
||||
public static final String AUTOMATION_HEADER = "X-Stirling-Automation";
|
||||
|
||||
/**
|
||||
* Header carrying the parent policy's name onto each sub-step dispatch, read from MDC key
|
||||
* {@link #POLICY_NAME_MDC_KEY} (set by the policy runner on the worker thread). Lets the audit
|
||||
* layer attribute a tool step to the policy that ran it, instead of showing it as a bare direct
|
||||
* call.
|
||||
*/
|
||||
public static final String POLICY_NAME_HEADER = "X-Stirling-Policy-Name";
|
||||
|
||||
/** MDC key the policy runner stamps with the running policy's name; forwarded as a header. */
|
||||
public static final String POLICY_NAME_MDC_KEY = "auditPolicyName";
|
||||
|
||||
private final ServletContext servletContext;
|
||||
private final UserServiceInterface userService;
|
||||
private final TempFileManager tempFileManager;
|
||||
@@ -111,6 +123,27 @@ public class InternalApiClient {
|
||||
// step inside a policy run must bill as AUTOMATION, not AI). Set unconditionally because
|
||||
// every caller of this dispatcher is an automation surface by design.
|
||||
headers.add(AUTOMATION_HEADER, "true");
|
||||
// Propagate the current automation run id (set by the orchestrator around its dispatch
|
||||
// loop) so the PAYG interceptor groups every sub-step of this one run into a single charge,
|
||||
// and never merges two separate runs that happen to touch identical bytes. Absent → the
|
||||
// receiving call is treated as standalone. See AutomationRunContext.
|
||||
String runId = AutomationRunContext.current();
|
||||
if (runId != null && !runId.isEmpty()) {
|
||||
headers.add(AutomationRunContext.RUN_ID_HEADER, runId);
|
||||
}
|
||||
|
||||
// Forward the parent policy name (set in MDC by the policy runner) so the audited sub-step
|
||||
// ties back to its policy. Single-line, length-capped: it becomes an HTTP header value.
|
||||
String policyName = MDC.get(POLICY_NAME_MDC_KEY);
|
||||
if (policyName != null && !policyName.isBlank()) {
|
||||
String safe = policyName.replaceAll("[\\r\\n]", " ").trim();
|
||||
if (safe.length() > 200) {
|
||||
safe = safe.substring(0, 200);
|
||||
}
|
||||
if (!safe.isEmpty()) {
|
||||
headers.add(POLICY_NAME_HEADER, safe);
|
||||
}
|
||||
}
|
||||
|
||||
// A no-file ai/tools call (e.g. create-pdf-from-html-agent) sends only string params, so
|
||||
// without this RestTemplate would use urlencoded instead of the multipart the controller
|
||||
|
||||
@@ -7,6 +7,7 @@ import java.time.format.DateTimeFormatter;
|
||||
import java.util.Calendar;
|
||||
|
||||
import org.apache.pdfbox.pdmodel.PDDocument;
|
||||
import org.apache.pdfbox.pdmodel.PDDocumentInformation;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.beans.factory.annotation.Qualifier;
|
||||
import org.springframework.stereotype.Service;
|
||||
@@ -17,6 +18,9 @@ import stirling.software.common.model.PdfMetadata;
|
||||
@Service
|
||||
public class PdfMetadataService {
|
||||
|
||||
/** ({@code {labels}}). Written by the classify-and-label tool. */
|
||||
public static final String CLASSIFICATION_KEY = "StirlingPDFClassification";
|
||||
|
||||
private final ApplicationProperties applicationProperties;
|
||||
private final String stirlingPDFLabel;
|
||||
private final UserServiceInterface userService;
|
||||
@@ -177,4 +181,14 @@ public class PdfMetadataService {
|
||||
}
|
||||
pdf.getDocumentInformation().setAuthor(author);
|
||||
}
|
||||
|
||||
/**
|
||||
* Write the document classifier's JSON result into the custom Info-dictionary field {@link
|
||||
* #CLASSIFICATION_KEY}, leaving all other metadata untouched.
|
||||
*/
|
||||
public void setClassificationMetadata(PDDocument pdf, String classificationJson) {
|
||||
PDDocumentInformation info = pdf.getDocumentInformation();
|
||||
info.setCustomMetadataValue(CLASSIFICATION_KEY, classificationJson);
|
||||
pdf.setDocumentInformation(info);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -144,8 +144,10 @@ public class TempFileCleanupService {
|
||||
int directoriesDeletedCount = 0;
|
||||
for (Path directory : registry.getTempDirectories()) {
|
||||
try {
|
||||
if (Files.exists(directory)) {
|
||||
if (Files.exists(directory)
|
||||
&& shouldDeleteRegisteredDirectory(directory, maxAgeMillis)) {
|
||||
GeneralUtils.deleteDirectory(directory);
|
||||
registry.unregisterDirectory(directory);
|
||||
directoriesDeletedCount++;
|
||||
log.debug("Cleaned up temporary directory: {}", directory);
|
||||
}
|
||||
@@ -275,6 +277,21 @@ public class TempFileCleanupService {
|
||||
return totalDeletedCount.get();
|
||||
}
|
||||
|
||||
private boolean shouldDeleteRegisteredDirectory(Path directory, long maxAgeMillis) {
|
||||
if (maxAgeMillis <= 0) {
|
||||
return true;
|
||||
}
|
||||
|
||||
try {
|
||||
long currentTime = System.currentTimeMillis();
|
||||
long lastModified = Files.getLastModifiedTime(directory).toMillis();
|
||||
return (currentTime - lastModified) > maxAgeMillis;
|
||||
} catch (IOException e) {
|
||||
log.debug("Could not check directory age, skipping cleanup: {}", directory, e);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/** Get the system temp directory path based on configuration or system property. */
|
||||
private Path getSystemTempPath() {
|
||||
String systemTempDir =
|
||||
|
||||
@@ -944,6 +944,81 @@ public class FormUtils {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Create new AcroForm fields from a list of definitions (used by Auto Form Detection). Reuses
|
||||
* the same field-creation and appearance logic as the rest of this class, and creates the
|
||||
* AcroForm (with a Helvetica default resource) when the document has none. Field names are made
|
||||
* unique against any existing fields.
|
||||
*/
|
||||
public void addFields(PDDocument document, List<NewFormFieldDefinition> definitions)
|
||||
throws IOException {
|
||||
if (document == null || definitions == null || definitions.isEmpty()) {
|
||||
return;
|
||||
}
|
||||
PDDocumentCatalog documentCatalog = document.getDocumentCatalog();
|
||||
PDAcroForm acroForm = documentCatalog.getAcroForm();
|
||||
if (acroForm == null) {
|
||||
acroForm = new PDAcroForm(document);
|
||||
PDResources dr = new PDResources();
|
||||
dr.put(COSName.getPDFName("Helv"), new PDType1Font(Standard14Fonts.FontName.HELVETICA));
|
||||
acroForm.setDefaultResources(dr);
|
||||
acroForm.setNeedAppearances(true);
|
||||
documentCatalog.setAcroForm(acroForm);
|
||||
}
|
||||
|
||||
Set<String> existingNames = new java.util.HashSet<>();
|
||||
for (PDField field : acroForm.getFieldTree()) {
|
||||
if (field.getPartialName() != null) {
|
||||
existingNames.add(field.getPartialName());
|
||||
}
|
||||
}
|
||||
|
||||
int pageCount = document.getNumberOfPages();
|
||||
for (NewFormFieldDefinition definition : definitions) {
|
||||
Integer pageIndex = definition.pageIndex();
|
||||
if (pageIndex == null
|
||||
|| pageIndex < 0
|
||||
|| pageIndex >= pageCount
|
||||
|| definition.x() == null
|
||||
|| definition.y() == null
|
||||
|| definition.width() == null
|
||||
|| definition.height() == null) {
|
||||
continue;
|
||||
}
|
||||
PDPage page = document.getPage(pageIndex);
|
||||
PDRectangle rectangle =
|
||||
new PDRectangle(
|
||||
definition.x(),
|
||||
definition.y(),
|
||||
definition.width(),
|
||||
definition.height());
|
||||
FormFieldTypeSupport handler = FormFieldTypeSupport.forTypeName(definition.type());
|
||||
if (handler == null || handler.doesNotsupportsDefinitionCreation()) {
|
||||
handler = FormFieldTypeSupport.TEXT;
|
||||
}
|
||||
String baseName =
|
||||
(definition.name() != null && !definition.name().isBlank())
|
||||
? definition.name()
|
||||
: handler.typeName() + "_" + (pageIndex + 1);
|
||||
String uniqueName = generateUniqueFieldName(baseName, existingNames);
|
||||
existingNames.add(uniqueName);
|
||||
try {
|
||||
createNewField(
|
||||
handler,
|
||||
acroForm,
|
||||
page,
|
||||
rectangle,
|
||||
uniqueName,
|
||||
definition,
|
||||
definition.options());
|
||||
} catch (Exception e) {
|
||||
log.warn("Failed to create detected field '{}': {}", uniqueName, e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
ensureAppearances(acroForm);
|
||||
}
|
||||
|
||||
public String filterSingleChoiceSelection(
|
||||
String selection, List<String> allowedOptions, String fieldName) {
|
||||
if (selection == null || selection.trim().isEmpty()) return null;
|
||||
|
||||
@@ -57,6 +57,16 @@ public class RequestUriUtils {
|
||||
return true;
|
||||
}
|
||||
|
||||
// Admin portal SPA shell (mounted at /processor — must match the frontend
|
||||
// PORTAL_BASENAME). Served publicly like the editor root so a direct nav /
|
||||
// refresh to /processor loads the app (the JWT lives in localStorage, not a
|
||||
// cookie, so the server can't authenticate the navigation itself). The
|
||||
// portal gates access via its own auth gate + RequirePortalAccess, and its
|
||||
// data APIs stay protected, so serving the shell pre-auth is safe.
|
||||
if (normalizedUri.equals("/processor") || normalizedUri.startsWith("/processor/")) {
|
||||
return true;
|
||||
}
|
||||
|
||||
// Treat common static file extensions as static resources
|
||||
return normalizedUri.endsWith(".svg")
|
||||
|| normalizedUri.endsWith(".png")
|
||||
|
||||
@@ -155,6 +155,7 @@ public class TempFileManager {
|
||||
if (directory != null && Files.isDirectory(directory)) {
|
||||
try {
|
||||
GeneralUtils.deleteDirectory(directory);
|
||||
registry.unregisterDirectory(directory);
|
||||
log.debug("Deleted temp directory: {}", directory.toString());
|
||||
} catch (IOException e) {
|
||||
log.warn("Failed to delete temp directory: {}", directory.toString(), e);
|
||||
|
||||
@@ -85,6 +85,18 @@ public class TempFileRegistry {
|
||||
return directory;
|
||||
}
|
||||
|
||||
/**
|
||||
* Unregister a temporary directory from the registry.
|
||||
*
|
||||
* @param directory The directory to unregister
|
||||
*/
|
||||
public void unregisterDirectory(Path directory) {
|
||||
if (directory != null) {
|
||||
tempDirectories.remove(directory);
|
||||
log.debug("Unregistered temp directory: {}", directory.toString());
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Register a third-party temporary file that requires special handling.
|
||||
*
|
||||
|
||||
@@ -176,11 +176,13 @@ class TempFileCleanupServiceMoreTest {
|
||||
class ScheduledCleanup {
|
||||
|
||||
@Test
|
||||
@DisplayName("deletes registered temp directories and reports counts")
|
||||
@DisplayName("deletes stale registered temp directories and reports counts")
|
||||
void deletesRegisteredDirectories() throws IOException {
|
||||
when(tempFileManager.cleanupOldTempFiles(anyLong())).thenReturn(2);
|
||||
Path regDir = Files.createDirectories(tempDir.resolve("registeredDir"));
|
||||
Files.createFile(regDir.resolve("inside.txt"));
|
||||
Files.setLastModifiedTime(
|
||||
regDir, FileTime.fromMillis(System.currentTimeMillis() - 2L * 60 * 60 * 1000));
|
||||
Set<Path> dirs = new HashSet<>();
|
||||
dirs.add(regDir);
|
||||
when(registry.getTempDirectories()).thenReturn(dirs);
|
||||
@@ -193,6 +195,22 @@ class TempFileCleanupServiceMoreTest {
|
||||
verify(tempFileManager).cleanupOldTempFiles(anyLong());
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("keeps a fresh registered temp directory")
|
||||
void keepsFreshRegisteredDirectory() throws IOException {
|
||||
when(tempFileManager.cleanupOldTempFiles(anyLong())).thenReturn(0);
|
||||
Path regDir = Files.createDirectories(tempDir.resolve("freshRegisteredDir"));
|
||||
Files.createFile(regDir.resolve("inside.txt"));
|
||||
Set<Path> dirs = new HashSet<>();
|
||||
dirs.add(regDir);
|
||||
when(registry.getTempDirectories()).thenReturn(dirs);
|
||||
lenient().when(registry.contains(any(File.class))).thenReturn(false);
|
||||
|
||||
withIsolatedUserHome(cleanupService::scheduledCleanup);
|
||||
|
||||
assertThat(Files.exists(regDir)).isTrue();
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("skips a registered directory that no longer exists")
|
||||
void skipsMissingRegisteredDirectory() {
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
package stirling.software.common.util;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertNotNull;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
import org.apache.pdfbox.pdmodel.PDDocument;
|
||||
import org.apache.pdfbox.pdmodel.PDPage;
|
||||
import org.apache.pdfbox.pdmodel.common.PDRectangle;
|
||||
import org.apache.pdfbox.pdmodel.interactive.form.PDAcroForm;
|
||||
import org.apache.pdfbox.pdmodel.interactive.form.PDCheckBox;
|
||||
import org.apache.pdfbox.pdmodel.interactive.form.PDField;
|
||||
import org.apache.pdfbox.pdmodel.interactive.form.PDTextField;
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import stirling.software.common.util.FormUtils.NewFormFieldDefinition;
|
||||
|
||||
class FormUtilsAddFieldsTest {
|
||||
|
||||
private NewFormFieldDefinition def(String type, int page, float x, float y, float w, float h) {
|
||||
return new NewFormFieldDefinition(
|
||||
null, null, type, page, x, y, w, h, false, null, null, null, null);
|
||||
}
|
||||
|
||||
@Test
|
||||
void createsTextAndCheckboxFieldsOnPagelessDocument() throws IOException {
|
||||
try (PDDocument doc = new PDDocument()) {
|
||||
doc.addPage(new PDPage(new PDRectangle(612, 792)));
|
||||
|
||||
FormUtils.addFields(
|
||||
doc,
|
||||
List.of(
|
||||
def("text", 0, 100f, 700f, 200f, 20f),
|
||||
def("checkbox", 0, 100f, 650f, 15f, 15f)));
|
||||
|
||||
PDAcroForm form = doc.getDocumentCatalog().getAcroForm();
|
||||
assertNotNull(form, "AcroForm should be created");
|
||||
|
||||
List<PDField> fields = new ArrayList<>();
|
||||
form.getFieldTree().forEach(fields::add);
|
||||
assertEquals(2, fields.size());
|
||||
|
||||
boolean hasText = fields.stream().anyMatch(f -> f instanceof PDTextField);
|
||||
boolean hasCheckbox = fields.stream().anyMatch(f -> f instanceof PDCheckBox);
|
||||
assertTrue(hasText, "expected a text field");
|
||||
assertTrue(hasCheckbox, "expected a checkbox field");
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void skipsOutOfRangePageAndKeepsNamesUnique() throws IOException {
|
||||
try (PDDocument doc = new PDDocument()) {
|
||||
doc.addPage(new PDPage(new PDRectangle(612, 792)));
|
||||
|
||||
FormUtils.addFields(
|
||||
doc,
|
||||
List.of(
|
||||
def("text", 0, 10f, 10f, 50f, 12f),
|
||||
def("text", 0, 10f, 40f, 50f, 12f),
|
||||
def("text", 5, 10f, 70f, 50f, 12f))); // page 5 out of range -> skipped
|
||||
|
||||
PDAcroForm form = doc.getDocumentCatalog().getAcroForm();
|
||||
List<PDField> fields = new ArrayList<>();
|
||||
form.getFieldTree().forEach(fields::add);
|
||||
assertEquals(2, fields.size());
|
||||
|
||||
long distinctNames = fields.stream().map(PDField::getPartialName).distinct().count();
|
||||
assertEquals(2, distinctNames, "field names must be unique");
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
void noOpOnEmptyDefinitions() throws IOException {
|
||||
try (PDDocument doc = new PDDocument()) {
|
||||
doc.addPage(new PDPage(new PDRectangle(612, 792)));
|
||||
FormUtils.addFields(doc, List.of());
|
||||
// no AcroForm forced into existence when there is nothing to add
|
||||
assertEquals(null, doc.getDocumentCatalog().getAcroForm());
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -73,6 +73,14 @@ class RequestUriUtilsTest {
|
||||
assertTrue(RequestUriUtils.isStaticResource("/mobile-scanner"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void testIsStaticResource_portalShell() {
|
||||
// The admin portal SPA shell (/processor) is served pre-auth so it's directly navigable.
|
||||
assertTrue(RequestUriUtils.isStaticResource("/processor"));
|
||||
assertTrue(RequestUriUtils.isStaticResource("/processor/users"));
|
||||
assertTrue(RequestUriUtils.isStaticResource("/app", "/app/processor"));
|
||||
}
|
||||
|
||||
// --- isFrontendRoute tests ---
|
||||
|
||||
@Test
|
||||
|
||||
@@ -175,6 +175,14 @@ springBoot {
|
||||
// Frontend build tasks - only enabled with -PbuildWithFrontend=true
|
||||
def buildWithFrontend = project.hasProperty('buildWithFrontend') && project.property('buildWithFrontend') == 'true'
|
||||
def buildPrototypes = project.hasProperty('prototypesMode') && project.property('prototypesMode') == 'true'
|
||||
// The admin portal ships as a lazy route inside the editor bundle (see
|
||||
// proprietary/routes/adminRouteExtensions). -PbuildWithPortal=true includes that
|
||||
// chunk via VITE_INCLUDE_PORTAL on the editor build; the deploy GHA sets it when
|
||||
// the portal or AI layers change. Building the portal implies building the editor.
|
||||
def buildWithPortal = project.hasProperty('buildWithPortal') && project.property('buildWithPortal') == 'true'
|
||||
if (buildWithPortal) {
|
||||
buildWithFrontend = true
|
||||
}
|
||||
// Workspace root holds package.json and node_modules (shared across editor /
|
||||
// future portal). Editor-specific paths (src, public, dist, tauri) live one
|
||||
// level deeper under frontend/editor/.
|
||||
@@ -297,9 +305,11 @@ tasks.register('npmBuild', Exec) {
|
||||
// Override VITE_API_BASE_URL to use relative paths for production builds
|
||||
// This ensures JARs work regardless of how they're deployed (direct, proxied, etc.)
|
||||
environment 'VITE_API_BASE_URL', '/'
|
||||
// Include the admin portal's lazy route/chunk in the editor build when requested.
|
||||
environment 'VITE_INCLUDE_PORTAL', (buildWithPortal ? 'true' : 'false')
|
||||
|
||||
doFirst {
|
||||
println "Building editor frontend application for production (mode=${frontendMode}, VITE_API_BASE_URL=/)"
|
||||
println "Building editor frontend application for production (mode=${frontendMode}, VITE_API_BASE_URL=/, portal=${buildWithPortal})"
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -10,6 +10,7 @@ import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
import java.util.Map.Entry;
|
||||
import java.util.UUID;
|
||||
|
||||
import org.springframework.core.io.FileSystemResource;
|
||||
import org.springframework.core.io.Resource;
|
||||
@@ -27,6 +28,7 @@ import stirling.software.SPDF.model.PipelineConfig;
|
||||
import stirling.software.SPDF.model.PipelineOperation;
|
||||
import stirling.software.SPDF.model.PipelineResult;
|
||||
import stirling.software.SPDF.service.ApiDocService;
|
||||
import stirling.software.common.service.AutomationRunContext;
|
||||
import stirling.software.common.service.InternalApiClient;
|
||||
import stirling.software.common.util.TempFileManager;
|
||||
import stirling.software.common.util.ZipExtractionUtils;
|
||||
@@ -71,6 +73,17 @@ public class PipelineProcessor {
|
||||
|
||||
PipelineResult runPipelineAgainstFiles(List<Resource> outputFiles, PipelineConfig config)
|
||||
throws Exception {
|
||||
// One pipeline execution = one automation run. Scope a run id so every tool sub-step
|
||||
// dispatched via InternalApiClient groups into a single charge on the SaaS billing side
|
||||
// (see AutomationRunContext); pipeline steps run synchronously on this thread.
|
||||
try (AutomationRunContext.Scope ignored =
|
||||
AutomationRunContext.open(UUID.randomUUID().toString())) {
|
||||
return runPipelineAgainstFilesInternal(outputFiles, config);
|
||||
}
|
||||
}
|
||||
|
||||
private PipelineResult runPipelineAgainstFilesInternal(
|
||||
List<Resource> outputFiles, PipelineConfig config) throws Exception {
|
||||
PipelineResult result = new PipelineResult();
|
||||
|
||||
ByteArrayOutputStream logStream = new ByteArrayOutputStream();
|
||||
|
||||
@@ -305,6 +305,23 @@ public class GetInfoOnPDF {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Info-dictionary keys exposed above via typed getters; any other key in the dictionary is
|
||||
* surfaced as custom metadata (e.g. the classification policy's StirlingPDFClassification
|
||||
* entry).
|
||||
*/
|
||||
private static final java.util.Set<String> STANDARD_INFO_KEYS =
|
||||
java.util.Set.of(
|
||||
"Title",
|
||||
"Author",
|
||||
"Subject",
|
||||
"Keywords",
|
||||
"Producer",
|
||||
"Creator",
|
||||
"CreationDate",
|
||||
"ModDate",
|
||||
"Trapped");
|
||||
|
||||
private static ObjectNode extractMetadata(PDDocument document) {
|
||||
ObjectNode metadata = objectMapper.createObjectNode();
|
||||
|
||||
@@ -335,6 +352,18 @@ public class GetInfoOnPDF {
|
||||
if (modificationDate != null) {
|
||||
metadata.put("ModificationDate", modificationDate);
|
||||
}
|
||||
|
||||
// Surface custom Info-dictionary entries (anything beyond the
|
||||
// standard fields above) — e.g. StirlingPDFClassification
|
||||
for (String key : info.getMetadataKeys()) {
|
||||
if (STANDARD_INFO_KEYS.contains(key)) {
|
||||
continue;
|
||||
}
|
||||
String value = info.getCustomMetadataValue(key);
|
||||
if (value != null && !value.isBlank()) {
|
||||
metadata.put(key, value);
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (Exception e) {
|
||||
log.error("Error extracting metadata: {}", e.getMessage());
|
||||
|
||||
@@ -136,11 +136,17 @@ public class RedactController {
|
||||
+ "Users can provide text patterns to redact, with options for regex and whole word matching. "
|
||||
+ "Input:PDF Output:PDF Type:SISO")
|
||||
public ResponseEntity<Resource> redactPdf(@ModelAttribute RedactPdfRequest request) {
|
||||
String[] listOfText = request.getListOfText().split("\n");
|
||||
String rawListOfText = request.getListOfText();
|
||||
boolean useRegex = Boolean.TRUE.equals(request.getUseRegex());
|
||||
boolean wholeWordSearchBool = Boolean.TRUE.equals(request.getWholeWordSearch());
|
||||
|
||||
if (listOfText.length == 0 || (listOfText.length == 1 && listOfText[0].trim().isEmpty())) {
|
||||
if (rawListOfText == null || rawListOfText.trim().isEmpty()) {
|
||||
throw ExceptionUtils.createIllegalArgumentException(
|
||||
"error.redaction.no.patterns", "No text patterns provided for redaction");
|
||||
}
|
||||
|
||||
String[] listOfText = rawListOfText.split("\n");
|
||||
if (listOfText.length == 1 && listOfText[0].trim().isEmpty()) {
|
||||
throw ExceptionUtils.createIllegalArgumentException(
|
||||
"error.redaction.no.patterns", "No text patterns provided for redaction");
|
||||
}
|
||||
|
||||
@@ -367,6 +367,13 @@ aiEngine:
|
||||
url: http://localhost:5001 # URL of the Python AI engine
|
||||
timeoutSeconds: 120 # Timeout in seconds for AI engine requests
|
||||
|
||||
formDetection:
|
||||
enabled: true # Master on/off switch for the Auto Form Detection feature
|
||||
executionMode: auto # Where detection runs: 'auto' (browser first, server fallback), 'browser' (in-browser WASM only, PDF never leaves the device), or 'server' (backend inference)
|
||||
activeModelId: "" # Id of the installed Auto Form Detection model (set automatically after an admin installs one)
|
||||
modelDir: "" # Optional override directory for downloaded .onnx models; blank uses <configs>/models/form-detection
|
||||
preinstalledModelDir: "" # Read-only dir of models baked into the image (Docker pre-downloads FFDNet-S here); seeded into the model dir on startup. Blank disables.
|
||||
|
||||
policies:
|
||||
# Folder automations can read from and write to the directories you allow here, so treat this as a
|
||||
# security boundary. Leave allowedFolderRoots empty (default) to disable folder sources/outputs
|
||||
|
||||
|
Before Width: | Height: | Size: 6.4 KiB After Width: | Height: | Size: 6.4 KiB |
@@ -1,18 +1 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" version="1.1" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 24 24">
|
||||
<symbol id="icon-redact-auto" viewBox="0 0 24 24"> <g id="Layer_2" data-name="Layer 2">
|
||||
<g id="Layer_1-2" data-name="Layer 1">
|
||||
<rect width="24" height="24" style="fill: none"/>
|
||||
<g>
|
||||
<path d="M17.541,15.64258a.91793.91793,0,0,1,.55469-.18555h1.1084a.91586.91586,0,0,1,.55469.18555,1.30889,1.30889,0,0,1,.40429.499,1.57206,1.57206,0,0,1,.15039.68457v5.47754H19.2041V20.21094H18.0957v2.09277H16.9873V16.82617a1.55843,1.55843,0,0,1,.15039-.68457A1.2979,1.2979,0,0,1,17.541,15.64258Zm1.66309,1.10547H18.0957v2.17187h1.1084Z" style="fill: currentColor"/>
|
||||
<path d="M5.68653,22.30351a2.00588,2.00588,0,0,1-2-2v-16A1.92585,1.92585,0,0,1,4.274,2.891a1.92585,1.92585,0,0,1,1.4125-.5875h8l6,6v5.66931h-2V9.30351h-5v-5h-7v16h9.74021v2Z" style="fill: currentColor"/>
|
||||
<rect x="7.69809" y="10.43189" width="4.33778" height="0.79501" style="fill: currentColor"/>
|
||||
<rect x="7.69809" y="12.16889" width="7.31192" height="1.21288" style="fill: currentColor"/>
|
||||
<rect x="7.69809" y="17.14555" width="7.31192" height="1.21288" style="fill: currentColor"/>
|
||||
<rect x="7.69809" y="14.32375" width="7.31192" height="0.57517" style="fill: currentColor"/>
|
||||
<rect x="7.69809" y="15.79848" width="5.25578" height="0.4475" style="fill: currentColor"/>
|
||||
</g>
|
||||
</g>
|
||||
</g>
|
||||
</symbol>
|
||||
</svg>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" viewBox="0 0 24 24"><symbol id="icon-redact-auto" viewBox="0 0 24 24"><g id="Layer_2" data-name="Layer 2"><g id="Layer_1-2" data-name="Layer 1"><rect width="24" height="24" style="fill:none"/><g><path d="M17.541,15.64258a.91793.91793,0,0,1,.55469-.18555h1.1084a.91586.91586,0,0,1,.55469.18555,1.30889,1.30889,0,0,1,.40429.499,1.57206,1.57206,0,0,1,.15039.68457v5.47754H19.2041V20.21094H18.0957v2.09277H16.9873V16.82617a1.55843,1.55843,0,0,1,.15039-.68457A1.2979,1.2979,0,0,1,17.541,15.64258Zm1.66309,1.10547H18.0957v2.17187h1.1084Z" style="fill:currentColor"/><path d="M5.68653,22.30351a2.00588,2.00588,0,0,1-2-2v-16A1.92585,1.92585,0,0,1,4.274,2.891a1.92585,1.92585,0,0,1,1.4125-.5875h8l6,6v5.66931h-2V9.30351h-5v-5h-7v16h9.74021v2Z" style="fill:currentColor"/><rect width="4.338" height=".795" x="7.698" y="10.432" style="fill:currentColor"/><rect width="7.312" height="1.213" x="7.698" y="12.169" style="fill:currentColor"/><rect width="7.312" height="1.213" x="7.698" y="17.146" style="fill:currentColor"/><rect width="7.312" height=".575" x="7.698" y="14.324" style="fill:currentColor"/><rect width="5.256" height=".448" x="7.698" y="15.798" style="fill:currentColor"/></g></g></g></symbol></svg>
|
||||
|
Before Width: | Height: | Size: 1.5 KiB After Width: | Height: | Size: 1.3 KiB |
@@ -1,13 +1 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!-- Uploaded to: SVG Repo, www.svgrepo.com, Generator: SVG Repo Mixer Tools -->
|
||||
<svg width="800px" height="800px" viewBox="0 0 512 512" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
|
||||
<title>rename</title>
|
||||
<symbol id="icon-rename" viewBox="0 0 512 512">
|
||||
<g id="Page-1" stroke="none" stroke-width="1" fill="none" fill-rule="evenodd">
|
||||
<g id="Combined-Shape" fill="currentColor" transform="translate(42.666667, 64.000000)">
|
||||
<path d="M362.666667,1.42108547e-14 L362.666667,21.3333333 L320,21.333 L320,362.666 L362.666667,362.666667 L362.666667,384 L320,383.999 L320,384 L298.666667,384 L298.666,383.999 L256,384 L256,362.666667 L298.666,362.666 L298.666,21.333 L256,21.3333333 L256,1.42108547e-14 L362.666667,1.42108547e-14 Z M426.666667,64 L426.666667,320 L341.333333,320 L341.333333,277.333333 L384,277.333333 L384,106.666667 L341.333333,106.666667 L341.333333,64 L426.666667,64 Z M277.333333,64 L277.333333,320 L3.55271368e-14,320 L3.55271368e-14,64 L277.333333,64 Z M179.2,89.6 L149.333333,89.6 L149.333333,234.666667 C149.333333,248 148.5,256.333333 147.875,264.354167 L147.792993,265.422171 L147.792993,265.422171 L147.714003,266.48894 C147.417695,270.579012 147.2,274.696296 147.2,279.466667 L147.2,279.466667 L177.066667,279.466667 L177.066667,260.266667 C184.941497,273.926888 199.708077,282.130544 215.466667,281.6 C229.540046,281.805757 242.921593,275.508559 251.733333,264.533333 C263.162478,248.989677 269.832496,230.461848 270.933333,211.2 C270.933333,170.666667 249.6,142.933333 217.6,142.933333 C202.507405,142.999748 188.308689,150.099106 179.2,162.133333 L179.2,162.133333 L179.2,89.6 Z M119.466667,162.133333 C107.961824,149.843793 91.4322333,143.546807 74.6666667,145.066667 C57.6785115,144.485924 40.8138255,148.15216 25.6,155.733333 L25.6,155.733333 L34.1333333,177.066667 C45.3979052,171.147831 57.7246848,167.522308 70.4,166.4 C78.5613135,165.511423 86.6853595,168.371259 92.4903835,174.176283 C98.2954074,179.981307 101.155244,188.105353 100.266667,196.266667 L100.266667,196.266667 L100.266667,198.4 L78.9333333,198.4 C65.8181975,197.679203 52.705771,199.864608 40.5333333,204.8 C26.2806563,210.950309 17.6507691,225.621117 19.2,241.066667 C19.0625857,252.057651 23.6679763,262.574827 31.8381493,269.927982 C40.0083223,277.281138 50.9508304,280.757072 61.8666667,279.466667 C77.2795695,280.291768 92.2192911,274.001359 102.4,262.4 L102.4,262.4 L102.4,277.333333 L130.133333,277.333333 C128.292479,266.054406 127.577851,254.620365 128,243.2 L128,243.2 L128,204.8 C129.999138,190.023932 126.995128,175.003882 119.466667,162.133333 Z M98.1333333,213.333333 L98.1333333,238.933333 C92.082572,249.988391 80.836024,257.218314 68.2666667,258.133333 C63.0655139,258.520242 57.9538681,256.621996 54.2659359,252.934064 C50.5780036,249.246132 48.6797582,244.134486 49.0666667,238.933333 C49.0666667,224 59.7333333,215.466667 85.3333333,213.333333 L85.3333333,213.333333 L98.1333333,213.333333 Z M209.066667,166.4 C226.133333,166.4 238.933333,183.466667 238.933333,211.2 C238.933333,238.933333 228.266667,256 211.2,256 C197.298049,255.69869 184.825037,247.383349 179.2,234.666667 L179.2,234.666667 L179.2,187.733333 C185.154203,176.240507 196.263981,168.304951 209.066667,166.4 Z">
|
||||
</path>
|
||||
</g>
|
||||
</g>
|
||||
</symbol>
|
||||
</svg>
|
||||
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="800" height="800" version="1.1" viewBox="0 0 512 512"><title>rename</title><symbol id="icon-rename" viewBox="0 0 512 512"><g id="Page-1" fill="none" fill-rule="evenodd" stroke="none" stroke-width="1"><g id="Combined-Shape" fill="currentColor"><path d="M362.666667,1.42108547e-14 L362.666667,21.3333333 L320,21.333 L320,362.666 L362.666667,362.666667 L362.666667,384 L320,383.999 L320,384 L298.666667,384 L298.666,383.999 L256,384 L256,362.666667 L298.666,362.666 L298.666,21.333 L256,21.3333333 L256,1.42108547e-14 L362.666667,1.42108547e-14 Z M426.666667,64 L426.666667,320 L341.333333,320 L341.333333,277.333333 L384,277.333333 L384,106.666667 L341.333333,106.666667 L341.333333,64 L426.666667,64 Z M277.333333,64 L277.333333,320 L3.55271368e-14,320 L3.55271368e-14,64 L277.333333,64 Z M179.2,89.6 L149.333333,89.6 L149.333333,234.666667 C149.333333,248 148.5,256.333333 147.875,264.354167 L147.792993,265.422171 L147.792993,265.422171 L147.714003,266.48894 C147.417695,270.579012 147.2,274.696296 147.2,279.466667 L147.2,279.466667 L177.066667,279.466667 L177.066667,260.266667 C184.941497,273.926888 199.708077,282.130544 215.466667,281.6 C229.540046,281.805757 242.921593,275.508559 251.733333,264.533333 C263.162478,248.989677 269.832496,230.461848 270.933333,211.2 C270.933333,170.666667 249.6,142.933333 217.6,142.933333 C202.507405,142.999748 188.308689,150.099106 179.2,162.133333 L179.2,162.133333 L179.2,89.6 Z M119.466667,162.133333 C107.961824,149.843793 91.4322333,143.546807 74.6666667,145.066667 C57.6785115,144.485924 40.8138255,148.15216 25.6,155.733333 L25.6,155.733333 L34.1333333,177.066667 C45.3979052,171.147831 57.7246848,167.522308 70.4,166.4 C78.5613135,165.511423 86.6853595,168.371259 92.4903835,174.176283 C98.2954074,179.981307 101.155244,188.105353 100.266667,196.266667 L100.266667,196.266667 L100.266667,198.4 L78.9333333,198.4 C65.8181975,197.679203 52.705771,199.864608 40.5333333,204.8 C26.2806563,210.950309 17.6507691,225.621117 19.2,241.066667 C19.0625857,252.057651 23.6679763,262.574827 31.8381493,269.927982 C40.0083223,277.281138 50.9508304,280.757072 61.8666667,279.466667 C77.2795695,280.291768 92.2192911,274.001359 102.4,262.4 L102.4,262.4 L102.4,277.333333 L130.133333,277.333333 C128.292479,266.054406 127.577851,254.620365 128,243.2 L128,243.2 L128,204.8 C129.999138,190.023932 126.995128,175.003882 119.466667,162.133333 Z M98.1333333,213.333333 L98.1333333,238.933333 C92.082572,249.988391 80.836024,257.218314 68.2666667,258.133333 C63.0655139,258.520242 57.9538681,256.621996 54.2659359,252.934064 C50.5780036,249.246132 48.6797582,244.134486 49.0666667,238.933333 C49.0666667,224 59.7333333,215.466667 85.3333333,213.333333 L85.3333333,213.333333 L98.1333333,213.333333 Z M209.066667,166.4 C226.133333,166.4 238.933333,183.466667 238.933333,211.2 C238.933333,238.933333 228.266667,256 211.2,256 C197.298049,255.69869 184.825037,247.383349 179.2,234.666667 L179.2,234.666667 L179.2,187.733333 C185.154203,176.240507 196.263981,168.304951 209.066667,166.4 Z" transform="translate(42.666667, 64.000000)"/></g></g></symbol></svg>
|
||||
|
Before Width: | Height: | Size: 3.3 KiB After Width: | Height: | Size: 3.1 KiB |
|
Before Width: | Height: | Size: 6.1 KiB After Width: | Height: | Size: 5.3 KiB |
@@ -299,6 +299,18 @@ class RedactControllerMoreTest {
|
||||
verify(pdfDocumentFactory, never()).load(any(MultipartFile.class));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("null listOfText throws an illegal-argument error before any load")
|
||||
void nullPatternsThrows() throws Exception {
|
||||
RedactPdfRequest request = new RedactPdfRequest();
|
||||
request.setFileInput(pdfFile(new byte[] {1, 2, 3}));
|
||||
request.setListOfText(null);
|
||||
|
||||
assertThatThrownBy(() -> controller.redactPdf(request))
|
||||
.isInstanceOf(RuntimeException.class);
|
||||
verify(pdfDocumentFactory, never()).load(any(MultipartFile.class));
|
||||
}
|
||||
|
||||
@Test
|
||||
@DisplayName("null file input is reported as a failure")
|
||||
void nullFileThrows() {
|
||||
|
||||
@@ -3,6 +3,14 @@ repositories {
|
||||
maven { url = "https://build.shibboleth.net/maven/releases" }
|
||||
}
|
||||
|
||||
ext {
|
||||
// CPU build; self-extracting natives for linux/win/mac x64+arm64. Keep major.minor aligned
|
||||
// with the frontend onnxruntime-web pin so the same .onnx runs identically on both paths.
|
||||
// Must be >=1.21 to load opset-22 models: the FFDNet exports are stamped ai.onnx opset 22,
|
||||
// which 1.19/1.20 hard-reject (ORT_FAIL "support is till opset 21"). 1.26.0 matches the
|
||||
// onnxruntime build the upstream CommonForms inference reference runs on.
|
||||
onnxruntimeVersion = '1.26.0'
|
||||
}
|
||||
bootRun {
|
||||
enabled = false
|
||||
}
|
||||
@@ -66,6 +74,18 @@ dependencies {
|
||||
|
||||
implementation "com.google.code.gson:gson:${gsonVersion}"
|
||||
|
||||
// ONNX Runtime (Java) for SERVER-SIDE Auto Form Detection inference.
|
||||
// Compiled against everywhere so the module always builds, but the multi-platform native
|
||||
// (~42MB) is only BUNDLED when explicitly requested via -PbundleOnnxRuntime=true - i.e. the
|
||||
// Docker server image (which then slims it to one Linux arch, ~8MB). Desktop/local/core builds
|
||||
// ship WITHOUT it: server-side detection cleanly reports "unavailable" and the in-browser
|
||||
// (onnxruntime-web) engine handles detection instead. Tests always get it.
|
||||
compileOnly "com.microsoft.onnxruntime:onnxruntime:$onnxruntimeVersion"
|
||||
testImplementation "com.microsoft.onnxruntime:onnxruntime:$onnxruntimeVersion"
|
||||
if ((project.findProperty('bundleOnnxRuntime') ?: 'false').toString().toBoolean()) {
|
||||
runtimeOnly "com.microsoft.onnxruntime:onnxruntime:$onnxruntimeVersion"
|
||||
}
|
||||
|
||||
api 'io.micrometer:micrometer-registry-prometheus'
|
||||
|
||||
api "io.jsonwebtoken:jjwt-api:${jwtVersion}"
|
||||
|
||||
@@ -4,7 +4,9 @@ import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
|
||||
import stirling.software.proprietary.access.service.DefaultPrincipalResolver;
|
||||
import stirling.software.proprietary.access.service.DefaultTeamLeadLookup;
|
||||
import stirling.software.proprietary.access.service.PrincipalResolver;
|
||||
import stirling.software.proprietary.access.service.TeamLeadLookup;
|
||||
|
||||
/** Access-layer bean wiring. */
|
||||
@@ -17,4 +19,11 @@ public class AccessConfig {
|
||||
TeamLeadLookup defaultTeamLeadLookup() {
|
||||
return new DefaultTeamLeadLookup();
|
||||
}
|
||||
|
||||
/** USER/TEAM projection unless another bean is defined (e.g. the saas resolver). */
|
||||
@Bean
|
||||
@ConditionalOnMissingBean(PrincipalResolver.class)
|
||||
PrincipalResolver defaultPrincipalResolver() {
|
||||
return new DefaultPrincipalResolver();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -25,7 +25,9 @@ import stirling.software.proprietary.access.model.PrincipalType;
|
||||
import stirling.software.proprietary.access.model.ResourceGrant;
|
||||
import stirling.software.proprietary.access.model.ResourceType;
|
||||
import stirling.software.proprietary.access.service.ResourceAccessService;
|
||||
import stirling.software.proprietary.security.database.repository.UserRepository;
|
||||
import stirling.software.proprietary.security.model.User;
|
||||
import stirling.software.proprietary.security.repository.TeamRepository;
|
||||
|
||||
/** Admin endpoints to grant/revoke access to gated resources (the portal, integration configs). */
|
||||
@RestController
|
||||
@@ -36,6 +38,8 @@ import stirling.software.proprietary.security.model.User;
|
||||
public class ResourceGrantController {
|
||||
|
||||
private final ResourceAccessService accessService;
|
||||
private final UserRepository userRepository;
|
||||
private final TeamRepository teamRepository;
|
||||
|
||||
@GetMapping("/grants")
|
||||
public ResponseEntity<?> list(
|
||||
@@ -45,6 +49,14 @@ public class ResourceGrantController {
|
||||
return ResponseEntity.ok(grants.stream().map(this::toDto).toList());
|
||||
}
|
||||
|
||||
@GetMapping("/grants/by-principal")
|
||||
public ResponseEntity<?> listByPrincipal(
|
||||
@RequestParam PrincipalType principalType, @RequestParam Long principalId) {
|
||||
List<ResourceGrant> grants =
|
||||
accessService.listGrantsForPrincipal(principalType, principalId);
|
||||
return ResponseEntity.ok(grants.stream().map(this::toDto).toList());
|
||||
}
|
||||
|
||||
@PostMapping("/grants")
|
||||
public ResponseEntity<?> create(
|
||||
@RequestBody GrantRequest request, @AuthenticationPrincipal User admin) {
|
||||
@@ -57,17 +69,26 @@ public class ResourceGrantController {
|
||||
"error",
|
||||
"resourceType, principalType and principalId are required"));
|
||||
}
|
||||
// PORTAL is a singleton (empty resourceId); every other type must name a resource.
|
||||
boolean portal = request.resourceType() == ResourceType.PORTAL;
|
||||
if (!portal && (request.resourceId() == null || request.resourceId().isBlank())) {
|
||||
return ResponseEntity.badRequest()
|
||||
.body(Map.of("error", "resourceId is required for " + request.resourceType()));
|
||||
}
|
||||
Long principalId = request.principalId();
|
||||
String principalError = validatePrincipalExists(request.principalType(), principalId);
|
||||
if (principalError != null) {
|
||||
return ResponseEntity.badRequest().body(Map.of("error", principalError));
|
||||
}
|
||||
AccessPermission permission =
|
||||
request.permission() == null ? AccessPermission.USE : request.permission();
|
||||
// PORTAL is a singleton resource; its grants always target the whole type.
|
||||
String resourceId =
|
||||
request.resourceType() == ResourceType.PORTAL ? "" : request.resourceId();
|
||||
String resourceId = portal ? "" : request.resourceId();
|
||||
ResourceGrant grant =
|
||||
accessService.grant(
|
||||
request.resourceType(),
|
||||
resourceId,
|
||||
request.principalType(),
|
||||
request.principalId(),
|
||||
principalId,
|
||||
permission,
|
||||
admin);
|
||||
return ResponseEntity.ok(toDto(grant));
|
||||
@@ -79,6 +100,14 @@ public class ResourceGrantController {
|
||||
return ResponseEntity.ok(Map.of("message", "Grant revoked"));
|
||||
}
|
||||
|
||||
// Rejects grants to nonexistent principals (dead rows otherwise).
|
||||
private String validatePrincipalExists(PrincipalType type, Long id) {
|
||||
return switch (type) {
|
||||
case USER -> userRepository.existsById(id) ? null : "User " + id + " does not exist";
|
||||
case TEAM -> teamRepository.existsById(id) ? null : "Team " + id + " does not exist";
|
||||
};
|
||||
}
|
||||
|
||||
private Map<String, Object> toDto(ResourceGrant g) {
|
||||
Map<String, Object> m = new HashMap<>();
|
||||
m.put("id", g.getId());
|
||||
|
||||
@@ -54,4 +54,15 @@ public abstract class OwnedResource {
|
||||
public Long getOwnerTeamId() {
|
||||
return ownerTeam != null ? ownerTeam.getId() : null;
|
||||
}
|
||||
|
||||
/** Owner as a principal ref; null when server-owned (admin-only ownership). */
|
||||
public PrincipalRef getOwnerRef() {
|
||||
if (getOwnerUserId() != null) {
|
||||
return PrincipalRef.user(getOwnerUserId());
|
||||
}
|
||||
if (getOwnerTeamId() != null) {
|
||||
return PrincipalRef.team(getOwnerTeamId());
|
||||
}
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
package stirling.software.proprietary.access.model;
|
||||
|
||||
import java.util.Locale;
|
||||
|
||||
/** A (type, id) principal pair; the atom grants and ownership are expressed in. */
|
||||
public record PrincipalRef(PrincipalType type, Long id) {
|
||||
|
||||
public static PrincipalRef user(Long id) {
|
||||
return new PrincipalRef(PrincipalType.USER, id);
|
||||
}
|
||||
|
||||
public static PrincipalRef team(Long id) {
|
||||
return new PrincipalRef(PrincipalType.TEAM, id);
|
||||
}
|
||||
|
||||
/** Canonical engine wire form, e.g. "user:12". */
|
||||
public String token() {
|
||||
return type.name().toLowerCase(Locale.ROOT) + ":" + id;
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
package stirling.software.proprietary.access.model;
|
||||
|
||||
/** Who a {@link ResourceGrant} is granted to. Org-wide access is expressed via default policy. */
|
||||
/** Who a {@link ResourceGrant} is granted to. */
|
||||
public enum PrincipalType {
|
||||
USER,
|
||||
TEAM
|
||||
|
||||
@@ -3,11 +3,15 @@ package stirling.software.proprietary.access.repository;
|
||||
import java.util.List;
|
||||
|
||||
import org.springframework.data.jpa.repository.JpaRepository;
|
||||
import org.springframework.data.jpa.repository.Modifying;
|
||||
import org.springframework.data.jpa.repository.Query;
|
||||
import org.springframework.data.repository.query.Param;
|
||||
import org.springframework.stereotype.Repository;
|
||||
|
||||
import stirling.software.proprietary.access.model.PrincipalType;
|
||||
import stirling.software.proprietary.access.model.ResourceGrant;
|
||||
import stirling.software.proprietary.access.model.ResourceType;
|
||||
import stirling.software.proprietary.security.model.User;
|
||||
|
||||
@Repository
|
||||
public interface ResourceGrantRepository extends JpaRepository<ResourceGrant, Long> {
|
||||
@@ -18,8 +22,20 @@ public interface ResourceGrantRepository extends JpaRepository<ResourceGrant, Lo
|
||||
List<ResourceGrant> findByResourceTypeAndPrincipalTypeAndPrincipalId(
|
||||
ResourceType resourceType, PrincipalType principalType, Long principalId);
|
||||
|
||||
/** All grants held by a principal, across resource types (for the manage-access view). */
|
||||
List<ResourceGrant> findByPrincipalTypeAndPrincipalId(
|
||||
PrincipalType principalType, Long principalId);
|
||||
|
||||
void deleteByResourceTypeAndResourceId(ResourceType resourceType, String resourceId);
|
||||
|
||||
/** Removes every grant held by a principal; used when the user/team behind it is deleted. */
|
||||
void deleteByPrincipalTypeAndPrincipalId(PrincipalType principalType, Long principalId);
|
||||
|
||||
// Detach issued grants so deleting the granting user does not hit the FK.
|
||||
@Modifying
|
||||
@Query("update ResourceGrant g set g.grantedBy = null where g.grantedBy = :user")
|
||||
void clearGrantedBy(@Param("user") User user);
|
||||
|
||||
boolean existsByResourceTypeAndResourceIdAndPrincipalTypeAndPrincipalId(
|
||||
ResourceType resourceType,
|
||||
String resourceId,
|
||||
|
||||
@@ -11,7 +11,12 @@ import stirling.software.proprietary.access.service.ResourceAccessService;
|
||||
import stirling.software.proprietary.security.model.User;
|
||||
import stirling.software.proprietary.security.service.UserService;
|
||||
|
||||
/** {@code @PreAuthorize} bean for portal-access checks. Active in self-hosted and saas. */
|
||||
/**
|
||||
* {@code @PreAuthorize} bean for portal-access checks. Active in self-hosted and saas. Convention:
|
||||
* every portal-exclusive endpoint is gated with
|
||||
* {@code @PreAuthorize("@resourceAccess.canUsePortal()")}; endpoints shared with the editor (e.g.
|
||||
* the policies API) must NOT be.
|
||||
*/
|
||||
@Component("resourceAccess")
|
||||
@RequiredArgsConstructor
|
||||
public class ResourceAccessSecurity {
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
package stirling.software.proprietary.access.service;
|
||||
|
||||
import java.util.HashSet;
|
||||
import java.util.Set;
|
||||
|
||||
import stirling.software.proprietary.access.model.PrincipalRef;
|
||||
import stirling.software.proprietary.security.model.User;
|
||||
|
||||
/**
|
||||
* Self-hosted projection: the user and their team. One deployment = one org, so ORG_ALL is open.
|
||||
*/
|
||||
public class DefaultPrincipalResolver implements PrincipalResolver {
|
||||
|
||||
@Override
|
||||
public Set<PrincipalRef> principalsOf(User user) {
|
||||
if (user == null) {
|
||||
return Set.of();
|
||||
}
|
||||
Set<PrincipalRef> principals = new HashSet<>();
|
||||
principals.add(PrincipalRef.user(user.getId()));
|
||||
if (user.getTeam() != null) {
|
||||
principals.add(PrincipalRef.team(user.getTeam().getId()));
|
||||
}
|
||||
return principals;
|
||||
}
|
||||
|
||||
// Self-hosted is a single deployment-wide org, so ORG_ALL admits every authenticated user.
|
||||
@Override
|
||||
public boolean allowsDeploymentWideAccess() {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
package stirling.software.proprietary.access.service;
|
||||
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
import lombok.RequiredArgsConstructor;
|
||||
|
||||
import stirling.software.common.model.enumeration.TeamRole;
|
||||
import stirling.software.proprietary.security.model.User;
|
||||
import stirling.software.proprietary.security.repository.TeamMembershipRepository;
|
||||
|
||||
/** Real lookup backed by team_memberships LEADER rows; wins over the no-op default bean. */
|
||||
@Component
|
||||
@RequiredArgsConstructor
|
||||
public class MembershipTeamLeadLookup implements TeamLeadLookup {
|
||||
|
||||
private final TeamMembershipRepository memberships;
|
||||
|
||||
@Override
|
||||
public boolean isAnyTeamLeader(User user) {
|
||||
return user != null
|
||||
&& user.getId() != null
|
||||
&& memberships.existsByUserIdAndRole(user.getId(), TeamRole.LEADER);
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean isLeaderOfTeam(User user, Long teamId) {
|
||||
return user != null
|
||||
&& user.getId() != null
|
||||
&& teamId != null
|
||||
&& memberships.existsByTeamIdAndUserIdAndRole(
|
||||
teamId, user.getId(), TeamRole.LEADER);
|
||||
}
|
||||
}
|
||||
@@ -36,15 +36,19 @@ public class OwnershipService {
|
||||
return accessService.canUseResource(
|
||||
type,
|
||||
String.valueOf(resource.getId()),
|
||||
resource.getOwnerUserId(),
|
||||
resource.getOwnerRef(),
|
||||
resource.getDefaultAccess(),
|
||||
user);
|
||||
}
|
||||
|
||||
/** Whether the user may manage the resource. */
|
||||
public boolean canManage(ResourceType type, OwnedResource resource, User user) {
|
||||
// Disabled resources bypass grants for MANAGE too: admin/owner only.
|
||||
if (!resource.isEnabled()) {
|
||||
return isAdmin(user) || isOwner(resource, user);
|
||||
}
|
||||
return accessService.canManageResource(
|
||||
type, String.valueOf(resource.getId()), resource.getOwnerUserId(), user);
|
||||
type, String.valueOf(resource.getId()), resource.getOwnerRef(), user);
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
package stirling.software.proprietary.access.service;
|
||||
|
||||
import java.util.Set;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
import stirling.software.proprietary.access.model.PrincipalRef;
|
||||
import stirling.software.proprietary.security.model.User;
|
||||
|
||||
/** Projects a user onto the set of principals they act as. */
|
||||
public interface PrincipalResolver {
|
||||
|
||||
/** Every principal the user acts as; empty for a null user. */
|
||||
Set<PrincipalRef> principalsOf(User user);
|
||||
|
||||
/**
|
||||
* Whether this deployment treats every authenticated user as one org, so the {@code ORG_ALL}
|
||||
* default policy admits anyone. Self-hosted: true. Multi-tenant saas: false, so an {@code
|
||||
* ORG_ALL} resource can't leak across tenants. Defaults to false (deny) for safety.
|
||||
*/
|
||||
default boolean allowsDeploymentWideAccess() {
|
||||
return false;
|
||||
}
|
||||
|
||||
/** Canonical wire tokens for the engine, e.g. "user:12". */
|
||||
default Set<String> principalTokens(User user) {
|
||||
return principalsOf(user).stream().map(PrincipalRef::token).collect(Collectors.toSet());
|
||||
}
|
||||
}
|
||||
@@ -14,6 +14,7 @@ import lombok.extern.slf4j.Slf4j;
|
||||
import stirling.software.common.model.enumeration.Role;
|
||||
import stirling.software.proprietary.access.model.AccessPermission;
|
||||
import stirling.software.proprietary.access.model.DefaultAccessPolicy;
|
||||
import stirling.software.proprietary.access.model.PrincipalRef;
|
||||
import stirling.software.proprietary.access.model.PrincipalType;
|
||||
import stirling.software.proprietary.access.model.ResourceGrant;
|
||||
import stirling.software.proprietary.access.model.ResourceType;
|
||||
@@ -29,6 +30,7 @@ public class ResourceAccessService {
|
||||
|
||||
private final ResourceGrantRepository grantRepository;
|
||||
private final TeamLeadLookup teamLeadLookup;
|
||||
private final PrincipalResolver principalResolver;
|
||||
|
||||
@Value("${security.portal.defaultAccess:ADMINS_AND_TEAM_LEADS}")
|
||||
private DefaultAccessPolicy portalDefaultPolicy;
|
||||
@@ -44,28 +46,28 @@ public class ResourceAccessService {
|
||||
public boolean canUseResource(
|
||||
ResourceType type,
|
||||
String resourceId,
|
||||
Long ownerUserId,
|
||||
PrincipalRef owner,
|
||||
DefaultAccessPolicy defaultPolicy,
|
||||
User user) {
|
||||
if (user == null) {
|
||||
return false;
|
||||
}
|
||||
if (isOwner(ownerUserId, user) || isAdmin(user)) {
|
||||
if (isOwner(owner, user) || isAdmin(user)) {
|
||||
return true;
|
||||
}
|
||||
if (hasGrant(type, normalize(resourceId), user, AccessPermission.USE)) {
|
||||
return true;
|
||||
}
|
||||
return matchesDefault(defaultPolicy, user);
|
||||
return matchesDefault(defaultPolicy, owner, user);
|
||||
}
|
||||
|
||||
/** Whether the user may manage (edit/delete/share) a resource. No default-policy fallback. */
|
||||
public boolean canManageResource(
|
||||
ResourceType type, String resourceId, Long ownerUserId, User user) {
|
||||
ResourceType type, String resourceId, PrincipalRef owner, User user) {
|
||||
if (user == null) {
|
||||
return false;
|
||||
}
|
||||
if (isOwner(ownerUserId, user) || isAdmin(user)) {
|
||||
if (isOwner(owner, user) || isAdmin(user)) {
|
||||
return true;
|
||||
}
|
||||
return hasGrant(type, normalize(resourceId), user, AccessPermission.MANAGE);
|
||||
@@ -110,21 +112,22 @@ public class ResourceAccessService {
|
||||
return grantRepository.findByResourceTypeAndResourceId(type, normalize(resourceId));
|
||||
}
|
||||
|
||||
/** Resource ids of the given type that this user (or their team) holds any grant on. */
|
||||
/** Every grant a principal holds, for the per-user/per-team manage-access view. */
|
||||
public List<ResourceGrant> listGrantsForPrincipal(
|
||||
PrincipalType principalType, Long principalId) {
|
||||
return grantRepository.findByPrincipalTypeAndPrincipalId(principalType, principalId);
|
||||
}
|
||||
|
||||
/** Resource ids of the given type that any of the user's principals holds a grant on. */
|
||||
public Set<String> grantedResourceIds(ResourceType type, User user) {
|
||||
if (user == null) {
|
||||
return Set.of();
|
||||
}
|
||||
Set<String> ids = new HashSet<>();
|
||||
for (ResourceGrant g :
|
||||
grantRepository.findByResourceTypeAndPrincipalTypeAndPrincipalId(
|
||||
type, PrincipalType.USER, user.getId())) {
|
||||
ids.add(g.getResourceId());
|
||||
}
|
||||
if (user.getTeam() != null) {
|
||||
for (PrincipalRef principal : principalResolver.principalsOf(user)) {
|
||||
for (ResourceGrant g :
|
||||
grantRepository.findByResourceTypeAndPrincipalTypeAndPrincipalId(
|
||||
type, PrincipalType.TEAM, user.getTeam().getId())) {
|
||||
type, principal.type(), principal.id())) {
|
||||
ids.add(g.getResourceId());
|
||||
}
|
||||
}
|
||||
@@ -135,18 +138,12 @@ public class ResourceAccessService {
|
||||
|
||||
private boolean hasGrant(
|
||||
ResourceType type, String resourceId, User user, AccessPermission required) {
|
||||
Long teamId = user.getTeam() != null ? user.getTeam().getId() : null;
|
||||
Set<PrincipalRef> principals = principalResolver.principalsOf(user);
|
||||
for (ResourceGrant g : grantRepository.findByResourceTypeAndResourceId(type, resourceId)) {
|
||||
if (!permissionSatisfies(g.getPermission(), required)) {
|
||||
continue;
|
||||
}
|
||||
if (g.getPrincipalType() == PrincipalType.USER
|
||||
&& g.getPrincipalId().equals(user.getId())) {
|
||||
return true;
|
||||
}
|
||||
if (g.getPrincipalType() == PrincipalType.TEAM
|
||||
&& teamId != null
|
||||
&& g.getPrincipalId().equals(teamId)) {
|
||||
if (principals.contains(new PrincipalRef(g.getPrincipalType(), g.getPrincipalId()))) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
@@ -161,20 +158,40 @@ public class ResourceAccessService {
|
||||
return held == AccessPermission.MANAGE;
|
||||
}
|
||||
|
||||
private boolean matchesDefault(DefaultAccessPolicy policy, User user) {
|
||||
private boolean matchesDefault(DefaultAccessPolicy policy, PrincipalRef owner, User user) {
|
||||
if (policy == null) {
|
||||
return false;
|
||||
}
|
||||
return switch (policy) {
|
||||
case ORG_ALL -> true;
|
||||
// Admins already pass above; only team leads here.
|
||||
case ADMINS_AND_TEAM_LEADS -> teamLeadLookup.isAnyTeamLeader(user);
|
||||
// Deployment-wide only where the resolver treats everyone as one org; saas resolvers
|
||||
// return false, so ORG_ALL cannot leak a tenant's resource to another tenant's users.
|
||||
case ORG_ALL -> principalResolver.allowsDeploymentWideAccess();
|
||||
// Admins already pass above; only team leads here, scoped to the owning team.
|
||||
case ADMINS_AND_TEAM_LEADS -> matchesTeamLeadDefault(owner, user);
|
||||
case EXPLICIT_ONLY -> false;
|
||||
};
|
||||
}
|
||||
|
||||
private boolean isOwner(Long ownerUserId, User user) {
|
||||
return ownerUserId != null && ownerUserId.equals(user.getId());
|
||||
// Portal (no owner) admits any team lead; a team-owned resource admits only that team's
|
||||
// leads; a user-owned resource admits no extra leads.
|
||||
private boolean matchesTeamLeadDefault(PrincipalRef owner, User user) {
|
||||
if (owner == null) {
|
||||
return teamLeadLookup.isAnyTeamLeader(user);
|
||||
}
|
||||
return owner.type() == PrincipalType.TEAM
|
||||
&& owner.id() != null
|
||||
&& teamLeadLookup.isLeaderOfTeam(user, owner.id());
|
||||
}
|
||||
|
||||
// Team owners are the owning team's leaders; plain members are not.
|
||||
private boolean isOwner(PrincipalRef owner, User user) {
|
||||
if (owner == null || owner.id() == null) {
|
||||
return false;
|
||||
}
|
||||
return switch (owner.type()) {
|
||||
case USER -> owner.id().equals(user.getId());
|
||||
case TEAM -> teamLeadLookup.isLeaderOfTeam(user, owner.id());
|
||||
};
|
||||
}
|
||||
|
||||
private boolean isAdmin(User user) {
|
||||
|
||||
@@ -18,15 +18,26 @@ public class SecretMasker {
|
||||
// Cap recursion so a pathologically nested payload cannot overflow the stack.
|
||||
private static final int MAX_DEPTH = 32;
|
||||
|
||||
// Key-name substrings that mark a value sensitive. Over-masking a non-secret is
|
||||
// safe; leaking a secret is not, so this errs broad - but a per-type schema
|
||||
// whitelist would be a stronger boundary for free-form config (follow-up).
|
||||
private static final Set<String> SENSITIVE_HINTS =
|
||||
Set.of(
|
||||
"secret",
|
||||
"password",
|
||||
"passphrase",
|
||||
"pwd",
|
||||
"token",
|
||||
"apikey",
|
||||
"accesskey",
|
||||
"credential",
|
||||
"privatekey");
|
||||
"privatekey",
|
||||
"authorization",
|
||||
"cookie",
|
||||
"session",
|
||||
"connectionstring",
|
||||
"bearer",
|
||||
"signature");
|
||||
|
||||
/** Replace sensitive values with the mask (recursively) for safe display. */
|
||||
public Map<String, Object> mask(Map<String, Object> config) {
|
||||
@@ -73,18 +84,24 @@ public class SecretMasker {
|
||||
|
||||
private Map<String, Object> merge(
|
||||
Map<String, Object> stored, Map<String, Object> incoming, int depth) {
|
||||
Map<String, Object> out = new LinkedHashMap<>(stored);
|
||||
// Replace semantics (PUT): the result is the incoming document, except a redacted secret
|
||||
// keeps its stored value. Keys absent from incoming are dropped, so edits can remove them.
|
||||
Map<String, Object> out = new LinkedHashMap<>();
|
||||
for (Map.Entry<String, Object> e : incoming.entrySet()) {
|
||||
String key = e.getKey();
|
||||
Object value = e.getValue();
|
||||
if (isSensitive(key)) {
|
||||
if (!isRedacted(value, depth)) {
|
||||
if (isRedacted(value, depth)) {
|
||||
if (stored.containsKey(key)) {
|
||||
out.put(key, stored.get(key)); // keep the stored secret
|
||||
}
|
||||
} else {
|
||||
out.put(key, value); // a real new secret replaces the stored one
|
||||
}
|
||||
continue; // redacted (blank / mask) -> keep stored
|
||||
continue;
|
||||
}
|
||||
if (depth < MAX_DEPTH
|
||||
&& out.get(key) instanceof Map<?, ?> s
|
||||
&& stored.get(key) instanceof Map<?, ?> s
|
||||
&& value instanceof Map<?, ?> i) {
|
||||
out.put(key, merge(castMap(s), castMap(i), depth + 1));
|
||||
} else {
|
||||
|
||||
@@ -142,6 +142,9 @@ public class AuditAspect {
|
||||
auditService.addTimingData(
|
||||
auditData, startTime, resp, auditedAnnotation.level(), isHttpRequest);
|
||||
|
||||
// Merge controller-set policy context + the internal-automation marker onto the event.
|
||||
auditService.addAutomationContext(auditData, req);
|
||||
|
||||
// Resolve the event type based on annotation and context
|
||||
String httpMethod = null;
|
||||
String path = null;
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
package stirling.software.proprietary.audit;
|
||||
|
||||
/**
|
||||
* Request-scoped keys a controller can set to enrich its own audit event with context the generic
|
||||
* aspect can't infer from the HTTP request alone (e.g. the policy a pipeline run belongs to). The
|
||||
* aspect copies these into the audit data in its {@code finally} block, after the controller body
|
||||
* has run. See {@code AuditService#addAutomationContext}.
|
||||
*/
|
||||
public final class AuditContext {
|
||||
|
||||
/** Request attribute: the name of the policy/pipeline a run executes. */
|
||||
public static final String REQ_ATTR_POLICY_NAME = "stirling.audit.policyName";
|
||||
|
||||
/** Request attribute: the ordered tool endpoint paths a run executes. */
|
||||
public static final String REQ_ATTR_POLICY_STEPS = "stirling.audit.policySteps";
|
||||
|
||||
private AuditContext() {}
|
||||
}
|
||||
@@ -130,6 +130,7 @@ public class ControllerAuditAspect {
|
||||
|
||||
String previousPrincipal = MDC.get("auditPrincipal");
|
||||
String previousOrigin = MDC.get("auditOrigin");
|
||||
String previousSource = MDC.get("auditSource");
|
||||
String previousIp = MDC.get("auditIp");
|
||||
|
||||
// EARLY CAPTURE: Capture from SecurityContext on request thread, store in MDC for async
|
||||
@@ -161,6 +162,14 @@ public class ControllerAuditAspect {
|
||||
return joinPoint.proceed();
|
||||
}
|
||||
|
||||
// Stamp the free-UI source only for non-@Audited controller traffic — an actual
|
||||
// tool / UI action. @Audited events (login, settings) return above without a source,
|
||||
// so they never count as an "active editor" or a free UI run. The finally block
|
||||
// restores auditSource, so a pooled thread can't leak a stale "WEB" into them.
|
||||
if (previousSource == null) {
|
||||
MDC.put("auditSource", auditService.captureCurrentSource());
|
||||
}
|
||||
|
||||
long start = System.currentTimeMillis();
|
||||
|
||||
// Use auditService to create the base audit data
|
||||
@@ -196,6 +205,10 @@ public class ControllerAuditAspect {
|
||||
// Call auditService but with isHttpRequest=true to skip additional timing
|
||||
auditService.addTimingData(data, start, resp, level, true);
|
||||
|
||||
// Merge controller-set policy context + the internal-automation marker (set after
|
||||
// the body ran, so it must happen here rather than with the pre-proceed HTTP data).
|
||||
auditService.addAutomationContext(data, req);
|
||||
|
||||
// Resolve the event type using the unified method
|
||||
AuditEventType eventType =
|
||||
auditService.resolveEventType(
|
||||
@@ -247,6 +260,7 @@ public class ControllerAuditAspect {
|
||||
} finally {
|
||||
restoreMdcValue("auditPrincipal", previousPrincipal);
|
||||
restoreMdcValue("auditOrigin", previousOrigin);
|
||||
restoreMdcValue("auditSource", previousSource);
|
||||
restoreMdcValue("auditIp", previousIp);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
package stirling.software.proprietary.audit;
|
||||
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
/** Self-hosted default: admins see the whole-server audit log, everyone else is denied. */
|
||||
@Component
|
||||
public class DefaultPortalAuditScopeResolver implements PortalAuditScopeResolver {
|
||||
|
||||
@Override
|
||||
public PortalAuditScope resolve() {
|
||||
return PortalAuditScopeResolver.hasAdminAuthority()
|
||||
? PortalAuditScope.server()
|
||||
: PortalAuditScope.denied();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
package stirling.software.proprietary.audit;
|
||||
|
||||
import java.time.Instant;
|
||||
|
||||
/** Immutable, cacheable projection of an {@code audit_events} row, shared across portal views. */
|
||||
public record PortalAuditEventRow(
|
||||
long id, String principal, String type, String data, Instant timestamp) {}
|
||||
@@ -0,0 +1,21 @@
|
||||
package stirling.software.proprietary.audit;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/** Resolved audit visibility: fullServer (admin), principals-scoped (team lead), or !allowed. */
|
||||
public record PortalAuditScope(
|
||||
boolean allowed, boolean fullServer, List<String> principals, String cacheKey) {
|
||||
|
||||
public static PortalAuditScope denied() {
|
||||
return new PortalAuditScope(false, false, List.of(), "denied");
|
||||
}
|
||||
|
||||
// Named server()/team() to avoid colliding with the record's fullServer() accessor.
|
||||
public static PortalAuditScope server() {
|
||||
return new PortalAuditScope(true, true, List.of(), "server");
|
||||
}
|
||||
|
||||
public static PortalAuditScope team(String cacheKey, List<String> principals) {
|
||||
return new PortalAuditScope(true, false, List.copyOf(principals), cacheKey);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
package stirling.software.proprietary.audit;
|
||||
|
||||
import org.springframework.security.core.Authentication;
|
||||
import org.springframework.security.core.context.SecurityContextHolder;
|
||||
|
||||
/** Resolves which slice of the audit log the caller may see. */
|
||||
public interface PortalAuditScopeResolver {
|
||||
|
||||
PortalAuditScope resolve();
|
||||
|
||||
/** True when the current authentication carries {@code ROLE_ADMIN}. */
|
||||
static boolean hasAdminAuthority() {
|
||||
Authentication auth = SecurityContextHolder.getContext().getAuthentication();
|
||||
return auth != null
|
||||
&& auth.getAuthorities().stream()
|
||||
.anyMatch(a -> "ROLE_ADMIN".equals(a.getAuthority()));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,136 @@
|
||||
package stirling.software.proprietary.classification;
|
||||
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty;
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.DeleteMapping;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.PutMapping;
|
||||
import org.springframework.web.bind.annotation.RequestBody;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
import org.springframework.web.server.ResponseStatusException;
|
||||
|
||||
import io.swagger.v3.oas.annotations.Hidden;
|
||||
import io.swagger.v3.oas.annotations.Operation;
|
||||
import io.swagger.v3.oas.annotations.tags.Tag;
|
||||
|
||||
import lombok.RequiredArgsConstructor;
|
||||
|
||||
import stirling.software.common.model.ApplicationProperties;
|
||||
import stirling.software.common.service.UserServiceInterface;
|
||||
import stirling.software.proprietary.classification.model.ClassificationLabels;
|
||||
import stirling.software.proprietary.classification.model.LabelsValidator;
|
||||
import stirling.software.proprietary.classification.store.ClassificationLabelStore;
|
||||
import stirling.software.proprietary.classification.store.TeamLabelsEntity;
|
||||
import stirling.software.proprietary.policy.config.PolicyManagementAuthority;
|
||||
|
||||
/**
|
||||
* Read/write the team's classification label set — the flat vocabulary the document classifier runs
|
||||
* against. Shared and team-scoped exactly like policies: every user reads their own team's labels,
|
||||
* and only a user who may edit policies (a team leader on SaaS, the global admin self-hosted; see
|
||||
* {@link PolicyManagementAuthority}) may change it — gated only when login is enabled, since
|
||||
* single-user deployments trust the local operator. A team with no stored labels reads as {@code
|
||||
* 204}; that team has no vocabulary, so its documents are not classified (there is no built-in
|
||||
* default on the backend or the engine — the label data lives only in the frontend).
|
||||
*/
|
||||
@RestController
|
||||
@RequestMapping("/api/v1/classification/labels")
|
||||
@Hidden
|
||||
@RequiredArgsConstructor
|
||||
@Tag(name = "Classification", description = "Team-scoped document-classification labels")
|
||||
@ConditionalOnBooleanProperty(name = "policies.enabled")
|
||||
public class ClassificationLabelsController {
|
||||
|
||||
private final ClassificationLabelStore labelStore;
|
||||
private final PolicyManagementAuthority policyManagementAuthority;
|
||||
private final ApplicationProperties applicationProperties;
|
||||
private final UserServiceInterface userService;
|
||||
|
||||
@GetMapping
|
||||
@Operation(
|
||||
summary = "Get the team's classification labels",
|
||||
description =
|
||||
"Returns the caller's team label set, or 204 when the team has none (its"
|
||||
+ " documents are then not classified).")
|
||||
public ResponseEntity<ClassificationLabels> getTeamLabels() {
|
||||
return labelStore
|
||||
.findByTeam(currentTeamId())
|
||||
.map(ResponseEntity::ok)
|
||||
.orElseGet(() -> ResponseEntity.noContent().build());
|
||||
}
|
||||
|
||||
@PutMapping(consumes = MediaType.APPLICATION_JSON_VALUE)
|
||||
@Operation(
|
||||
summary = "Save the team's classification labels",
|
||||
description =
|
||||
"Validates and stores the label set for the caller's team, shared by everyone"
|
||||
+ " on the team. Requires the policy-editor role for the team.")
|
||||
public ResponseEntity<ClassificationLabels> saveTeamLabels(
|
||||
@RequestBody ClassificationLabels labels) {
|
||||
requireEditingAllowed();
|
||||
validate(labels);
|
||||
ClassificationLabels saved = labelStore.save(currentTeamId(), labels, currentUsername());
|
||||
return ResponseEntity.ok(saved);
|
||||
}
|
||||
|
||||
@DeleteMapping
|
||||
@Operation(
|
||||
summary = "Reset the team's classification labels",
|
||||
description =
|
||||
"Removes the team's stored label set; its documents are then not classified"
|
||||
+ " until labels are saved again. Requires the policy-editor role for the"
|
||||
+ " team.")
|
||||
public ResponseEntity<Void> resetTeamLabels() {
|
||||
requireEditingAllowed();
|
||||
labelStore.deleteByTeam(currentTeamId());
|
||||
return ResponseEntity.noContent().build();
|
||||
}
|
||||
|
||||
private static void validate(ClassificationLabels labels) {
|
||||
try {
|
||||
LabelsValidator.validate(labels);
|
||||
} catch (IllegalArgumentException e) {
|
||||
throw new ResponseStatusException(HttpStatus.BAD_REQUEST, e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Editing the team labels requires the editor role for the caller's team — the same gate
|
||||
* policies use (team leader on SaaS, global admin self-hosted). Single-user deployments (login
|
||||
* disabled) have no such role, so they trust the local operator.
|
||||
*/
|
||||
private void requireEditingAllowed() {
|
||||
if (!applicationProperties.getSecurity().isEnableLogin()) {
|
||||
return;
|
||||
}
|
||||
if (!policyManagementAuthority.canEditPolicies()) {
|
||||
throw new ResponseStatusException(
|
||||
HttpStatus.FORBIDDEN,
|
||||
"The team classification labels may only be changed by a team leader");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The caller's team key. With login disabled the single operator owns the {@link
|
||||
* TeamLabelsEntity#NO_TEAM} sentinel row; with login enabled a caller with no resolvable team
|
||||
* is an error rather than being dropped into the shared sentinel bucket (which would let
|
||||
* unteamed users read and overwrite each other's "team" labels).
|
||||
*/
|
||||
private Long currentTeamId() {
|
||||
Long teamId = policyManagementAuthority.currentUserTeamId();
|
||||
if (teamId != null) {
|
||||
return teamId;
|
||||
}
|
||||
if (!applicationProperties.getSecurity().isEnableLogin()) {
|
||||
return TeamLabelsEntity.NO_TEAM;
|
||||
}
|
||||
throw new ResponseStatusException(
|
||||
HttpStatus.UNAUTHORIZED, "Could not resolve the current user's team");
|
||||
}
|
||||
|
||||
private String currentUsername() {
|
||||
return userService == null ? null : userService.getCurrentUsername();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
package stirling.software.proprietary.classification.model;
|
||||
|
||||
/**
|
||||
* One entry in the classification vocabulary. {@code id} is the label's stable identity (a slug,
|
||||
* unique within a set): it is what the engine returns and what is stored on the document. {@code
|
||||
* name} is the human display text the classifier model reasons over. {@code icon} is an optional
|
||||
* presentational key (a Material Symbols name shown in the file sidebar); the engine never sees it.
|
||||
*/
|
||||
public record ClassificationLabel(String id, String name, String icon) {}
|
||||
@@ -0,0 +1,16 @@
|
||||
package stirling.software.proprietary.classification.model;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* A flat multi-label classification vocabulary — the set of labels a document may be assigned.
|
||||
* Stored per team (admin-edited, shared by everyone on the team); the classifier runs against these
|
||||
* label names. A team with no stored set has no vocabulary, so its documents are not classified —
|
||||
* neither the backend nor the engine holds a default of its own.
|
||||
*/
|
||||
public record ClassificationLabels(List<ClassificationLabel> labels) {
|
||||
|
||||
public ClassificationLabels {
|
||||
labels = labels == null ? List.of() : List.copyOf(labels);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
package stirling.software.proprietary.classification.model;
|
||||
|
||||
import java.util.HashSet;
|
||||
import java.util.Locale;
|
||||
import java.util.Set;
|
||||
import java.util.regex.Pattern;
|
||||
|
||||
/**
|
||||
* Structural validation for a user- or admin-supplied label set, run before it is stored so a
|
||||
* malformed vocabulary can never reach the classifier. Mirrors the invariants the engine relies on:
|
||||
* non-blank ids and names, each unique within the set (ids exactly, names case-insensitively).
|
||||
*/
|
||||
public final class LabelsValidator {
|
||||
|
||||
private LabelsValidator() {}
|
||||
|
||||
// Generous upper bounds so a legitimate label set is never blocked, but a single team or user
|
||||
// can't store an unbounded blob that would bloat the row, balloon the classifier prompt, or
|
||||
// exhaust memory on deserialize.
|
||||
static final int MAX_LABELS = 500;
|
||||
static final int MAX_TEXT_LENGTH = 128;
|
||||
|
||||
// Icon is a Material Symbols key (lowercase, digits, hyphens). Enforce the SHAPE server-side —
|
||||
// the exact allowlist lives in the frontend — so a client bypassing the UI can't store
|
||||
// arbitrary
|
||||
// text that would render as garbage (or worse) in every teammate's sidebar.
|
||||
private static final Pattern ICON_KEY = Pattern.compile("^[a-z0-9-]+$");
|
||||
|
||||
/**
|
||||
* @throws IllegalArgumentException with a human-readable message when the label set is invalid.
|
||||
*/
|
||||
public static void validate(ClassificationLabels labels) {
|
||||
if (labels == null || labels.labels() == null) {
|
||||
throw new IllegalArgumentException("Labels are required");
|
||||
}
|
||||
if (labels.labels().size() > MAX_LABELS) {
|
||||
throw new IllegalArgumentException("Too many labels (max " + MAX_LABELS + ")");
|
||||
}
|
||||
Set<String> ids = new HashSet<>();
|
||||
Set<String> names = new HashSet<>();
|
||||
for (ClassificationLabel label : labels.labels()) {
|
||||
requireText(label.id(), "Label id");
|
||||
requireText(label.name(), "Label name");
|
||||
if (label.icon() != null && !label.icon().isEmpty()) {
|
||||
if (label.icon().length() > MAX_TEXT_LENGTH) {
|
||||
throw new IllegalArgumentException(
|
||||
"Label icon is too long (max " + MAX_TEXT_LENGTH + " characters)");
|
||||
}
|
||||
if (!ICON_KEY.matcher(label.icon()).matches()) {
|
||||
throw new IllegalArgumentException("Invalid label icon: " + label.icon());
|
||||
}
|
||||
}
|
||||
if (!ids.add(label.id().trim())) {
|
||||
throw new IllegalArgumentException("Duplicate label id: " + label.id());
|
||||
}
|
||||
if (!names.add(label.name().trim().toLowerCase(Locale.ROOT))) {
|
||||
throw new IllegalArgumentException("Duplicate label name: " + label.name());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static void requireText(String value, String field) {
|
||||
if (value == null || value.isBlank()) {
|
||||
throw new IllegalArgumentException(field + " must not be blank");
|
||||
}
|
||||
if (value.trim().length() > MAX_TEXT_LENGTH) {
|
||||
throw new IllegalArgumentException(
|
||||
field + " is too long (max " + MAX_TEXT_LENGTH + " characters)");
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
package stirling.software.proprietary.classification.store;
|
||||
|
||||
import java.util.Optional;
|
||||
|
||||
import stirling.software.proprietary.classification.model.ClassificationLabels;
|
||||
|
||||
/**
|
||||
* Stores one {@link ClassificationLabels} set per team. A {@code null} teamId addresses the
|
||||
* unteamed set (login disabled / no resolvable team), mirroring how the policy store treats a null
|
||||
* team.
|
||||
*/
|
||||
public interface ClassificationLabelStore {
|
||||
|
||||
/** The team's stored labels, or empty when it has none (callers then skip classification). */
|
||||
Optional<ClassificationLabels> findByTeam(Long teamId);
|
||||
|
||||
/** Create or replace the team's labels. Returns the stored value. */
|
||||
ClassificationLabels save(Long teamId, ClassificationLabels labels, String updatedBy);
|
||||
|
||||
/** Remove the team's labels (reset to default). Returns whether a set existed. */
|
||||
boolean deleteByTeam(Long teamId);
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
package stirling.software.proprietary.classification.store;
|
||||
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
|
||||
import stirling.software.proprietary.classification.model.ClassificationLabels;
|
||||
|
||||
/**
|
||||
* In-memory {@link ClassificationLabelStore} for tests and any future no-database mode. {@link
|
||||
* JpaClassificationLabelStore} is the runtime bean.
|
||||
*/
|
||||
public class InProcessClassificationLabelStore implements ClassificationLabelStore {
|
||||
|
||||
private final Map<Long, ClassificationLabels> byTeam = new ConcurrentHashMap<>();
|
||||
|
||||
@Override
|
||||
public Optional<ClassificationLabels> findByTeam(Long teamId) {
|
||||
return Optional.ofNullable(byTeam.get(key(teamId)));
|
||||
}
|
||||
|
||||
@Override
|
||||
public ClassificationLabels save(Long teamId, ClassificationLabels labels, String updatedBy) {
|
||||
byTeam.put(key(teamId), labels);
|
||||
return labels;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean deleteByTeam(Long teamId) {
|
||||
return byTeam.remove(key(teamId)) != null;
|
||||
}
|
||||
|
||||
private static long key(Long teamId) {
|
||||
return teamId == null ? TeamLabelsEntity.NO_TEAM : teamId;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
package stirling.software.proprietary.classification.store;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.util.Optional;
|
||||
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import lombok.RequiredArgsConstructor;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import stirling.software.proprietary.classification.model.ClassificationLabels;
|
||||
|
||||
import tools.jackson.core.JacksonException;
|
||||
import tools.jackson.databind.ObjectMapper;
|
||||
|
||||
/**
|
||||
* Durable {@link ClassificationLabelStore} backed by JPA; the runtime store. Gated on {@code
|
||||
* policies.enabled} — stored labels only matter when the Classification policy can run — so it
|
||||
* shares the policy subsystem's on/off switch. Each label set is persisted as JSON via {@link
|
||||
* TeamLabelsEntity}.
|
||||
*/
|
||||
@Slf4j
|
||||
@Service
|
||||
@RequiredArgsConstructor
|
||||
@ConditionalOnBooleanProperty(name = "policies.enabled")
|
||||
public class JpaClassificationLabelStore implements ClassificationLabelStore {
|
||||
|
||||
private final TeamLabelsRepository teamRepository;
|
||||
private final ObjectMapper objectMapper;
|
||||
|
||||
@Override
|
||||
public Optional<ClassificationLabels> findByTeam(Long teamId) {
|
||||
return teamRepository
|
||||
.findById(key(teamId))
|
||||
.flatMap(entity -> parse(entity.getLabelsJson(), "team " + teamId));
|
||||
}
|
||||
|
||||
@Override
|
||||
public ClassificationLabels save(Long teamId, ClassificationLabels labels, String updatedBy) {
|
||||
TeamLabelsEntity entity = new TeamLabelsEntity();
|
||||
entity.setTeamId(key(teamId));
|
||||
entity.setLabelsJson(objectMapper.writeValueAsString(labels));
|
||||
entity.setUpdatedAt(Instant.now());
|
||||
entity.setUpdatedBy(updatedBy);
|
||||
teamRepository.save(entity);
|
||||
return labels;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean deleteByTeam(Long teamId) {
|
||||
long id = key(teamId);
|
||||
if (!teamRepository.existsById(id)) {
|
||||
return false;
|
||||
}
|
||||
teamRepository.deleteById(id);
|
||||
return true;
|
||||
}
|
||||
|
||||
private Optional<ClassificationLabels> parse(String json, String owner) {
|
||||
try {
|
||||
return Optional.of(objectMapper.readValue(json, ClassificationLabels.class));
|
||||
} catch (JacksonException e) {
|
||||
// A stored label set that no longer parses (corruption / manual DB edit) must not break
|
||||
// classification: drop it so the caller treats the team as having no labels (and skips
|
||||
// classification) rather than surfacing a 500 on every upload.
|
||||
log.warn("Discarding unparseable stored labels for {}: {}", owner, e.getMessage());
|
||||
return Optional.empty();
|
||||
}
|
||||
}
|
||||
|
||||
/** Map the nullable team id onto the entity's non-null key (sentinel for the unteamed case). */
|
||||
private static long key(Long teamId) {
|
||||
return teamId == null ? TeamLabelsEntity.NO_TEAM : teamId;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
package stirling.software.proprietary.classification.store;
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.time.Instant;
|
||||
|
||||
import jakarta.persistence.Column;
|
||||
import jakarta.persistence.Entity;
|
||||
import jakarta.persistence.Id;
|
||||
import jakarta.persistence.Table;
|
||||
|
||||
import lombok.Getter;
|
||||
import lombok.NoArgsConstructor;
|
||||
import lombok.Setter;
|
||||
|
||||
/**
|
||||
* JPA row for a team's classification labels — one row per team. The label set lives as JSON in
|
||||
* {@code labelsJson} (authoritative on read). {@code teamId} is the natural key; the sentinel
|
||||
* {@link #NO_TEAM} stands in for the unteamed (login-disabled / self-hosted single-team) case,
|
||||
* since a primary key can't be null (policies store a nullable {@code team_id}, but this table is
|
||||
* keyed one-per-team). Kept decoupled from the security entities — {@code teamId} is a plain value,
|
||||
* not a foreign key — so classification can be enabled or disabled without touching them.
|
||||
*/
|
||||
@Entity
|
||||
@Table(name = "classification_labels")
|
||||
@NoArgsConstructor
|
||||
@Getter
|
||||
@Setter
|
||||
public class TeamLabelsEntity implements Serializable {
|
||||
|
||||
private static final long serialVersionUID = 1L;
|
||||
|
||||
/** Sentinel key for the unteamed label set (login disabled / no resolvable team). */
|
||||
public static final long NO_TEAM = 0L;
|
||||
|
||||
@Id
|
||||
@Column(name = "team_id")
|
||||
private long teamId;
|
||||
|
||||
@Column(name = "labels_json", columnDefinition = "text")
|
||||
private String labelsJson;
|
||||
|
||||
@Column(name = "updated_at")
|
||||
private Instant updatedAt;
|
||||
|
||||
@Column(name = "updated_by")
|
||||
private String updatedBy;
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
package stirling.software.proprietary.classification.store;
|
||||
|
||||
import org.springframework.data.jpa.repository.JpaRepository;
|
||||
import org.springframework.stereotype.Repository;
|
||||
|
||||
@Repository
|
||||
public interface TeamLabelsRepository extends JpaRepository<TeamLabelsEntity, Long> {}
|
||||
@@ -133,29 +133,45 @@ public final class S3Clients {
|
||||
* storage.s3.allow-private-endpoints=true}.
|
||||
*/
|
||||
static void validateEndpointHost(URI endpoint, boolean allowPrivate) {
|
||||
validateEndpointHost(
|
||||
endpoint,
|
||||
allowPrivate,
|
||||
"storage.s3.endpoint",
|
||||
"set storage.s3.allow-private-endpoints=true to opt in"
|
||||
+ " (e.g. for MinIO or in-cluster S3).");
|
||||
}
|
||||
|
||||
/**
|
||||
* The same private-address guard for S3 endpoints configured outside the {@code storage.s3.*}
|
||||
* block (e.g. per-source policy config), with the setting named in messages supplied by the
|
||||
* caller.
|
||||
*/
|
||||
public static void validateEndpointHost(
|
||||
URI endpoint, boolean allowPrivate, String settingName, String optInHint) {
|
||||
if (allowPrivate) {
|
||||
return;
|
||||
}
|
||||
String host = endpoint.getHost();
|
||||
if (host == null || host.isBlank()) {
|
||||
throw new IllegalStateException("storage.s3.endpoint must include a host: " + endpoint);
|
||||
throw new IllegalStateException(settingName + " must include a host: " + endpoint);
|
||||
}
|
||||
InetAddress[] addresses;
|
||||
try {
|
||||
addresses = InetAddress.getAllByName(host);
|
||||
} catch (UnknownHostException e) {
|
||||
throw new IllegalStateException(
|
||||
"Unable to resolve storage.s3.endpoint host '" + host + "'", e);
|
||||
"Unable to resolve " + settingName + " host '" + host + "'", e);
|
||||
}
|
||||
for (InetAddress address : addresses) {
|
||||
if (isPrivateOrLocal(address)) {
|
||||
throw new IllegalStateException(
|
||||
"storage.s3.endpoint host '"
|
||||
settingName
|
||||
+ " host '"
|
||||
+ host
|
||||
+ "' resolves to private/link-local address "
|
||||
+ address.getHostAddress()
|
||||
+ "; set storage.s3.allow-private-endpoints=true to opt in"
|
||||
+ " (e.g. for MinIO or in-cluster S3).");
|
||||
+ "; "
|
||||
+ optInHint);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -60,6 +60,8 @@ public class CustomAuditEventRepository implements AuditEventRepository {
|
||||
clean.put("requestId", rid);
|
||||
}
|
||||
|
||||
String source = MDC.get("auditSource");
|
||||
|
||||
String auditEventData = mapper.writeValueAsString(clean);
|
||||
log.debug("AuditEvent data (JSON): {}", auditEventData);
|
||||
|
||||
@@ -67,6 +69,7 @@ public class CustomAuditEventRepository implements AuditEventRepository {
|
||||
PersistentAuditEvent.builder()
|
||||
.principal(safePrincipal(ev.getPrincipal()))
|
||||
.type(ev.getType())
|
||||
.source(source)
|
||||
.data(auditEventData)
|
||||
.timestamp(ev.getTimestamp())
|
||||
.build();
|
||||
|
||||
@@ -0,0 +1,218 @@
|
||||
package stirling.software.proprietary.controller.api;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.LinkedHashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
|
||||
import org.apache.pdfbox.pdmodel.PDDocument;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.core.io.Resource;
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.PostMapping;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import org.springframework.web.bind.annotation.RequestParam;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
import org.springframework.web.multipart.MultipartFile;
|
||||
|
||||
import io.github.pixee.security.Filenames;
|
||||
import io.swagger.v3.oas.annotations.Hidden;
|
||||
import io.swagger.v3.oas.annotations.Operation;
|
||||
import io.swagger.v3.oas.annotations.tags.Tag;
|
||||
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import stirling.software.common.service.CustomPDFDocumentFactory;
|
||||
import stirling.software.common.service.PdfMetadataService;
|
||||
import stirling.software.common.service.UserServiceInterface;
|
||||
import stirling.software.common.util.TempFileManager;
|
||||
import stirling.software.common.util.WebResponseUtils;
|
||||
import stirling.software.proprietary.classification.model.ClassificationLabel;
|
||||
import stirling.software.proprietary.classification.store.ClassificationLabelStore;
|
||||
import stirling.software.proprietary.model.api.ai.AiPageText;
|
||||
import stirling.software.proprietary.policy.config.PolicyManagementAuthority;
|
||||
import stirling.software.proprietary.service.AiEngineClient;
|
||||
import stirling.software.proprietary.service.PdfContentExtractor;
|
||||
|
||||
import tools.jackson.databind.JsonNode;
|
||||
import tools.jackson.databind.ObjectMapper;
|
||||
import tools.jackson.databind.node.ObjectNode;
|
||||
|
||||
/**
|
||||
* Dispatchable tool that classifies a PDF and writes the result into its metadata.
|
||||
*
|
||||
* <p>Runs as a Classification-policy pipeline step: it reads a bounded page window, asks the AI
|
||||
* engine to classify the document against the caller's team label set, and stores the engine's JSON
|
||||
* answer — minus the transport-only {@code outcome} field — in the custom Info-dictionary key
|
||||
* {@link PdfMetadataService#CLASSIFICATION_KEY}. Returns the labelled PDF. Not intended for direct
|
||||
* client use.
|
||||
*/
|
||||
@Slf4j
|
||||
@Hidden
|
||||
@RestController
|
||||
@RequestMapping("/api/v1/ai/tools")
|
||||
@Tag(name = "AI Tools", description = "Dispatchable AI-backed tools.")
|
||||
public class ClassifyLabelController {
|
||||
|
||||
/** Pages read from each end of the document — mirrors the engine's window. */
|
||||
private static final int WINDOW_PAGES = 2;
|
||||
|
||||
private static final String CLASSIFY_ENDPOINT = "/api/v1/documents/classify";
|
||||
|
||||
private final CustomPDFDocumentFactory pdfDocumentFactory;
|
||||
private final TempFileManager tempFileManager;
|
||||
private final PdfContentExtractor pdfContentExtractor;
|
||||
private final PdfMetadataService pdfMetadataService;
|
||||
private final AiEngineClient aiEngineClient;
|
||||
private final ObjectMapper objectMapper;
|
||||
private final UserServiceInterface userService;
|
||||
|
||||
/**
|
||||
* Present only when the policy subsystem is enabled ({@code policies.enabled}); the store and
|
||||
* team authority are gated on it. Null otherwise, in which case there are no team labels to
|
||||
* classify against and the document is passed through unlabelled.
|
||||
*/
|
||||
private final ClassificationLabelStore labelStore;
|
||||
|
||||
private final PolicyManagementAuthority policyManagementAuthority;
|
||||
|
||||
public ClassifyLabelController(
|
||||
CustomPDFDocumentFactory pdfDocumentFactory,
|
||||
TempFileManager tempFileManager,
|
||||
PdfContentExtractor pdfContentExtractor,
|
||||
PdfMetadataService pdfMetadataService,
|
||||
AiEngineClient aiEngineClient,
|
||||
ObjectMapper objectMapper,
|
||||
@Autowired(required = false) UserServiceInterface userService,
|
||||
@Autowired(required = false) ClassificationLabelStore labelStore,
|
||||
@Autowired(required = false) PolicyManagementAuthority policyManagementAuthority) {
|
||||
this.pdfDocumentFactory = pdfDocumentFactory;
|
||||
this.tempFileManager = tempFileManager;
|
||||
this.pdfContentExtractor = pdfContentExtractor;
|
||||
this.pdfMetadataService = pdfMetadataService;
|
||||
this.aiEngineClient = aiEngineClient;
|
||||
this.objectMapper = objectMapper;
|
||||
this.userService = userService;
|
||||
this.labelStore = labelStore;
|
||||
this.policyManagementAuthority = policyManagementAuthority;
|
||||
}
|
||||
|
||||
@PostMapping(value = "/classify-and-label", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
|
||||
@Operation(
|
||||
summary = "Classify a PDF and label its metadata",
|
||||
description =
|
||||
"Reads the first two and last two pages, classifies the document via the AI"
|
||||
+ " engine, and stores the result in the StirlingPDFClassification"
|
||||
+ " metadata field. Dispatched by the Classification policy; not"
|
||||
+ " intended for direct client use.")
|
||||
public ResponseEntity<Resource> classifyAndLabel(
|
||||
@RequestParam("fileInput") MultipartFile fileInput) throws IOException {
|
||||
try (PDDocument document = pdfDocumentFactory.load(fileInput, true)) {
|
||||
String fileName = safeFileName(fileInput.getOriginalFilename());
|
||||
|
||||
List<EngineLabel> allowed = resolveAllowedLabels();
|
||||
if (allowed.isEmpty()) {
|
||||
// No vocabulary to classify against (the team stored no labels): pass the file
|
||||
// through unlabelled rather than ask the engine to classify against nothing.
|
||||
log.debug("[classify-and-label] {} has no team labels; skipping", fileName);
|
||||
return WebResponseUtils.pdfDocToWebResponse(document, fileName, tempFileManager);
|
||||
}
|
||||
|
||||
List<AiPageText> pages = extractWindow(document);
|
||||
String requestBody =
|
||||
objectMapper.writeValueAsString(
|
||||
new ClassifyEngineRequest(fileName, pages, allowed));
|
||||
|
||||
String userId = userService != null ? userService.getCurrentUsername() : null;
|
||||
String responseJson = aiEngineClient.post(CLASSIFY_ENDPOINT, requestBody, userId);
|
||||
|
||||
pdfMetadataService.setClassificationMetadata(document, toMetadataValue(responseJson));
|
||||
log.debug("[classify-and-label] labelled {} ({} window pages)", fileName, pages.size());
|
||||
|
||||
return WebResponseUtils.pdfDocToWebResponse(document, fileName, tempFileManager);
|
||||
}
|
||||
}
|
||||
|
||||
private List<AiPageText> extractWindow(PDDocument document) throws IOException {
|
||||
List<AiPageText> pages = new ArrayList<>();
|
||||
for (int pageNumber : windowPageNumbers(document.getNumberOfPages(), WINDOW_PAGES)) {
|
||||
String text = pdfContentExtractor.extractPageTextRaw(document, pageNumber);
|
||||
if (text != null && !text.isBlank()) {
|
||||
pages.add(new AiPageText(pageNumber, text));
|
||||
}
|
||||
}
|
||||
return pages;
|
||||
}
|
||||
|
||||
/** First and last {@code window} page numbers (1-based), de-duplicated and in order. */
|
||||
static List<Integer> windowPageNumbers(int pageCount, int window) {
|
||||
Set<Integer> numbers = new LinkedHashSet<>();
|
||||
for (int page = 1; page <= Math.min(window, pageCount); page++) {
|
||||
numbers.add(page);
|
||||
}
|
||||
for (int page = Math.max(1, pageCount - window + 1); page <= pageCount; page++) {
|
||||
numbers.add(page);
|
||||
}
|
||||
return new ArrayList<>(numbers);
|
||||
}
|
||||
|
||||
/** Drop the transport-only {@code outcome} discriminator; keep the rest verbatim. */
|
||||
private String toMetadataValue(String engineResponseJson) {
|
||||
JsonNode node = objectMapper.readTree(engineResponseJson);
|
||||
if (node instanceof ObjectNode object) {
|
||||
object.remove("outcome");
|
||||
}
|
||||
return objectMapper.writeValueAsString(node);
|
||||
}
|
||||
|
||||
private static String safeFileName(String originalFilename) {
|
||||
String name = Filenames.toSimpleFileName(originalFilename);
|
||||
return (name == null || name.isBlank()) ? "classified.pdf" : name;
|
||||
}
|
||||
|
||||
/**
|
||||
* The allowed labels for the caller's team as {@code {id, name}} pairs, de-duplicated by id.
|
||||
* The engine shows the model the names and returns the ids (icons are presentational and never
|
||||
* sent). Returns an empty list — the caller then skips classification — when the policy
|
||||
* subsystem is disabled (no store) or the team has no stored labels. The engine holds no
|
||||
* default vocabulary of its own, so a team's stored labels are the only source.
|
||||
*/
|
||||
private List<EngineLabel> resolveAllowedLabels() {
|
||||
if (labelStore == null) {
|
||||
return List.of();
|
||||
}
|
||||
Long teamId =
|
||||
policyManagementAuthority == null
|
||||
? null
|
||||
: policyManagementAuthority.currentUserTeamId();
|
||||
|
||||
Map<String, EngineLabel> byId = new LinkedHashMap<>();
|
||||
labelStore.findByTeam(teamId).ifPresent(labels -> collectLabels(labels.labels(), byId));
|
||||
|
||||
return List.copyOf(byId.values());
|
||||
}
|
||||
|
||||
private static void collectLabels(
|
||||
List<ClassificationLabel> labels, Map<String, EngineLabel> into) {
|
||||
for (ClassificationLabel label : labels) {
|
||||
if (label.id() == null
|
||||
|| label.id().isBlank()
|
||||
|| label.name() == null
|
||||
|| label.name().isBlank()) {
|
||||
continue;
|
||||
}
|
||||
into.putIfAbsent(label.id(), new EngineLabel(label.id(), label.name()));
|
||||
}
|
||||
}
|
||||
|
||||
/** One allowed label sent to the engine: stable id + the name the model reasons over. */
|
||||
private record EngineLabel(String id, String name) {}
|
||||
|
||||
/** Request body for the engine's {@code /api/v1/documents/classify} endpoint. */
|
||||
private record ClassifyEngineRequest(
|
||||
String fileName, List<AiPageText> pages, List<EngineLabel> labels) {}
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
package stirling.software.proprietary.controller.api;
|
||||
|
||||
import java.time.Instant;
|
||||
import java.time.temporal.ChronoUnit;
|
||||
import java.util.List;
|
||||
|
||||
import org.springframework.context.annotation.Profile;
|
||||
import org.springframework.security.access.prepost.PreAuthorize;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
|
||||
import lombok.RequiredArgsConstructor;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import stirling.software.common.model.enumeration.Role;
|
||||
import stirling.software.proprietary.audit.AuditLevel;
|
||||
import stirling.software.proprietary.config.AuditConfigurationProperties;
|
||||
import stirling.software.proprietary.model.api.usage.FleetUsageStats;
|
||||
import stirling.software.proprietary.repository.PersistentAuditEventRepository;
|
||||
import stirling.software.proprietary.security.config.EnterpriseEndpoint;
|
||||
import stirling.software.proprietary.security.database.repository.UserRepository;
|
||||
|
||||
/**
|
||||
* Admin endpoint exposing free-editor fleet usage for the portal Usage card. Audit-derived figures
|
||||
* (active editors, PDFs processed) are null (rendered as "N/A") rather than a misleading 0 whenever
|
||||
* the data can't exist: the events they count (PDF_PROCESS, FILE_OPERATION, HTTP_REQUEST) are all
|
||||
* STANDARD level, so a gate on {@code isEnabled()} alone would still return 0 at level=OFF/BASIC —
|
||||
* we gate on {@code isLevelEnabled(STANDARD)} instead.
|
||||
*
|
||||
* <p>Known limitation: on a login-disabled self-hosted instance every request is anonymous, so its
|
||||
* audit origin is SYSTEM (not WEB) and it is excluded from these WEB-only counts — active/PDFs then
|
||||
* read 0 despite real usage. Historical audit rows written before the {@code source} column existed
|
||||
* carry {@code source=null}, so the cumulative "PDFs edited" figure effectively starts at deploy.
|
||||
*/
|
||||
@Slf4j
|
||||
@RestController
|
||||
@RequestMapping("/api/v1/usage")
|
||||
@PreAuthorize("hasRole('ADMIN')")
|
||||
@RequiredArgsConstructor
|
||||
@EnterpriseEndpoint
|
||||
// Self-hosted only: counts are server-wide. On SaaS this endpoint is owned by the team-scoped
|
||||
// SaasFleetUsageController (@Profile("saas")) so one backend can't leak another tenant's usage.
|
||||
@Profile("!saas")
|
||||
public class FleetUsageController {
|
||||
|
||||
private final PersistentAuditEventRepository auditRepository;
|
||||
private final UserRepository userRepository;
|
||||
private final AuditConfigurationProperties auditConfig;
|
||||
|
||||
@GetMapping("/fleet-stats")
|
||||
public FleetUsageStats fleetStats() {
|
||||
// Exclude the reserved INTERNAL_API_USER row that InitialSecuritySetup creates on every
|
||||
// install, so a fresh single-admin instance reads 1 editor, not 2.
|
||||
Long deployed = userRepository.countByUsernameNot(Role.INTERNAL_API_USER.getRoleId());
|
||||
// STANDARD is the level at which the counted events are recorded; below it the data
|
||||
// can't exist, so report N/A instead of a 0 that would misrepresent an empty table.
|
||||
boolean auditOn = auditConfig.isLevelEnabled(AuditLevel.STANDARD);
|
||||
Instant since = Instant.now().minus(30, ChronoUnit.DAYS);
|
||||
Long active =
|
||||
auditOn
|
||||
? auditRepository.countDistinctPrincipalsBySourceExcludingTypeAfter(
|
||||
"WEB", "UI_DATA", since)
|
||||
: null;
|
||||
Long pdfs =
|
||||
auditOn
|
||||
? auditRepository.countByTypeInAndSourceAndTimestampAfter(
|
||||
List.of("PDF_PROCESS", "FILE_OPERATION"), "WEB", Instant.EPOCH)
|
||||
: null;
|
||||
if (active != null && deployed != null && active > deployed) {
|
||||
active = deployed; // active editors are a subset of those deployed
|
||||
}
|
||||
return new FleetUsageStats(deployed, active, pdfs);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
package stirling.software.proprietary.controller.api;
|
||||
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.RequestParam;
|
||||
|
||||
import io.swagger.v3.oas.annotations.Operation;
|
||||
|
||||
import lombok.RequiredArgsConstructor;
|
||||
|
||||
import stirling.software.common.annotations.api.ProprietaryUiDataApi;
|
||||
import stirling.software.proprietary.audit.PortalAuditScope;
|
||||
import stirling.software.proprietary.audit.PortalAuditScopeResolver;
|
||||
import stirling.software.proprietary.model.api.documents.PortalDocumentsResponseDto;
|
||||
import stirling.software.proprietary.security.config.EnterpriseEndpoint;
|
||||
import stirling.software.proprietary.service.PortalDocumentsService;
|
||||
|
||||
/** Serves the portal Documents review queue, derived from real audit data and scoped per caller. */
|
||||
@ProprietaryUiDataApi
|
||||
@RequiredArgsConstructor
|
||||
@EnterpriseEndpoint
|
||||
public class PortalDocumentsController {
|
||||
|
||||
private final PortalDocumentsService portalDocumentsService;
|
||||
private final PortalAuditScopeResolver auditScopeResolver;
|
||||
|
||||
// tier accepted for mock-seam symmetry; ignored (queue isn't tier-scoped).
|
||||
@GetMapping("/documents")
|
||||
@Operation(
|
||||
summary = "Documents review queue",
|
||||
description = "Files processed through the org, derived from the audit trail.")
|
||||
public ResponseEntity<PortalDocumentsResponseDto> getDocuments(
|
||||
@RequestParam(value = "tier", required = false) String tier) {
|
||||
PortalAuditScope scope = auditScopeResolver.resolve();
|
||||
if (!scope.allowed()) {
|
||||
return ResponseEntity.status(HttpStatus.FORBIDDEN).build();
|
||||
}
|
||||
PortalDocumentsResponseDto body =
|
||||
scope.fullServer()
|
||||
? portalDocumentsService.serverDocuments()
|
||||
: portalDocumentsService.scopedDocuments(
|
||||
scope.cacheKey(), scope.principals());
|
||||
return ResponseEntity.ok(body);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
package stirling.software.proprietary.controller.api;
|
||||
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.RequestParam;
|
||||
|
||||
import io.swagger.v3.oas.annotations.Operation;
|
||||
|
||||
import lombok.RequiredArgsConstructor;
|
||||
|
||||
import stirling.software.common.annotations.api.ProprietaryUiDataApi;
|
||||
import stirling.software.proprietary.audit.PortalAuditScope;
|
||||
import stirling.software.proprietary.audit.PortalAuditScopeResolver;
|
||||
import stirling.software.proprietary.model.api.audit.InfraAuditLogResponse;
|
||||
import stirling.software.proprietary.security.config.EnterpriseEndpoint;
|
||||
import stirling.software.proprietary.service.PortalInfraAuditService;
|
||||
|
||||
/** Serves the Infrastructure → Audit tab from real audit data, scoped and cached per caller. */
|
||||
@ProprietaryUiDataApi
|
||||
@RequiredArgsConstructor
|
||||
@EnterpriseEndpoint
|
||||
public class PortalInfraAuditController {
|
||||
|
||||
private final PortalInfraAuditService portalInfraAuditService;
|
||||
private final PortalAuditScopeResolver auditScopeResolver;
|
||||
|
||||
// tier accepted for endpoint symmetry; ignored (audit log isn't tier-scoped).
|
||||
@GetMapping("/infrastructure/audit-log")
|
||||
@Operation(
|
||||
summary = "Infrastructure audit log",
|
||||
description = "Recent audit events shaped for the portal Infrastructure → Audit tab.")
|
||||
public ResponseEntity<InfraAuditLogResponse> getInfrastructureAuditLog(
|
||||
@RequestParam(value = "tier", required = false) String tier) {
|
||||
PortalAuditScope scope = auditScopeResolver.resolve();
|
||||
if (!scope.allowed()) {
|
||||
// Return 403 (not throw) so the tab shows its access message, not a generic 500.
|
||||
return ResponseEntity.status(HttpStatus.FORBIDDEN).build();
|
||||
}
|
||||
InfraAuditLogResponse body =
|
||||
scope.fullServer()
|
||||
? portalInfraAuditService.serverAuditLog()
|
||||
: portalInfraAuditService.scopedAuditLog(
|
||||
scope.cacheKey(), scope.principals());
|
||||
return ResponseEntity.ok(body);
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import static stirling.software.common.util.ProviderUtils.validateProvider;
|
||||
import java.time.Instant;
|
||||
import java.time.temporal.ChronoUnit;
|
||||
import java.util.*;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
import org.springframework.beans.factory.annotation.Qualifier;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
@@ -28,13 +29,16 @@ import stirling.software.common.model.ApplicationProperties.Security.OAUTH2.Clie
|
||||
import stirling.software.common.model.ApplicationProperties.Security.SAML2;
|
||||
import stirling.software.common.model.FileInfo;
|
||||
import stirling.software.common.model.enumeration.Role;
|
||||
import stirling.software.common.model.enumeration.TeamRole;
|
||||
import stirling.software.common.model.oauth2.GitHubProvider;
|
||||
import stirling.software.common.model.oauth2.GoogleProvider;
|
||||
import stirling.software.common.model.oauth2.KeycloakProvider;
|
||||
import stirling.software.proprietary.access.service.ResourceAccessService;
|
||||
import stirling.software.proprietary.audit.AuditEventType;
|
||||
import stirling.software.proprietary.audit.AuditLevel;
|
||||
import stirling.software.proprietary.config.AuditConfigurationProperties;
|
||||
import stirling.software.proprietary.model.Team;
|
||||
import stirling.software.proprietary.model.TeamMembership;
|
||||
import stirling.software.proprietary.model.dto.TeamWithUserCountDTO;
|
||||
import stirling.software.proprietary.repository.PersistentAuditEventRepository;
|
||||
import stirling.software.proprietary.security.config.EnterpriseEndpoint;
|
||||
@@ -44,6 +48,7 @@ import stirling.software.proprietary.security.model.Authority;
|
||||
import stirling.software.proprietary.security.model.SessionEntity;
|
||||
import stirling.software.proprietary.security.model.User;
|
||||
import stirling.software.proprietary.security.model.dto.AdminUserSummary;
|
||||
import stirling.software.proprietary.security.repository.TeamMembershipRepository;
|
||||
import stirling.software.proprietary.security.repository.TeamRepository;
|
||||
import stirling.software.proprietary.security.saml2.CustomSaml2AuthenticatedPrincipal;
|
||||
import stirling.software.proprietary.security.service.DatabaseServiceInterface;
|
||||
@@ -65,6 +70,7 @@ public class ProprietaryUIDataController {
|
||||
private final SessionPersistentRegistry sessionPersistentRegistry;
|
||||
private final UserRepository userRepository;
|
||||
private final TeamRepository teamRepository;
|
||||
private final TeamMembershipRepository teamMembershipRepository;
|
||||
private final SessionRepository sessionRepository;
|
||||
private final DatabaseServiceInterface databaseService;
|
||||
private final boolean runningEE;
|
||||
@@ -73,6 +79,7 @@ public class ProprietaryUIDataController {
|
||||
private final PersistentAuditEventRepository auditRepository;
|
||||
private final MfaService mfaService;
|
||||
private final LoginAttemptService loginAttemptService;
|
||||
private final ResourceAccessService resourceAccessService;
|
||||
|
||||
public ProprietaryUIDataController(
|
||||
ApplicationProperties applicationProperties,
|
||||
@@ -80,6 +87,7 @@ public class ProprietaryUIDataController {
|
||||
SessionPersistentRegistry sessionPersistentRegistry,
|
||||
UserRepository userRepository,
|
||||
TeamRepository teamRepository,
|
||||
TeamMembershipRepository teamMembershipRepository,
|
||||
SessionRepository sessionRepository,
|
||||
DatabaseServiceInterface databaseService,
|
||||
ObjectMapper objectMapper,
|
||||
@@ -87,12 +95,14 @@ public class ProprietaryUIDataController {
|
||||
UserLicenseSettingsService licenseSettingsService,
|
||||
PersistentAuditEventRepository auditRepository,
|
||||
MfaService mfaService,
|
||||
LoginAttemptService loginAttemptService) {
|
||||
LoginAttemptService loginAttemptService,
|
||||
ResourceAccessService resourceAccessService) {
|
||||
this.applicationProperties = applicationProperties;
|
||||
this.auditConfig = auditConfig;
|
||||
this.sessionPersistentRegistry = sessionPersistentRegistry;
|
||||
this.userRepository = userRepository;
|
||||
this.teamRepository = teamRepository;
|
||||
this.teamMembershipRepository = teamMembershipRepository;
|
||||
this.sessionRepository = sessionRepository;
|
||||
this.databaseService = databaseService;
|
||||
this.objectMapper = objectMapper;
|
||||
@@ -101,6 +111,7 @@ public class ProprietaryUIDataController {
|
||||
this.auditRepository = auditRepository;
|
||||
this.mfaService = mfaService;
|
||||
this.loginAttemptService = loginAttemptService;
|
||||
this.resourceAccessService = resourceAccessService;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -370,8 +381,11 @@ public class ProprietaryUIDataController {
|
||||
boolean premiumEnabled = applicationProperties.getPremium().isEnabled();
|
||||
|
||||
// Convert User entities to AdminUserSummary DTOs to exclude sensitive fields
|
||||
Set<Long> leaderUserIds = leaderUserIds();
|
||||
List<AdminUserSummary> userSummaries =
|
||||
sortedUsers.stream().map(this::convertUserToSummary).toList();
|
||||
sortedUsers.stream()
|
||||
.map(user -> convertUserToSummary(user, leaderUserIds))
|
||||
.toList();
|
||||
|
||||
AdminSettingsData data = new AdminSettingsData();
|
||||
data.setUsers(userSummaries);
|
||||
@@ -390,6 +404,10 @@ public class ProprietaryUIDataController {
|
||||
data.setLicenseMaxUsers(licenseMaxUsers);
|
||||
data.setPremiumEnabled(premiumEnabled);
|
||||
data.setMailEnabled(applicationProperties.getMail().isEnabled());
|
||||
// Email invites need the invites toggle AND SMTP on; matches the inviteUsers precondition.
|
||||
data.setEmailInvitesEnabled(
|
||||
applicationProperties.getMail().isEnableInvites()
|
||||
&& applicationProperties.getMail().isEnabled());
|
||||
data.setUserSettings(userSettings);
|
||||
data.setLockedUsers(loginAttemptService.getAllBlockedUsers());
|
||||
|
||||
@@ -468,9 +486,18 @@ public class ProprietaryUIDataController {
|
||||
teamLastRequest.put(teamId, lastActivity);
|
||||
}
|
||||
|
||||
Map<Long, List<String>> teamOwners = new HashMap<>();
|
||||
for (TeamMembership row :
|
||||
teamMembershipRepository.findByRoleFetchingUserAndTeam(TeamRole.LEADER)) {
|
||||
teamOwners
|
||||
.computeIfAbsent(row.getTeam().getId(), id -> new ArrayList<>())
|
||||
.add(row.getUser().getUsername());
|
||||
}
|
||||
|
||||
TeamsData data = new TeamsData();
|
||||
data.setTeamsWithCounts(teamsWithCounts);
|
||||
data.setTeamLastRequest(teamLastRequest);
|
||||
data.setTeamOwners(teamOwners);
|
||||
|
||||
return ResponseEntity.ok(data);
|
||||
}
|
||||
@@ -510,11 +537,17 @@ public class ProprietaryUIDataController {
|
||||
userLastRequest.put(username, lastRequest);
|
||||
}
|
||||
|
||||
Set<Long> ownerUserIds =
|
||||
teamMembershipRepository.findByTeamIdAndRole(id, TeamRole.LEADER).stream()
|
||||
.map(row -> row.getUser().getId())
|
||||
.collect(Collectors.toSet());
|
||||
|
||||
TeamDetailsData data = new TeamDetailsData();
|
||||
data.setTeam(team);
|
||||
data.setTeamUsers(teamUsers);
|
||||
data.setAvailableUsers(availableUsers);
|
||||
data.setUserLastRequest(userLastRequest);
|
||||
data.setOwnerUserIds(ownerUserIds);
|
||||
|
||||
return ResponseEntity.ok(data);
|
||||
}
|
||||
@@ -535,13 +568,24 @@ public class ProprietaryUIDataController {
|
||||
return ResponseEntity.ok(data);
|
||||
}
|
||||
|
||||
/** User ids holding a LEADER membership on any team. */
|
||||
private Set<Long> leaderUserIds() {
|
||||
return teamMembershipRepository.findByRoleFetchingUserAndTeam(TeamRole.LEADER).stream()
|
||||
.map(row -> row.getUser().getId())
|
||||
.collect(Collectors.toSet());
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert User entity to AdminUserSummary DTO, excluding sensitive fields like password and
|
||||
* apiKey.
|
||||
*/
|
||||
private AdminUserSummary convertUserToSummary(User user) {
|
||||
private AdminUserSummary convertUserToSummary(User user, Set<Long> leaderUserIds) {
|
||||
AdminUserSummary summary = new AdminUserSummary();
|
||||
summary.setId(user.getId());
|
||||
summary.setTeamLead(leaderUserIds.contains(user.getId()));
|
||||
// Authoritative portal access, same call /me uses, so the roster honors the configured
|
||||
// policy instead of the frontend guessing from role/team-leadership.
|
||||
summary.setPortalAccess(resourceAccessService.canAccessPortal(user));
|
||||
summary.setUsername(user.getUsername());
|
||||
summary.setEmail(user.getUsername()); // Use username as email for consistency
|
||||
summary.setRoleName(user.getRoleName());
|
||||
@@ -609,6 +653,7 @@ public class ProprietaryUIDataController {
|
||||
private int licenseMaxUsers;
|
||||
private boolean premiumEnabled;
|
||||
private boolean mailEnabled;
|
||||
private boolean emailInvitesEnabled;
|
||||
private Map<String, Map<String, String>> userSettings;
|
||||
private List<String> lockedUsers;
|
||||
}
|
||||
@@ -629,6 +674,7 @@ public class ProprietaryUIDataController {
|
||||
public static class TeamsData {
|
||||
private List<TeamWithUserCountDTO> teamsWithCounts;
|
||||
private Map<Long, Date> teamLastRequest;
|
||||
private Map<Long, List<String>> teamOwners;
|
||||
}
|
||||
|
||||
@Data
|
||||
@@ -637,6 +683,7 @@ public class ProprietaryUIDataController {
|
||||
private List<User> teamUsers;
|
||||
private List<User> availableUsers;
|
||||
private Map<String, Date> userLastRequest;
|
||||
private Set<Long> ownerUserIds;
|
||||
}
|
||||
|
||||
@Data
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
package stirling.software.proprietary.formdetection.catalog;
|
||||
|
||||
import java.io.InputStream;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
|
||||
import org.springframework.core.io.ClassPathResource;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import jakarta.annotation.PostConstruct;
|
||||
|
||||
import lombok.RequiredArgsConstructor;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import stirling.software.proprietary.formdetection.model.ModelCatalogEntry;
|
||||
|
||||
import tools.jackson.core.type.TypeReference;
|
||||
import tools.jackson.databind.ObjectMapper;
|
||||
|
||||
/** Loads the curated Auto Form Detection model catalog from a bundled JSON resource. */
|
||||
@Slf4j
|
||||
@Service
|
||||
@RequiredArgsConstructor
|
||||
public class ModelCatalogService {
|
||||
|
||||
private static final String CATALOG_RESOURCE = "formdetection/model-catalog.json";
|
||||
|
||||
private final ObjectMapper objectMapper;
|
||||
|
||||
private volatile List<ModelCatalogEntry> entries = List.of();
|
||||
private volatile Map<String, ModelCatalogEntry> byId = Map.of();
|
||||
|
||||
@PostConstruct
|
||||
void load() {
|
||||
try (InputStream is = new ClassPathResource(CATALOG_RESOURCE).getInputStream()) {
|
||||
List<ModelCatalogEntry> loaded =
|
||||
objectMapper.readValue(is, new TypeReference<List<ModelCatalogEntry>>() {});
|
||||
Map<String, ModelCatalogEntry> map = new LinkedHashMap<>();
|
||||
for (ModelCatalogEntry entry : loaded) {
|
||||
if (entry.getId() != null && !entry.getId().isBlank()) {
|
||||
map.put(entry.getId(), entry);
|
||||
}
|
||||
}
|
||||
this.entries = List.copyOf(map.values());
|
||||
this.byId = Map.copyOf(map);
|
||||
log.info("Loaded {} Auto Form Detection model catalog entries", entries.size());
|
||||
} catch (Exception e) {
|
||||
log.error(
|
||||
"Failed to load Auto Form Detection model catalog from {}",
|
||||
CATALOG_RESOURCE,
|
||||
e);
|
||||
}
|
||||
}
|
||||
|
||||
public List<ModelCatalogEntry> getAll() {
|
||||
return entries;
|
||||
}
|
||||
|
||||
public Optional<ModelCatalogEntry> getById(String id) {
|
||||
return Optional.ofNullable(id == null ? null : byId.get(id));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,174 @@
|
||||
package stirling.software.proprietary.formdetection.controller;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import org.apache.pdfbox.pdmodel.PDDocument;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.PostMapping;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import org.springframework.web.bind.annotation.RequestParam;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
import org.springframework.web.multipart.MultipartFile;
|
||||
|
||||
import io.github.pixee.security.Filenames;
|
||||
import io.swagger.v3.oas.annotations.Operation;
|
||||
import io.swagger.v3.oas.annotations.tags.Tag;
|
||||
|
||||
import lombok.RequiredArgsConstructor;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import stirling.software.common.service.CustomPDFDocumentFactory;
|
||||
import stirling.software.common.util.FormUtils;
|
||||
import stirling.software.common.util.FormUtils.NewFormFieldDefinition;
|
||||
import stirling.software.common.util.TempFileManager;
|
||||
import stirling.software.common.util.WebResponseUtils;
|
||||
import stirling.software.proprietary.formdetection.inference.OnnxFormDetector;
|
||||
import stirling.software.proprietary.formdetection.inference.Yolo;
|
||||
import stirling.software.proprietary.formdetection.model.DetectedField;
|
||||
import stirling.software.proprietary.formdetection.model.ModelCatalogEntry;
|
||||
import stirling.software.proprietary.formdetection.render.CoordinateMapper;
|
||||
import stirling.software.proprietary.formdetection.render.PageRasterizer;
|
||||
import stirling.software.proprietary.formdetection.service.FormDetectionModelManager;
|
||||
|
||||
/**
|
||||
* Server-side detection endpoint. Gated behind the {@code form-detection} endpoint key, which is
|
||||
* disabled until a model is installed (so the tool tile is greyed in the UI). Returns the shared
|
||||
* detection schema, or - when {@code applyToPdf=true} - the AcroForm-applied PDF.
|
||||
*/
|
||||
@Slf4j
|
||||
@RestController
|
||||
@RequestMapping("/api/v1/ai/form-detection")
|
||||
@ConditionalOnClass(name = "ai.onnxruntime.OrtEnvironment")
|
||||
@RequiredArgsConstructor
|
||||
@Tag(name = "Auto Form Detection")
|
||||
public class FormDetectionController {
|
||||
|
||||
private final FormDetectionModelManager manager;
|
||||
private final OnnxFormDetector detector;
|
||||
private final PageRasterizer rasterizer;
|
||||
private final CustomPDFDocumentFactory pdfDocumentFactory;
|
||||
private final TempFileManager tempFileManager;
|
||||
|
||||
@PostMapping(value = "/detect", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
|
||||
@Operation(
|
||||
summary = "Detect form fields with the installed AI model",
|
||||
description =
|
||||
"Runs the installed ONNX model over each page and returns detected fields in"
|
||||
+ " PDF points. With applyToPdf=true, returns the fillable PDF instead.")
|
||||
public ResponseEntity<?> detect(
|
||||
@RequestParam("file") MultipartFile file,
|
||||
@RequestParam(value = "confThreshold", required = false) Float confThreshold,
|
||||
@RequestParam(value = "applyToPdf", required = false, defaultValue = "false")
|
||||
boolean applyToPdf)
|
||||
throws IOException {
|
||||
|
||||
if (!manager.isReady()) {
|
||||
return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
|
||||
.body(
|
||||
Map.of(
|
||||
"reason",
|
||||
"DEPENDENCY",
|
||||
"message",
|
||||
"AI form-detection model is not installed"));
|
||||
}
|
||||
ModelCatalogEntry spec = manager.getActiveEntry().orElse(null);
|
||||
if (spec == null) {
|
||||
return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
|
||||
.body(
|
||||
Map.of(
|
||||
"reason",
|
||||
"DEPENDENCY",
|
||||
"message",
|
||||
"Active model spec unavailable"));
|
||||
}
|
||||
float score = confThreshold != null ? confThreshold : spec.getScoreThreshold();
|
||||
byte[] pdfBytes = file.getBytes();
|
||||
|
||||
List<DetectedField> detections = new ArrayList<>();
|
||||
try {
|
||||
for (PageRasterizer.RasterPage page :
|
||||
rasterizer.rasterize(pdfBytes, spec.getInputSize())) {
|
||||
Yolo.Preprocessed pre =
|
||||
Yolo.preprocess(page.rgba(), page.widthPx(), page.heightPx(), spec);
|
||||
Yolo.RawOutput out = detector.infer(pre.chw(), spec.getInputSize());
|
||||
for (Yolo.Detection d : Yolo.decode(out, spec, pre, score)) {
|
||||
DetectedField.RectPt rect = CoordinateMapper.toPdfPoints(d, page);
|
||||
detections.add(
|
||||
new DetectedField(
|
||||
fieldType(spec, d.classId()),
|
||||
page.pageIndex(),
|
||||
rect,
|
||||
d.score()));
|
||||
}
|
||||
}
|
||||
} catch (IllegalStateException e) {
|
||||
// e.g. ONNX Runtime native unavailable for this OS/arch - report unavailable cleanly
|
||||
// rather than a 500. Cannot happen on a normally-built jar (all platforms bundled), but
|
||||
// keeps a slimmed/mis-targeted build from erroring.
|
||||
log.warn("Auto Form Detection inference unavailable: {}", e.getMessage());
|
||||
return ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE)
|
||||
.body(Map.of("reason", "DEPENDENCY", "message", e.getMessage()));
|
||||
}
|
||||
|
||||
if (applyToPdf) {
|
||||
try (PDDocument document = pdfDocumentFactory.load(file)) {
|
||||
FormUtils.repairMissingWidgetPageReferences(document);
|
||||
List<NewFormFieldDefinition> defs = new ArrayList<>();
|
||||
for (DetectedField f : detections) {
|
||||
defs.add(toDefinition(f));
|
||||
}
|
||||
FormUtils.addFields(document, defs);
|
||||
return WebResponseUtils.pdfDocToWebResponse(
|
||||
document, baseName(file) + ".pdf", tempFileManager);
|
||||
}
|
||||
}
|
||||
return ResponseEntity.ok(new DetectResponse(detections));
|
||||
}
|
||||
|
||||
private static String fieldType(ModelCatalogEntry spec, int classId) {
|
||||
List<String> types = spec.getClassFieldTypes();
|
||||
if (types != null && classId >= 0 && classId < types.size()) {
|
||||
return types.get(classId);
|
||||
}
|
||||
return "text";
|
||||
}
|
||||
|
||||
private static NewFormFieldDefinition toDefinition(DetectedField f) {
|
||||
DetectedField.RectPt r = f.rectInPdfPoints();
|
||||
return new NewFormFieldDefinition(
|
||||
null,
|
||||
null,
|
||||
f.type(),
|
||||
f.page(),
|
||||
(float) r.x(),
|
||||
(float) r.y(),
|
||||
(float) r.w(),
|
||||
(float) r.h(),
|
||||
Boolean.FALSE,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null);
|
||||
}
|
||||
|
||||
private static String baseName(MultipartFile file) {
|
||||
String original = Filenames.toSimpleFileName(file.getOriginalFilename());
|
||||
if (original == null || original.isBlank()) {
|
||||
original = "document";
|
||||
}
|
||||
String stem =
|
||||
original.toLowerCase().endsWith(".pdf")
|
||||
? original.substring(0, original.length() - 4)
|
||||
: original;
|
||||
return stem + "_form";
|
||||
}
|
||||
|
||||
/** Shared JSON response (mirrors the browser pipeline output). */
|
||||
public record DetectResponse(List<DetectedField> detections) {}
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
package stirling.software.proprietary.formdetection.controller;
|
||||
|
||||
import org.apache.commons.lang3.StringUtils;
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.security.access.prepost.PreAuthorize;
|
||||
import org.springframework.web.bind.annotation.DeleteMapping;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.PostMapping;
|
||||
import org.springframework.web.bind.annotation.RequestBody;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import org.springframework.web.bind.annotation.RequestParam;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
|
||||
import io.swagger.v3.oas.annotations.Operation;
|
||||
import io.swagger.v3.oas.annotations.tags.Tag;
|
||||
|
||||
import lombok.Data;
|
||||
import lombok.RequiredArgsConstructor;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import stirling.software.proprietary.formdetection.model.ModelStatusResponse;
|
||||
import stirling.software.proprietary.formdetection.service.FormDetectionModelManager;
|
||||
|
||||
/**
|
||||
* Admin-managed lifecycle for the Auto Form Detection model. Lives under the never-gated {@code
|
||||
* form-detection-model} endpoint key so install/status stay reachable while the feature itself (the
|
||||
* {@code form-detection} detect endpoint) is disabled until a model is ready.
|
||||
*/
|
||||
@Slf4j
|
||||
@RestController
|
||||
@RequestMapping("/api/v1/ai/form-detection-model")
|
||||
@RequiredArgsConstructor
|
||||
@Tag(name = "Auto Form Detection")
|
||||
public class FormDetectionModelController {
|
||||
|
||||
private final FormDetectionModelManager manager;
|
||||
|
||||
@GetMapping("/status")
|
||||
@Operation(summary = "Auto Form Detection model status, progress and catalog")
|
||||
public ResponseEntity<ModelStatusResponse> status() {
|
||||
return ResponseEntity.ok(manager.status());
|
||||
}
|
||||
|
||||
@PostMapping("/install")
|
||||
@PreAuthorize("hasRole('ADMIN')")
|
||||
@Operation(summary = "Install (download + checksum-verify) a catalog model")
|
||||
public ResponseEntity<ModelStatusResponse> install(@RequestBody InstallRequest request) {
|
||||
if (request == null || StringUtils.isBlank(request.getModelId())) {
|
||||
ModelStatusResponse s = manager.status();
|
||||
s.setError("modelId is required");
|
||||
return ResponseEntity.badRequest().body(s);
|
||||
}
|
||||
try {
|
||||
manager.startInstall(request.getModelId());
|
||||
return ResponseEntity.accepted().body(manager.status());
|
||||
} catch (IllegalStateException e) {
|
||||
ModelStatusResponse s = manager.status();
|
||||
s.setError(e.getMessage());
|
||||
return ResponseEntity.status(HttpStatus.CONFLICT).body(s);
|
||||
} catch (IllegalArgumentException e) {
|
||||
ModelStatusResponse s = manager.status();
|
||||
s.setError(e.getMessage());
|
||||
return ResponseEntity.badRequest().body(s);
|
||||
}
|
||||
}
|
||||
|
||||
@DeleteMapping
|
||||
@PreAuthorize("hasRole('ADMIN')")
|
||||
@Operation(summary = "Uninstall a model")
|
||||
public ResponseEntity<ModelStatusResponse> delete(
|
||||
@RequestParam(name = "modelId", required = false) String modelId) {
|
||||
try {
|
||||
manager.deleteModel(modelId);
|
||||
return ResponseEntity.ok(manager.status());
|
||||
} catch (IllegalStateException e) {
|
||||
ModelStatusResponse s = manager.status();
|
||||
s.setError(e.getMessage());
|
||||
return ResponseEntity.status(HttpStatus.CONFLICT).body(s);
|
||||
}
|
||||
}
|
||||
|
||||
@PostMapping("/config")
|
||||
@PreAuthorize("hasRole('ADMIN')")
|
||||
@Operation(
|
||||
summary = "Update the feature on/off switch and execution mode (auto/browser/server)")
|
||||
public ResponseEntity<ModelStatusResponse> config(@RequestBody ConfigRequest request) {
|
||||
if (request == null) {
|
||||
return ResponseEntity.badRequest().body(manager.status());
|
||||
}
|
||||
try {
|
||||
if (request.getEnabled() != null) {
|
||||
manager.setEnabled(request.getEnabled());
|
||||
}
|
||||
if (StringUtils.isNotBlank(request.getExecutionMode())) {
|
||||
manager.setExecutionMode(request.getExecutionMode());
|
||||
}
|
||||
return ResponseEntity.ok(manager.status());
|
||||
} catch (IllegalArgumentException e) {
|
||||
ModelStatusResponse s = manager.status();
|
||||
s.setError(e.getMessage());
|
||||
return ResponseEntity.badRequest().body(s);
|
||||
}
|
||||
}
|
||||
|
||||
@Data
|
||||
public static class ConfigRequest {
|
||||
/** Master on/off; {@code null} leaves it unchanged. */
|
||||
private Boolean enabled;
|
||||
|
||||
/** auto|browser|server; blank/null leaves it unchanged. */
|
||||
private String executionMode;
|
||||
}
|
||||
|
||||
@Data
|
||||
public static class InstallRequest {
|
||||
private String modelId;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
package stirling.software.proprietary.formdetection.controller;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.nio.file.Path;
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.Optional;
|
||||
|
||||
import org.springframework.core.io.FileSystemResource;
|
||||
import org.springframework.core.io.Resource;
|
||||
import org.springframework.core.io.support.ResourceRegion;
|
||||
import org.springframework.http.CacheControl;
|
||||
import org.springframework.http.HttpHeaders;
|
||||
import org.springframework.http.HttpRange;
|
||||
import org.springframework.http.HttpStatus;
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.http.ResponseEntity;
|
||||
import org.springframework.web.bind.annotation.GetMapping;
|
||||
import org.springframework.web.bind.annotation.RequestHeader;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
|
||||
import io.swagger.v3.oas.annotations.Operation;
|
||||
import io.swagger.v3.oas.annotations.tags.Tag;
|
||||
|
||||
import lombok.RequiredArgsConstructor;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import stirling.software.proprietary.formdetection.service.FormDetectionModelManager;
|
||||
|
||||
/**
|
||||
* Streams the installed .onnx to browsers for in-browser inference. Supports resumable range
|
||||
* requests with a stable (content-addressed) ETag and a public Cache-Control so each browser
|
||||
* fetches the ~200MB model at most once. The file is streamed from disk by Spring's resource
|
||||
* converters - never buffered into heap.
|
||||
*/
|
||||
@Slf4j
|
||||
@RestController
|
||||
@RequestMapping("/api/v1/ai/form-detection-model")
|
||||
@RequiredArgsConstructor
|
||||
@Tag(name = "Auto Form Detection")
|
||||
public class FormDetectionModelServeController {
|
||||
|
||||
private final FormDetectionModelManager manager;
|
||||
|
||||
@GetMapping("/file")
|
||||
@Operation(summary = "Stream the installed model (.onnx); supports HTTP range requests")
|
||||
public ResponseEntity<Object> serveModel(@RequestHeader HttpHeaders headers)
|
||||
throws IOException {
|
||||
Optional<Path> active = manager.getActiveModelFile();
|
||||
if (active.isEmpty()) {
|
||||
return ResponseEntity.notFound().build();
|
||||
}
|
||||
Path path = active.get();
|
||||
Resource resource = new FileSystemResource(path);
|
||||
long length = resource.contentLength();
|
||||
String etag =
|
||||
"\""
|
||||
+ manager.getActiveEtag()
|
||||
.orElseGet(() -> length + "-" + path.toFile().lastModified())
|
||||
+ "\"";
|
||||
// cachePublic + maxAge override the interceptor's blanket "no-store" on /api responses.
|
||||
CacheControl cacheControl = CacheControl.maxAge(Duration.ofDays(30)).cachePublic();
|
||||
|
||||
List<HttpRange> ranges;
|
||||
try {
|
||||
ranges = headers.getRange();
|
||||
} catch (IllegalArgumentException ex) {
|
||||
return ResponseEntity.status(HttpStatus.REQUESTED_RANGE_NOT_SATISFIABLE)
|
||||
.header(HttpHeaders.CONTENT_RANGE, "bytes */" + length)
|
||||
.build();
|
||||
}
|
||||
|
||||
if (ranges.isEmpty()) {
|
||||
return ResponseEntity.ok()
|
||||
.header(HttpHeaders.ACCEPT_RANGES, "bytes")
|
||||
.eTag(etag)
|
||||
.cacheControl(cacheControl)
|
||||
.contentType(MediaType.APPLICATION_OCTET_STREAM)
|
||||
.contentLength(length)
|
||||
.body(resource);
|
||||
}
|
||||
|
||||
// Serve a single region (the resumable-download case). Returning a bare List would lose its
|
||||
// generic element type when the method returns ResponseEntity<Object>, leaving the region
|
||||
// converter unable to match it.
|
||||
HttpRange range = ranges.get(0);
|
||||
long start = range.getRangeStart(length);
|
||||
long count = Math.min(range.getRangeEnd(length) - start + 1, length - start);
|
||||
ResourceRegion region = new ResourceRegion(resource, start, count);
|
||||
// No explicit content type: the region converter derives it. Presetting octet-stream makes
|
||||
// the ResourceRegion converter's content-negotiation reject the body.
|
||||
return ResponseEntity.status(HttpStatus.PARTIAL_CONTENT)
|
||||
.header(HttpHeaders.ACCEPT_RANGES, "bytes")
|
||||
.eTag(etag)
|
||||
.cacheControl(cacheControl)
|
||||
.body(region);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,153 @@
|
||||
package stirling.software.proprietary.formdetection.inference;
|
||||
|
||||
import java.nio.FloatBuffer;
|
||||
import java.nio.file.Path;
|
||||
import java.util.Collections;
|
||||
import java.util.concurrent.Semaphore;
|
||||
import java.util.concurrent.locks.ReentrantReadWriteLock;
|
||||
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import jakarta.annotation.PreDestroy;
|
||||
|
||||
import lombok.RequiredArgsConstructor;
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import stirling.software.proprietary.formdetection.model.ModelCatalogEntry;
|
||||
import stirling.software.proprietary.formdetection.service.FormDetectionModelManager;
|
||||
|
||||
import ai.onnxruntime.OnnxTensor;
|
||||
import ai.onnxruntime.OnnxValue;
|
||||
import ai.onnxruntime.OrtEnvironment;
|
||||
import ai.onnxruntime.OrtException;
|
||||
import ai.onnxruntime.OrtSession;
|
||||
|
||||
/**
|
||||
* Holds the ONNX Runtime session for the active model. Lazily (re)loads when the active model
|
||||
* changes, guards session swaps with a read/write lock, and bounds concurrent inferences to limit
|
||||
* memory. The session input is NCHW float32 {@code [1,3,N,N]}; the raw output is returned as-is for
|
||||
* {@link Yolo#decode} to interpret per the model spec.
|
||||
*/
|
||||
@Slf4j
|
||||
@Service
|
||||
@ConditionalOnClass(name = "ai.onnxruntime.OrtEnvironment")
|
||||
@RequiredArgsConstructor
|
||||
public class OnnxFormDetector {
|
||||
|
||||
private final FormDetectionModelManager manager;
|
||||
|
||||
private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
|
||||
private final Semaphore concurrency =
|
||||
new Semaphore(Math.max(1, Runtime.getRuntime().availableProcessors() / 2));
|
||||
|
||||
private volatile OrtSession session;
|
||||
private volatile String loadedModelId;
|
||||
private volatile String inputName;
|
||||
|
||||
public Yolo.RawOutput infer(float[] chw, int inputSize) {
|
||||
ensureLoaded();
|
||||
concurrency.acquireUninterruptibly();
|
||||
lock.readLock().lock();
|
||||
try {
|
||||
OrtEnvironment env = OrtEnvironment.getEnvironment();
|
||||
long[] shape = {1, 3, inputSize, inputSize};
|
||||
try (OnnxTensor tensor = OnnxTensor.createTensor(env, FloatBuffer.wrap(chw), shape);
|
||||
OrtSession.Result results =
|
||||
session.run(Collections.singletonMap(inputName, tensor))) {
|
||||
OnnxValue value = results.get(0);
|
||||
Object raw = value.getValue();
|
||||
if (!(raw instanceof float[][][] out3) || out3.length == 0) {
|
||||
throw new IllegalStateException(
|
||||
"Unexpected ONNX output type: "
|
||||
+ (raw == null ? "null" : raw.getClass()));
|
||||
}
|
||||
float[][] m = out3[0];
|
||||
int d1 = m.length;
|
||||
int d2 = d1 > 0 ? m[0].length : 0;
|
||||
float[] flat = new float[d1 * d2];
|
||||
for (int i = 0; i < d1; i++) {
|
||||
System.arraycopy(m[i], 0, flat, i * d2, d2);
|
||||
}
|
||||
return new Yolo.RawOutput(flat, d1, d2);
|
||||
}
|
||||
} catch (OrtException e) {
|
||||
throw new IllegalStateException("ONNX inference failed: " + e.getMessage(), e);
|
||||
} finally {
|
||||
lock.readLock().unlock();
|
||||
concurrency.release();
|
||||
}
|
||||
}
|
||||
|
||||
/** Force the next inference to reload from disk (called after install/uninstall). */
|
||||
public void unload() {
|
||||
lock.writeLock().lock();
|
||||
try {
|
||||
closeSession();
|
||||
loadedModelId = null;
|
||||
inputName = null;
|
||||
} finally {
|
||||
lock.writeLock().unlock();
|
||||
}
|
||||
}
|
||||
|
||||
private void ensureLoaded() {
|
||||
String activeId = manager.getActiveEntry().map(ModelCatalogEntry::getId).orElse(null);
|
||||
if (activeId == null) {
|
||||
throw new IllegalStateException("No Auto Form Detection model installed");
|
||||
}
|
||||
if (activeId.equals(loadedModelId) && session != null) {
|
||||
return;
|
||||
}
|
||||
lock.writeLock().lock();
|
||||
try {
|
||||
if (activeId.equals(loadedModelId) && session != null) {
|
||||
return;
|
||||
}
|
||||
Path file =
|
||||
manager.getActiveModelFile()
|
||||
.orElseThrow(
|
||||
() -> new IllegalStateException("Active model file missing"));
|
||||
try {
|
||||
OrtEnvironment env = OrtEnvironment.getEnvironment();
|
||||
OrtSession.SessionOptions opts = new OrtSession.SessionOptions();
|
||||
try {
|
||||
opts.setIntraOpNumThreads(
|
||||
Math.max(1, Runtime.getRuntime().availableProcessors() / 2));
|
||||
} catch (OrtException ignored) {
|
||||
// best-effort tuning
|
||||
}
|
||||
closeSession();
|
||||
session = env.createSession(file.toString(), opts);
|
||||
inputName = session.getInputNames().iterator().next();
|
||||
loadedModelId = activeId;
|
||||
log.info("Loaded ONNX session for Auto Form Detection model '{}'", activeId);
|
||||
} catch (OrtException | RuntimeException | LinkageError e) {
|
||||
// Native library missing/incompatible for this OS+arch (e.g. a Linux-slimmed jar
|
||||
// run on Windows), or a model load failure. Degrade gracefully instead of letting
|
||||
// an UnsatisfiedLinkError escape - the detect endpoint reports unavailable and the
|
||||
// server keeps running.
|
||||
throw new IllegalStateException(
|
||||
"ONNX Runtime is unavailable on this platform/build: " + e.getMessage(), e);
|
||||
}
|
||||
} finally {
|
||||
lock.writeLock().unlock();
|
||||
}
|
||||
}
|
||||
|
||||
@PreDestroy
|
||||
void close() {
|
||||
unload();
|
||||
}
|
||||
|
||||
private void closeSession() {
|
||||
if (session != null) {
|
||||
try {
|
||||
session.close();
|
||||
} catch (Exception ignored) {
|
||||
// ignore
|
||||
}
|
||||
session = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,252 @@
|
||||
package stirling.software.proprietary.formdetection.inference;
|
||||
|
||||
import java.awt.Color;
|
||||
import java.awt.Graphics2D;
|
||||
import java.awt.RenderingHints;
|
||||
import java.awt.image.BufferedImage;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
import lombok.extern.slf4j.Slf4j;
|
||||
|
||||
import stirling.software.proprietary.formdetection.model.ModelCatalogEntry;
|
||||
|
||||
/**
|
||||
* Pure pre/post-processing for a YOLO-style detector, driven entirely by the {@link
|
||||
* ModelCatalogEntry} spec. The browser pipeline mirrors this exactly so both inference paths agree.
|
||||
*
|
||||
* <p>Coordinate spaces: {@code preprocess} maps the source bitmap into the model's NxN input;
|
||||
* {@code decode} reads raw model output (boxes in input-pixel space), thresholds, runs NMS, and
|
||||
* un-projects boxes back to the original bitmap-pixel space (top-left origin). Mapping to PDF
|
||||
* points is done separately by {@code CoordinateMapper}.
|
||||
*/
|
||||
@Slf4j
|
||||
public final class Yolo {
|
||||
|
||||
private Yolo() {}
|
||||
|
||||
/** Normalised model input plus the transform needed to invert it. */
|
||||
public record Preprocessed(
|
||||
float[] chw,
|
||||
int inputSize,
|
||||
float scaleX,
|
||||
float scaleY,
|
||||
int padX,
|
||||
int padY,
|
||||
int srcW,
|
||||
int srcH) {}
|
||||
|
||||
/** Raw model output flattened to {@code data[i*d2 + j]} with dims {@code d1 x d2}. */
|
||||
public record RawOutput(float[] data, int d1, int d2) {}
|
||||
|
||||
/** A detection in original bitmap-pixel space, top-left origin. */
|
||||
public record Detection(int classId, float score, float x, float y, float w, float h) {}
|
||||
|
||||
/** Letterbox/stretch-resize, normalise and lay out as NCHW float32. */
|
||||
public static Preprocessed preprocess(byte[] rgba, int srcW, int srcH, ModelCatalogEntry spec) {
|
||||
int n = spec.getInputSize();
|
||||
boolean letterbox = !"stretch".equalsIgnoreCase(spec.getResizeMode());
|
||||
|
||||
float scaleX;
|
||||
float scaleY;
|
||||
int padX;
|
||||
int padY;
|
||||
int drawW;
|
||||
int drawH;
|
||||
if (letterbox) {
|
||||
float scale = Math.min((float) n / srcW, (float) n / srcH);
|
||||
drawW = Math.max(1, Math.round(srcW * scale));
|
||||
drawH = Math.max(1, Math.round(srcH * scale));
|
||||
padX = (n - drawW) / 2;
|
||||
padY = (n - drawH) / 2;
|
||||
scaleX = scale;
|
||||
scaleY = scale;
|
||||
} else {
|
||||
drawW = n;
|
||||
drawH = n;
|
||||
padX = 0;
|
||||
padY = 0;
|
||||
scaleX = (float) n / srcW;
|
||||
scaleY = (float) n / srcH;
|
||||
}
|
||||
|
||||
int[] pad = spec.getPadColor();
|
||||
Color fill =
|
||||
new Color(
|
||||
clampByte(pad != null && pad.length > 0 ? pad[0] : 114),
|
||||
clampByte(pad != null && pad.length > 1 ? pad[1] : 114),
|
||||
clampByte(pad != null && pad.length > 2 ? pad[2] : 114));
|
||||
|
||||
BufferedImage canvas = new BufferedImage(n, n, BufferedImage.TYPE_INT_RGB);
|
||||
Graphics2D g = canvas.createGraphics();
|
||||
try {
|
||||
g.setRenderingHint(
|
||||
RenderingHints.KEY_INTERPOLATION, RenderingHints.VALUE_INTERPOLATION_BILINEAR);
|
||||
g.setColor(fill);
|
||||
g.fillRect(0, 0, n, n);
|
||||
g.drawImage(rgbaToImage(rgba, srcW, srcH), padX, padY, drawW, drawH, null);
|
||||
} finally {
|
||||
g.dispose();
|
||||
}
|
||||
|
||||
boolean bgr = "bgr".equalsIgnoreCase(spec.getChannelOrder());
|
||||
float[] mean = orZeros(spec.getNormMean());
|
||||
float[] std = orOnes(spec.getNormStd());
|
||||
int plane = n * n;
|
||||
float[] chw = new float[3 * plane];
|
||||
int[] px = canvas.getRGB(0, 0, n, n, null, 0, n);
|
||||
for (int i = 0; i < plane; i++) {
|
||||
int rgb = px[i];
|
||||
float r = ((rgb >> 16) & 0xFF) / 255f;
|
||||
float gg = ((rgb >> 8) & 0xFF) / 255f;
|
||||
float b = (rgb & 0xFF) / 255f;
|
||||
float c0 = bgr ? b : r;
|
||||
float c1 = gg;
|
||||
float c2 = bgr ? r : b;
|
||||
chw[i] = (c0 - mean[0]) / std[0];
|
||||
chw[plane + i] = (c1 - mean[1]) / std[1];
|
||||
chw[2 * plane + i] = (c2 - mean[2]) / std[2];
|
||||
}
|
||||
return new Preprocessed(chw, n, scaleX, scaleY, padX, padY, srcW, srcH);
|
||||
}
|
||||
|
||||
/** Decode raw output, threshold, NMS, and un-project to original bitmap pixels. */
|
||||
public static List<Detection> decode(
|
||||
RawOutput out, ModelCatalogEntry spec, Preprocessed pre, float scoreThreshold) {
|
||||
int numClasses = spec.getClassNames() == null ? 0 : spec.getClassNames().size();
|
||||
if (numClasses == 0) {
|
||||
return List.of();
|
||||
}
|
||||
boolean obj = spec.isHasObjectness();
|
||||
boolean ncFirst = !"anchors_first".equalsIgnoreCase(spec.getOutputLayout());
|
||||
int channels = ncFirst ? out.d1() : out.d2();
|
||||
int anchors = ncFirst ? out.d2() : out.d1();
|
||||
int expected = 4 + (obj ? 1 : 0) + numClasses;
|
||||
if (channels < expected) {
|
||||
log.warn(
|
||||
"ONNX output channel count {} < expected {} (4 + obj + {} classes); skipping",
|
||||
channels,
|
||||
expected,
|
||||
numClasses);
|
||||
return List.of();
|
||||
}
|
||||
int classOffset = 4 + (obj ? 1 : 0);
|
||||
float[] data = out.data();
|
||||
|
||||
List<Detection> dets = new ArrayList<>();
|
||||
for (int a = 0; a < anchors; a++) {
|
||||
float objScore = obj ? at(data, ncFirst, anchors, channels, 4, a) : 1f;
|
||||
int bestClass = -1;
|
||||
float bestScore = 0f;
|
||||
for (int c = 0; c < numClasses; c++) {
|
||||
float s = at(data, ncFirst, anchors, channels, classOffset + c, a) * objScore;
|
||||
if (s > bestScore) {
|
||||
bestScore = s;
|
||||
bestClass = c;
|
||||
}
|
||||
}
|
||||
if (bestClass < 0 || bestScore < scoreThreshold) {
|
||||
continue;
|
||||
}
|
||||
float cx = at(data, ncFirst, anchors, channels, 0, a);
|
||||
float cy = at(data, ncFirst, anchors, channels, 1, a);
|
||||
float w = at(data, ncFirst, anchors, channels, 2, a);
|
||||
float h = at(data, ncFirst, anchors, channels, 3, a);
|
||||
float x1 = cx - w / 2f;
|
||||
float y1 = cy - h / 2f;
|
||||
float ox = (x1 - pre.padX()) / pre.scaleX();
|
||||
float oy = (y1 - pre.padY()) / pre.scaleY();
|
||||
float ow = w / pre.scaleX();
|
||||
float oh = h / pre.scaleY();
|
||||
// clamp to the source bitmap
|
||||
float cxl = Math.max(0, Math.min(ox, pre.srcW()));
|
||||
float cyl = Math.max(0, Math.min(oy, pre.srcH()));
|
||||
ow = Math.max(0, Math.min(ow, pre.srcW() - cxl));
|
||||
oh = Math.max(0, Math.min(oh, pre.srcH() - cyl));
|
||||
if (ow <= 0 || oh <= 0) {
|
||||
continue;
|
||||
}
|
||||
dets.add(new Detection(bestClass, bestScore, cxl, cyl, ow, oh));
|
||||
}
|
||||
return nms(dets, spec.getNms(), spec.getIou());
|
||||
}
|
||||
|
||||
private static float at(
|
||||
float[] data, boolean ncFirst, int anchors, int channels, int c, int a) {
|
||||
return ncFirst ? data[c * anchors + a] : data[a * channels + c];
|
||||
}
|
||||
|
||||
private static List<Detection> nms(List<Detection> dets, String mode, float iouThreshold) {
|
||||
if (dets.size() < 2 || "none".equalsIgnoreCase(mode)) {
|
||||
return dets;
|
||||
}
|
||||
boolean classAgnostic = mode != null && mode.toLowerCase().contains("agnostic");
|
||||
List<Detection> sorted = new ArrayList<>(dets);
|
||||
sorted.sort((x, y) -> Float.compare(y.score(), x.score()));
|
||||
boolean[] removed = new boolean[sorted.size()];
|
||||
List<Detection> keep = new ArrayList<>();
|
||||
for (int i = 0; i < sorted.size(); i++) {
|
||||
if (removed[i]) {
|
||||
continue;
|
||||
}
|
||||
Detection di = sorted.get(i);
|
||||
keep.add(di);
|
||||
for (int j = i + 1; j < sorted.size(); j++) {
|
||||
if (removed[j]) {
|
||||
continue;
|
||||
}
|
||||
Detection dj = sorted.get(j);
|
||||
if (!classAgnostic && di.classId() != dj.classId()) {
|
||||
continue;
|
||||
}
|
||||
if (iou(di, dj) > iouThreshold) {
|
||||
removed[j] = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
return keep;
|
||||
}
|
||||
|
||||
private static float iou(Detection a, Detection b) {
|
||||
float ax2 = a.x() + a.w();
|
||||
float ay2 = a.y() + a.h();
|
||||
float bx2 = b.x() + b.w();
|
||||
float by2 = b.y() + b.h();
|
||||
float ix1 = Math.max(a.x(), b.x());
|
||||
float iy1 = Math.max(a.y(), b.y());
|
||||
float ix2 = Math.min(ax2, bx2);
|
||||
float iy2 = Math.min(ay2, by2);
|
||||
float iw = Math.max(0, ix2 - ix1);
|
||||
float ih = Math.max(0, iy2 - iy1);
|
||||
float inter = iw * ih;
|
||||
float union = a.w() * a.h() + b.w() * b.h() - inter;
|
||||
return union <= 0 ? 0 : inter / union;
|
||||
}
|
||||
|
||||
private static BufferedImage rgbaToImage(byte[] rgba, int w, int h) {
|
||||
BufferedImage img = new BufferedImage(w, h, BufferedImage.TYPE_INT_ARGB);
|
||||
int[] px = new int[w * h];
|
||||
int pixels = Math.min(w * h, rgba.length / 4);
|
||||
for (int i = 0; i < pixels; i++) {
|
||||
int r = rgba[i * 4] & 0xFF;
|
||||
int g = rgba[i * 4 + 1] & 0xFF;
|
||||
int b = rgba[i * 4 + 2] & 0xFF;
|
||||
int a = rgba[i * 4 + 3] & 0xFF;
|
||||
px[i] = (a << 24) | (r << 16) | (g << 8) | b;
|
||||
}
|
||||
img.setRGB(0, 0, w, h, px, 0, w);
|
||||
return img;
|
||||
}
|
||||
|
||||
private static int clampByte(int v) {
|
||||
return Math.max(0, Math.min(255, v));
|
||||
}
|
||||
|
||||
private static float[] orZeros(float[] v) {
|
||||
return v != null && v.length >= 3 ? v : new float[] {0f, 0f, 0f};
|
||||
}
|
||||
|
||||
private static float[] orOnes(float[] v) {
|
||||
return v != null && v.length >= 3 ? v : new float[] {1f, 1f, 1f};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
package stirling.software.proprietary.formdetection.model;
|
||||
|
||||
/**
|
||||
* One detected form field. This is the shared schema returned by both the server detect endpoint
|
||||
* and the in-browser pipeline so the two paths are interchangeable.
|
||||
*
|
||||
* @param type AcroForm field type (text|checkbox|radio|signature)
|
||||
* @param page zero-based page index
|
||||
* @param rectInPdfPoints rectangle in PDF points (bottom-left origin)
|
||||
* @param confidence detection confidence 0-1
|
||||
*/
|
||||
public record DetectedField(String type, int page, RectPt rectInPdfPoints, double confidence) {
|
||||
|
||||
/** Rectangle in PDF points, bottom-left origin (PDF user space). */
|
||||
public record RectPt(double x, double y, double w, double h) {}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
package stirling.software.proprietary.formdetection.model;
|
||||
|
||||
import java.util.Locale;
|
||||
|
||||
/** Lifecycle state of the Auto Form Detection model install. */
|
||||
public enum FormDetectionStatus {
|
||||
NOT_INSTALLED,
|
||||
DOWNLOADING,
|
||||
VERIFYING,
|
||||
READY,
|
||||
FAILED;
|
||||
|
||||
/** Lowercase wire form sent to the frontend, e.g. {@code not_installed}. */
|
||||
public String wire() {
|
||||
return name().toLowerCase(Locale.ROOT);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
package stirling.software.proprietary.formdetection.model;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import lombok.Data;
|
||||
|
||||
/**
|
||||
* One installable form-detection model plus the data-driven pre/post-processing spec the inference
|
||||
* pipeline needs. The exact same numeric values are surfaced to the browser (via the model status
|
||||
* endpoint) so the in-browser and server inference paths stay equivalent.
|
||||
*
|
||||
* <p>NOTE: the pipeline-spec defaults below follow common Ultralytics-YOLO conventions. The precise
|
||||
* values for a given model (input size, resize mode, channel order, output layout, NMS, class
|
||||
* indices) MUST be verified against the actual exported {@code .onnx} before that entry's {@code
|
||||
* onnxUrl}/{@code sha256} are populated. An entry with a blank {@code onnxUrl} or {@code sha256} is
|
||||
* not installable, which keeps the distribution shippable without any bundled model.
|
||||
*/
|
||||
@Data
|
||||
public class ModelCatalogEntry {
|
||||
|
||||
// --- Identity / distribution -------------------------------------------------
|
||||
private String id;
|
||||
private String displayName;
|
||||
private String description;
|
||||
private String license;
|
||||
private long sizeBytes;
|
||||
|
||||
/** Direct download URL of the .onnx. Blank = not yet available (install is rejected). */
|
||||
private String onnxUrl;
|
||||
|
||||
/** Lower-hex SHA-256 of the .onnx. Blank = not yet available (install is rejected). */
|
||||
private String sha256;
|
||||
|
||||
// --- Pre-processing (parity-critical, mirrored by the browser) ---------------
|
||||
/** Square model input edge in pixels. */
|
||||
private int inputSize = 1024;
|
||||
|
||||
/** "letterbox" (aspect-preserving pad) or "stretch" (resize to square). */
|
||||
private String resizeMode = "letterbox";
|
||||
|
||||
/** RGB letterbox pad colour. */
|
||||
private int[] padColor = {114, 114, 114};
|
||||
|
||||
/** "rgb" or "bgr" channel order fed to the model. */
|
||||
private String channelOrder = "rgb";
|
||||
|
||||
/**
|
||||
* Per-channel mean subtracted after dividing the raw byte by 255 ({@code (raw/255 -
|
||||
* mean)/std}).
|
||||
*/
|
||||
private float[] normMean = {0f, 0f, 0f};
|
||||
|
||||
/** Per-channel std applied after mean subtraction. */
|
||||
private float[] normStd = {1f, 1f, 1f};
|
||||
|
||||
// --- Post-processing (parity-critical) ---------------------------------------
|
||||
/** "nc_first" => output [1, 4+nc, anchors]; "anchors_first" => [1, anchors, 4+nc]. */
|
||||
private String outputLayout = "nc_first";
|
||||
|
||||
/** True if an objectness score column precedes the class scores (YOLOv5 style). */
|
||||
private boolean hasObjectness = false;
|
||||
|
||||
/** Class index -> label. */
|
||||
private List<String> classNames = List.of("text", "choice", "signature");
|
||||
|
||||
/** Class index -> AcroForm field type (text|checkbox|radio|signature). */
|
||||
private List<String> classFieldTypes = List.of("text", "checkbox", "signature");
|
||||
|
||||
private float scoreThreshold = 0.25f;
|
||||
|
||||
/** "none", "classAgnostic" or "perClass". */
|
||||
private String nms = "perClass";
|
||||
|
||||
private float iou = 0.45f;
|
||||
}
|
||||