Compare commits

...
Author SHA1 Message Date
Connor Yoh afe11a85a1 PAYG usage card: review nits on the pending-sync note
- Gate the unsynced-only note on pendingUnits (not meterUnits) and interpolate
  pendingUnits directly, so the "pending sync from linked instances" wording is
  always exact — robust to a hypothetical zero-doc entry carrying synced units.
- Pluralise the key (unitsPending_one/_other + count) so it reads "1 meter unit"
  not "1 meter units", matching the project's _one/_other convention.

(Headline still shows the synced PDF count — "0 PDFs" in the unsynced-only state
is by design: count dimension vs meter units, with the note explaining the gap.)
2026-07-13 09:48:54 +01:00
Connor Yoh 4648cb8b41 PAYG usage card: address review follow-ups (avg/PDF, empty-state, unique wording)
Follow-up to #6957, from the review:

1. avg-per-PDF no longer blends unsynced units over synced-only docs. It now
   divides SYNCED units (spendUnitsThisPeriod) by synced docs, so numerator and
   denominator cover the same population; combined-billing pendingUnits (units-
   only, no doc count) no longer inflates the average. The "meter units" figure
   still shows synced+pending (total current usage).

2. Empty-state: docs==0 with pending meter units (combined-billing, unsynced-
   only) now shows a "{n} meter units pending sync from linked instances" note
   instead of a bare "0 PDFs" headline with a count-less summary and no split.
   New i18n key + a UnsyncedOnly story variant.

3. uniquePdfsThisPeriod wording softened to be accurate: the fingerprint is over
   a charge's whole input SET, so the same file reused across different groupings
   (standalone vs a later merge) counts per grouping — a close approximation,
   exact for the single-input common case. Updated the FE type doc + the
   WalletLedgerEntry.document_fingerprint javadoc.
2026-07-10 15:35:42 +01:00
Anthony StirlingandGitHub d06a367b87 SaaS role-based login landing (team leads → Processor) (#6960) 2026-07-10 15:23:00 +01:00
ConnorYohandGitHub ce6abe6e23 PAYG: size-scaled units + per-input-file PDF count + run-id grouping (#6957)
Reworks the Processor (PAYG) meter to **size-scaled units** while
keeping a true **PDF count** visible and distinct from units, and
replaces the fragile content+time lineage grouping with **explicit
per-run grouping**. Built as one PR across three slices.

> Status: **all three slices committed + verified.** `:saas` payg suite
green (418 tests, 0 failures); FE green (typecheck 0, 1260 tests, lint
0, format 0). Remaining before it takes effect in prod: run the
size-scaled default-policy SQL (below) in the Supabase SQL editor +
attach the $0.01/unit Stripe price.

## Model (what we're implementing)
- **Size scaling**: 1 unit per 50 MiB (bytes only, no page charge, no
cap). *(policy-row config, applied separately via SQL.)*
- **Charge = number of input files**: split (1→N outputs) = 1 charge;
merge (N→1) = N charges. `doc_count` = input files, fixed at open;
joined steps add 0.
- **Grouping by run id, not time**: a pipeline/policy/AI run = one
`run_id`; its tool sub-steps group into one charge (content-lineage
still maps split/merge journeys *within* the run). Two separate runs on
identical bytes = two charges. The 5-min window survives only as a
stale-job janitor.
- **10-tool split kept**: within a run's single-file lineage, an 11th
tool run opens a 2nd charge (step limit 10).
- **Count vs units surfaced**: usage page shows unique PDFs,
per-category (automation/AI/API) counts + units, and how many PDFs hit a
size multiplier with avg units/PDF.

## Slice 1 — run-id grouping (behavioural core)
- `AutomationRunContext` (common) — thread-scoped run id.
- `InternalApiClient` — stamps `X-Stirling-Run-Id`.
- Orchestrators open a run scope **on the worker thread that
dispatches** (async-safe): `PipelineProcessor.runPipelineAgainstFiles`,
`PolicyEngine.runToCompletion` (uses `run.getRunId()`),
`AiWorkflowService.orchestrate`.
- `ChargeContext` + `JobContext`: add `runId`; the charge interceptor
reads the header.
- `JobService.joinOrOpen`: `runId == null` → always open fresh
(standalone never joins); non-null → match scoped to the same `run_id`.
`JpaJobLineageStore`/`JobArtifactHashRepository`: add `run_id` filter to
the match query. Step-limit 10 unchanged.

## Slice 2 — doc_count + document_fingerprint
- V33 migration + entity fields.
- `JobService.openFresh`: set `docCount = inputs.size()`, compute
`document_fingerprint` from input signatures, and denormalise both onto
the DEBIT row in `JobChargeService.recordLedgerDebit`.

## Slice 3 — usage analytics API + FE
- `WalletLedgerRepository`: per-category `SUM(units)` +
`SUM(doc_count)`, `COUNT(DISTINCT document_fingerprint)`, and count of
rows whose units exceed their doc_count (a size multiplier fired), over
the period.
- `WalletSnapshotResponse` + `PaygWalletController`: add `categoryDocs`,
`docsProcessedThisPeriod`, `uniquePdfsThisPeriod`,
`sizeMultiplierPdfsThisPeriod`.
- FE `types.ts` + `PdfsProcessedCard` + `useWallet` + `walletFixtures` +
i18n: headline is the **PDF count**; a summary line shows "{unique}
unique · {units} meter units · {avg} avg units/PDF"; the split bar is
per-category PDF counts; a size-multiplier line shows how many PDFs
scaled. Count is separated from meter units so a 5-unit large PDF reads
as "1 PDF, 5 units".

## Config (out of PR — run in the Supabase SQL editor)
Wrap in one transaction. The partial-unique `is_default` index only
allows one default, so the old default is flipped off **before** the new
one is inserted. The new policy carries the prior default's
`free_tier_units` forward (change the literal if the launch grant should
differ).
```sql
BEGIN;

-- 1) flip default off the current policy + close its effective window
UPDATE stirling_pdf.pricing_policy
SET is_default = FALSE, effective_to = now()
WHERE is_default = TRUE;

-- 2) new default: 1 unit / 5 MiB, no page charge, no scaling cap.
--    free_tier_units carried from whatever the last policy granted (COALESCE→0).
INSERT INTO stirling_pdf.pricing_policy
  (version, effective_from, doc_pages_per_unit, doc_bytes_per_unit,
   min_charge_units, file_unit_cap, free_tier_units, is_default, notes, created_by)
VALUES
  ('v2-size-scaled-2026-07', now(),
   2147483647,        -- doc_pages_per_unit = INT_MAX → pages never drive units
   52428800,          -- doc_bytes_per_unit = 50 MiB → +1 unit per 50 MiB
   1,                 -- min_charge_units
   2147483647,        -- file_unit_cap = INT_MAX → no cap on size scaling
   COALESCE((SELECT free_tier_units FROM stirling_pdf.pricing_policy
             ORDER BY effective_from DESC LIMIT 1), 0),
   TRUE, 'Size-scaled: 1 unit/5MiB, bytes only, no cap', 'connor');

-- 3) per-source step limits: standalone ops = own charge; pipelines split at 10
INSERT INTO stirling_pdf.pricing_policy_step_limit (policy_id, job_source, step_limit)
SELECT p.policy_id, s.src, s.lim
FROM stirling_pdf.pricing_policy p
CROSS JOIN (VALUES
  ('WEB',1),('API',1),('DESKTOP_APP',1),('LINKED_INSTANCE',1),('PIPELINE',10)
) AS s(src, lim)
WHERE p.version = 'v2-size-scaled-2026-07';

-- 4) attach the $0.01/unit Stripe price (you handle the real price id)
INSERT INTO stirling_pdf.pricing_policy_stripe_price (policy_id, stripe_price_id)
SELECT policy_id, 'price_XXXXXXXX'
FROM stirling_pdf.pricing_policy WHERE version = 'v2-size-scaled-2026-07';

COMMIT;
```
Note: the `free_tier_units` subquery reads the most-recent policy
*before* the insert — run it as written (the new row doesn't exist yet
at step 2's SELECT).

## Self-hosted parity — tracked follow-up (not in this PR)
Combined-billing (`stirling.billing.account-link.enabled`) is a
**separate metering engine** (`app/proprietary/accountlink` —
`UsageMeterService`/`LocalUsageService`/`UsageSyncService`). The unit
*math* is shared (`DocumentUnitCalculator`), so size scaling matches
once the policy is pushed. But run-id grouping, `doc_count`, and
fingerprints must be mirrored there, and the usage-sync protocol
extended to report counts/fingerprints, before the self-hosted usage
page shows the same breakdown. Frozen/deferred, so this PR does SaaS;
self-hosted mirrors when it ships.
2026-07-10 13:38:22 +00:00
ConnorYohandGitHub ece3562dc9 Portal: team-scoped Free PDF Editors usage card for SaaS (#6924)
## What

Phase 2 of the Free PDF Editors usage card (self-hosted shipped in
#6919): make it work on **SaaS**, where one backend serves many teams so
every figure must be scoped to the **caller's team**.

| Metric | SaaS (per team) |
|---|---|
| **Editors deployed** | team member count (`team_memberships`) |
| **Active this month** | distinct members with a free-UI
(`source='WEB'`, non-`UI_DATA`) audit event in 30d, clamped ≤ deployed |
| **PDFs edited** | the team's cumulative free-UI
`PDF_PROCESS`+`FILE_OPERATION` events |

Cost stays `$0`; uncomputable figures render **N/A**.

## Backend

- **Gate the self-hosted controller** `@Profile("!saas")` — its counts
are server-wide, which would leak across tenants on SaaS. New
team-scoped `SaasFleetUsageController` `@Profile("saas")` owns the same
`/api/v1/usage/fleet-stats` path (mutually exclusive profiles → no
mapping conflict).
- **Team resolution** mirrors `PaygWalletController`:
`AuthenticationUtils.getCurrentUser(auth, userRepo)` →
`TeamMembershipRepository.findPrimaryMembership` → members via
`findByTeamId`. `@PreAuthorize("isAuthenticated()")` (team leaders
aren't global admins; any member sees their own team's totals).
- **Audit → team join**: on SaaS the audit `principal` is the user's
email and `User.username == email`, so principals join cleanly to a
team's member usernames (no hashing — only raw-JWT/over-long principals
get hashed). Two new `principal IN` count queries do the filtering,
served by the `(source, timestamp, principal)` index from #6919.
- Billing/ledger is deliberately **not** used — it only records billable
ops; free-editor activity comes from audit (same `source='WEB'` signal
as self-hosted).
- `null`→N/A when EE auditing < STANDARD; 401 on no-auth; empty-fleet
guard for the (post-migration-shouldn't-happen) teamless caller.

## Frontend

- New `src/portal-saas/api/fleetStats.ts` (rides the `@portal/*` cascade
from #6900) reads via **`apiClient.saas`** — the Supabase JWT the SaaS
backend uses to resolve the team. Re-exports `FleetStats` via
`@portal-proprietary`. **The card and `useAsync` hook are untouched.**

## Tests

`STIRLING_FLAVOR=saas` build green — `:proprietary` + `:saas` compile,
`SaasFleetUsageControllerTest` (team scoping, audit-off→null, clamp,
no-team→empty, unauth→401) and the existing suites pass; spotless clean.

## Notes

- Requires SaaS auditing at STANDARD (it is) — else N/A.
- Depends on #6900 (merged) for the portal-saas override layer and #6919
(merged) for the audit `source` column + DTO.
2026-07-10 13:28:33 +00:00
James BruntonandGitHub 84d4455682 Add virtual Editor source (#6959)
# Description of Changes
Adds Editor source permanently available in the Sources list. Excludes
it from the Pipelines list of available sources currently because it's
not a real source on the backend, so attempting to connect to it causes
an error. It'd be nice to extend in the future to be able to set up
policies in the editor from the pipelines page, but this'll do for now.
2026-07-10 13:27:40 +00:00
ConnorYohandGitHub b9a7f2083b Portal: realign home hero to the simplified marketing card (#6956)
## What

Reworks the free-tier home hero (`WelcomeBanner` + `SetupChecklist`) to
match marketing's reworked top card: a compact product header over
numbered getting-started steps, dropping the marketing chrome.

## aim
attachments/assets/75a80e5f-119e-46bb-80e7-fc4b9a62e5b6" />
<img width="1098" height="646" alt="01-aim-marketing-demo"
src="https://github.com/user-attachments/assets/681ca1b8-219e-4afe-9748-89435aafd440"
/>

## old hero
<img width="1800" height="1338" alt="02-before-old-hero"
src="https://github.com/user-attachments/assets/a396cec0-4752-4ac0-9951-9a49f9d50ea7"
/>


## new screenshots

<img width="1800" height="626" alt="03-after-onboarding-card"
src="https://github.com/user-attachments/assets/59332111-6c49-438d-af6d-200d99bf0f8f"
/>
<img width="1800" height="180" alt="04-after-deployed-header"
src="https://github.com/user-attachments/assets/ea631e6c-7878-4159-aaf7-1f0c2bd795ce"
/>
<img width="1024" height="1396" alt="05-after-install-modal-list"
src="https://github.com/user-attachments/assets/125d0a11-d799-40f0-bd8b-42f5dedcfe6d"
/>
<img width="1024" height="858" alt="06-after-install-modal-docker"
src="https://github.com/user-

## Changes

- **Compact dark header:** brand mark + "PDF Editor" + social-proof
stats (`30M downloads · 60+ PDF operations · Free forever`) + a single
**Open in browser** CTA (→ `EDITOR_URL`).
- **Dropped** the decorative editor mock, marketing
title/subtitle/"Open-source" badge/perks, the two extra banner buttons,
and the checklist's dismiss/progress/done tracking.
- **Numbered nav steps** (①②③) — each opens its in-app surface:

| # | Step | Goes to | Change |
|---|------|---------|--------|
| ① | Download the editor | `editor` view | was an external
`stirling.com/download` link → now in-app |
| ② | Confirm your policies | `policies` view | live active/recommended
counts retained |
| ③ | Invite teammates | `users` view | **replaces** "Connect your
sources" (sources dropped to match the demo) |

- **Enterprise rung** unchanged (Start Trial / Get Quote → procurement).

## Notes

- **Shared hero** — self-hosted sees it too (per decision).
- **One deliberate deviation from the demo:** the header CTA is blue
(brand primary) rather than the demo's white button. Trivial to flip —
say the word.
- Behaviour change: the hero is now a quick-start (navigational) rather
than a completion checklist — the dismiss control + per-step done chips
are gone to match the demo.
- Supersedes the incremental #6944 ("add Open in browser" 3-button
version) — that can be closed in favour of this.
- Portal `tsc` clean; `unusedTranslations` green (removed orphaned
welcome/onboarding keys, added the new ones).
2026-07-10 13:20:18 +00:00
Reece BrowneandGitHub b36f3e0875 Remove unused portal UI (#6949)
Removes some cluttered/unused UI from the portal:

- Search bar in the header
- The top bar entirely (breadcrumb, notification bell, plan switcher,
user menu)
- The plan/usage indicator in the sidebar footer
- The floating assistant badge

UI only. Where a component isn't deleted it's just no longer rendered,
so anything here is easy to restore.
2026-07-10 13:16:57 +00:00
ConnorYohandGitHub e4379184b5 fix(portal): translate policy category labels in PolicySummary (#6964)
## What

The portal's **"What runs on your PDFs"** table (`PolicySummary`)
rendered raw i18n keys instead of text:

- `portal.policies.categories.ingestion.label` / `.desc`
- `portal.policies.categories.security.label` / `.desc`
- …and the other three categories (compliance, routing, retention)

## Why it broke

[#6910 "Remove in-app portal
mocks"](https://github.com/Stirling-Tools/Stirling-PDF/pull/6910) moved
the policy catalogue to real data and converted each category's
`label`/`desc` (and each config's `summary`) into **i18n keys** — see
the `// values are i18n keys — render with t()` note in
`api/policies.ts`. Every consumer was updated to call `t()`
(`PolicyCategoryCard`, `PolicyDetailPanel`, `PolicySetupWizard`)… except
`PolicySummary`, which was not part of that PR and kept rendering the
fields verbatim.

The translation keys themselves already exist in
`en-US/translation.toml` (`[portal.policies.categories.*]`) — nothing
was missing, they just weren't being looked up.

## Fix

Wrap the values in `t()` in `PolicySummary.tsx` (the `t` from
`useTranslation` was already in scope):
- category `label` / `desc` in the Policy column
- `config.summary` in the Active-rule column (same keyed-value
treatment, latent until a policy is active)

## Test plan

- [ ] Open the portal Home / policies summary → each row shows the
translated category name + description (e.g. "Ingestion" / "Classify
documents…") instead of a dotted key.
- [ ] A row with an active policy shows its translated rule summary in
the Active rule column.
2026-07-10 13:05:46 +00:00
James BruntonandGitHub 5ccb56da2d Add S3 policy source (#6948)
# Description of Changes
* Adds an Amazon S3 Source & Output
* Removes folder source from SaaS
* Some miscellaneous UX fixes around pipelines
2026-07-10 12:19:41 +00:00
Reece BrowneandGitHub 16f589448d Remove the policies management surface from the editor sidebar (#6932)
## What

Removes the policy **management** surface from the editor's right rail —
the Policies list above Tools, the open-policy detail takeover, and the
collapsed-rail policy icons — along with the whole UI tree only they
used: the setup wizard and its tool-config steps (PII / redact /
watermark), the detail panel, delete modal, selection store,
enforcement-queue status chip, activity/stats derivation, the catalog
hook, their i18n keys, dead types, and the admin-gate spec that tested
the wizard flow.

**Enforcement is untouched.** Auto-run on upload, the viewer blocking
overlay, exit-point blocking, file badges, and export-time enforcement
all stay. `usePoliciesEnabled` moves to its own module (core stub /
proprietary / desktop shadow with the SaaS-connection check) since it
still gates mounting the headless `PolicyAutoRunController` from the
rail.

## Why

Policies are configured in the admin portal now
(`src/portal/views/Policies.tsx`). Keeping a second management UI in the
editor rail meant two surfaces to maintain for one feature; the editor
only needs to *enforce*.

## Notes for review

- The rail UI lived in the shared `core` `RightSidebar`, so this removes
it from every build flavour at once; the deleted `PoliciesSidebar`
module existed at the core (stub) / proprietary / desktop alias layers
and all three are gone.
- Every deleted module was verified to have zero remaining importers;
near-misses that stay: `enforcementQueue` (used by export enforcement),
`poll` (test-imported), `usePolicies` (used by auto-run).
- Net −3,900 lines.

## Testing

- `task frontend:check` green: typecheck, ESLint + dpdm, Prettier, all
1,196 tests.
- All build-variant typechecks pass (core / proprietary / saas /
desktop).
2026-07-10 11:55:59 +00:00
ConnorYohandGitHub 75ea3c9a1f Portal procurement: pricing realignment, combined accept flow, licence & invoice fixes (#6946)
## What this PR does

Brings the enterprise procurement flow in line with the new D71 pricing,
tidies up the buyer journey, and fixes a handful of things we found
testing it end to end.

### Pricing
- Priced on the new run-based model (per PDF, per policy), USD only.
Dropped the old currency picker.
- Added the policy posture choice (Essentials / Governed / Regulated)
and show roughly how many policies each covers (~2 / ~4 / ~7).
- The live estimate in the quote builder now matches the real quote the
backend produces.
- Contracts renew each year with a fixed 3% increase. The agreement
shows this plus the first renewal figure, and we save that figure on the
quote so it can't drift later.

### Trial and journey
- Starting a trial now asks for your deployment (Cloud / Self-hosted /
Air-gapped) and team size up front, and that seeds the quote.
- Quote and agreement are now one step: you review the quote and the
agreement together and click "Accept & subscribe" once. No more
accepting a quote and then separately signing.
- "Start a trial" on the home page opens the setup popup right there
instead of sending you off to another page.
- The calculator asks for number of users again and works the volume out
from that.
- Removed the demo-only buttons (reset, simulate payment) and the "Key
documents" button (it wasn't real).
- The licence key now lives behind its own "Licence key" button instead
of being shown inside every popup.

### Air-gapped licence file
- Air-gapped teams can download their licence file (.lic) during the
trial, not only after they pay.
- The popup warns that a trial file needs re-downloading once the
agreement is done, because the file is a snapshot and doesn't refresh
itself the way the online key does.

### Fixes found while testing
- Accepting a quote now upgrades the licence from trial to full straight
away (it wasn't before).
- The "Download invoice" button keeps working after a page refresh (we
now save the invoice PDF link).
- Invoice line items read differently from each other instead of all
showing the same name.

### Notes for reviewers
- The matching backend changes (Stripe quote/accept functions, database
migrations) live in the Stirling-PDF-SaaS repo on `v3`. They ship when
we do the full v3 release.
- All checks are green.
2026-07-10 11:51:00 +00:00
EthanHealy01andGitHub 7529190587 fix(ui): shared Button content-sizing + padding props, and button call-site cleanups (#6914)
## Summary

A batch of shared **design-system** fixes (Button, SegmentedControl,
Chip, a new CarouselDots) and the consumer/call-site cleanups they
unlock, following the button consolidation (#6787). Also includes
dark-theme token alignment and some portal/auth polish that rides on the
same components.

The shared Button now sizes to its content instead of clipping it, gains
per-axis padding controls, and no longer misbehaves while loading or
disabled; several call sites are then migrated onto the proper component
APIs.

## Shared components (`core/ui`)

### Button
- **Content-driven height.** `--button-height` is now a `min-height`,
not a fixed cap. Single-line buttons still land exactly on the shared
control-height scale (pixel-aligned with `ActionIcon` /
`SegmentedControl`), while taller content — wrapped labels, stacked
title + subtitle rows — grows the button instead of being clipped
mid-glyph. Short content is re-centered with `align-content`,
**without** overriding the root `display`, so a consumer's own layout
(e.g. a full-width list row) isn't disturbed.
- **Padding props.** New `p` / `px` / `py` props
(`none`/`xs`/`sm`/`md`/`lg`/`xl`) override the size-based padding per
axis. Vertical padding is applied through a `--sui-btn-py` CSS variable,
so consumers can also set it from their own class.
- **Loading no longer collapses.** A `fullWidth` button is never treated
as icon-only, so an execute button whose label is momentarily absent
while files hydrate (e.g. `ScopedOperationButton`) keeps its full width
with a centered spinner instead of shrinking to an icon-sized square for
a split second.
- **Disabled in dark mode.** A disabled *primary* button keeps a muted
version of its own accent fill (`opacity: 0.55`) instead of Mantine's
near-black `--mantine-color-disabled`, which blended into dark surfaces
and made the button all but disappear. Loading spinners are excluded so
they stay full-strength.

No breaking API changes — buttons that don't opt in render exactly as
before.

### SegmentedControl
- Fixed a bug where a segment marked `disabled` that also happened to be
the currently-selected value was rendered disabled, leaving the active
segment un-selectable/greyed. A disabled option is now only disabled
when it isn't the current value.

### CarouselDots (new)
- New shared dots indicator component (with Storybook story), used by
the login carousel.

### Chip / theme
- Dark-theme tokens in `theme.css` aligned to the portal's `tokens.css`
so the editor and portal (Processor) dark modes stop drifting (chrome
surfaces lift off the darker canvas); plus a Chip dark-mode styling fix
and a small `mantineTheme` cleanup.

## Consumer / call-site cleanups

- **Compare** tool: the swap control is now a regular shared Button
placed **between** the Original and Edited file cards (the bespoke
full-height vertical swap button and its CSS were removed), and the file
cards fill the full available width.
- **Certificate format**: replaced the inline-styled buttons with clean
two-state (primary / secondary) buttons.
- **ToolPicker**: restored the label selectors that #6787 renamed to the
never-emitted `.sui-btn__label`, and fixed the sidebar-search row
clipping.
- **File sidebar**: "View all files" row fix; `FileSidebarFileItem`
migrated off `display:flex` + `gap` on the Button root (which no longer
reaches the nested label) onto `leftSection` / `rightSection` + a
stacked label.

## Portal / auth polish

- Portal button consolidation and styling across Header, SettingsModal,
Home, Infrastructure, ApiKeyCard, and PopularUseCases.
- **Login**: onboarding text now shows the default starting username /
password; login carousel uses the new CarouselDots; desktop OAuth
styling tweak.

## Verification

- Storybook: button sizes measure exactly on the control-height scale
and match `ActionIcon`; icon-only buttons stay square and centered;
`fullWidth` loading buttons hold full width; disabled dark-mode primary
buttons render as a muted accent rather than grey.
- Single-line buttons are pixel-identical before/after; only buttons
whose content previously overflowed a fixed height render differently
(they now fit rather than clip).
- `task frontend:lint` clean; typecheck shows only the pre-existing
third-party `node_modules` noise also present on `main`.
2026-07-10 10:26:47 +00:00
Anthony StirlingandGitHub b9f9f84907 Route portal Users page to SaasTeamController on SaaS via usersBackend seam (#6940)
## Why

The portal Users page worked on self-hosted but **403'd on SaaS**. It
called the proprietary admin endpoints (`/api/v1/user/admin/*`,
`/api/v1/team/*`, `ui-data/admin-settings`), all `hasRole('ADMIN')`.
SaaS users are always `ROLE_USER` (never `ROLE_ADMIN`), so those
endpoints reject them. This is the last SaaS-release blocker for the
portal.

## What

Route the SaaS build's Users page to the **existing**
`SaasTeamController` (invitation-based team management) - no new
backend. Done via a build-time flavor seam, mirroring the existing
`usersCapabilities` pattern.

- **New seam `@app/portal/usersBackend`** (interface in
`portal/api/usersBackend.ts`) with two impls resolved by the `@app/*`
alias:
- `proprietary/portal/usersBackend.ts` re-exports the existing
admin-endpoint functions - **self-hosted behaves exactly as before**.
- `saas/portal/usersBackend.ts` calls `SaasTeamController`
(`/api/v1/team/*`) via `apiClient.local` (already flavor-aware: SaaS
backend + Supabase JWT). Resolves the leader's team from `GET
/api/v1/team/my`, maps `TeamMemberDTO`/`InvitationDTO` onto the portal
`Member`/`PendingInvitation` types.
- **`manageInvitations` capability** (SaaS `true` / self-hosted `false`)
gates a new **Pending invitations** panel (list from `GET
/{teamId}/invitations`, Cancel via `DELETE
/api/v1/team/invitations/{id}`).
- **Remove re-enabled on SaaS** (was gated off): the roster remove
action now works at team scope against `DELETE
/{teamId}/members/{memberId}`, with a flavor-aware label ("Remove from
team" vs "Remove from org") and confirm copy.
- Invite (email) and rename routed through the seam (`POST /invite`,
`POST /{teamId}/rename`); `fetchAuthConfig` on SaaS is static (no
spurious admin-endpoint 403).
- **MSW handlers** (`mocks/handlers/teamSaas.ts`) mirror the controller
so the SaaS Users page is exercisable in mock mode. Registered in
`handlers` but deliberately **not** `embeddedDataHandlers` (would clash
with the editor's own `/api/v1/team/*` routes when portal shares its
origin).

## Constraints honoured

- No new backend endpoints - reuses `SaasTeamController`.
- Self-hosted path unchanged (proprietary impl re-exports the same
functions).
- No SaaS user is ever `ROLE_ADMIN` - `adminRole`/admin-only UI stay
hidden.

## Notes from an adversarial self-review (both fixed in this PR)

- Solo SaaS users' auto-created **personal team** now hides the Rename
control (the backend rejects renaming personal teams with 400) -
`isPersonal` threaded through `Team`/`TeamGroup`.
- Expired-but-still-`PENDING` invitations are filtered in the adapter,
and the expiry label no longer mislabels a just-expired invite as
"Expires today".

## Testing

- `task frontend:typecheck:all` - all 8 flavors pass.
- Portal vitest project: **122 passing** (added SaaS adapter +
shape-mapping tests via MSW, PendingInvitations panel, and
remove/manageInvitations/personal-team gating).
- `task frontend:lint` (ESLint `--max-warnings=0` + dpdm no circular
deps) and prettier clean.

## Open questions

- **Team resolution on SaaS**: I resolve the leader's single manageable
team (prefer a real non-personal team they lead). If a leader owns
multiple real teams, only the primary is shown - matches the "single
team" framing in the spec; flag if multi-team management is wanted.
- **`isSelf` on SaaS** uses the LEADER role (the portal Users page is
leader-only on SaaS, so the leader row is the viewer). Verified there's
no multi-leader creation path today; revisit if that changes.

Draft - not marking ready until reviewed.
2026-07-10 09:22:05 +00:00
Anthony StirlingandGitHub 68ec176719 Portal empty states: add CTAs and hide stat boxes (#6952)
# Description of Changes

Empty-state polish across the four processor (portal) list pages, so a
fresh workspace gets clear next steps instead of a row of zeroed-out
stat boxes.

- **Sources / Pipelines** - hide the KPI stat strip when the list is
empty; the empty state now shows an icon plus a primary + secondary CTA
(Connect source / Read the docs; Create a pipeline / Connect a source).
Also closes a gap where a successfully-fetched empty list rendered stat
boxes over a blank page with no empty state at all.
- **Policies** - hide the summary stat strip until at least one policy
is configured; the catalogue cards stay as the "configure a policy"
CTAs.
- **Documents** - hide the filter-pill + search toolbar on an empty
queue; the empty state gains an icon plus Create a pipeline / Connect a
source CTAs.
- **Storybook** - added `Default` + `Empty` stories for all four views;
the preview now loads the real English copy so stories render shipped
text rather than raw i18n keys.

---

## Checklist

### General

- [ ] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [x] I have performed a self-review of my own code
- [x] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [x] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)

### Translations (if applicable)

- [ ] I ran
[`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md)

### UI Changes (if applicable)

- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)

### Testing (if applicable)

- [x] I have run `task check` to verify linters, typechecks, and tests
pass
- [x] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing)
for more details.
2026-07-10 08:54:05 +00:00
Anthony StirlingandGitHub d17c3f4fec Portal: wire remaining hardcoded strings to i18n (#6953)
# Description of Changes

- Audited every portal page/component for hardcoded UI strings not
routed through `t()`
- Wired the remaining ones to i18n (~80 new `portal.*` keys in
`en-US/translation.toml`):
- Infrastructure status/label maps (deploy, api-key, cert, key-mode,
attestation, audit, model, region, environment) + API-key permissions
- Procurement "Key documents" modal, editor-admin deploy targets, users
seats label, pipeline output-folder placeholder
- Follows the existing house pattern: label maps store i18n keys,
resolved via `t(MAP[value])` at the render site
- Documents CSV export now reuses the on-screen column keys, and fixes a
latent bug where the exported status leaked the raw key instead of the
translated label
- No UI-copy change: en-US values are identical to the previously
hardcoded strings; other locales fall back to en-US as before

---

## Checklist

### General

- [x] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [x] I have performed a self-review of my own code
- [x] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)

### Translations (if applicable)

- [ ] I ran
[`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md)

### UI Changes (if applicable)

- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)

### Testing (if applicable)

- [x] I have run `task check` to verify linters, typechecks, and tests
pass
- [ ] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing)
for more details.
2026-07-10 08:54:02 +00:00
Anthony StirlingandGitHub 783a51950f Update translations for 40 languages via GPT-5.5 (#6954)
# Description of Changes

- Adds and updates translations across **40 languages** (~1,400–2,200
keys each) using GPT-5.5, filling previously-missing UI strings.
- Switches the translation scripts' default model from the year-old
`gpt-5` (5.0) to `gpt-5.5`, adding a `--model` flag and token/cost
reporting.
- Purely additive and validated: no existing translations changed, all
40 files match the en-US key structure, and no new placeholder issues
introduced.

---

## Checklist

### General

- [ ] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [x] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [x] I have performed a self-review of my own code
- [x] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [x] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)

### Translations (if applicable)

- [x] I ran
[`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md)

### UI Changes (if applicable)

- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)

### Testing (if applicable)

- [ ] I have run `task check` to verify linters, typechecks, and tests
pass
- [ ] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing)
for more details.
2026-07-10 08:53:59 +00:00
323 changed files with 100728 additions and 7923 deletions
@@ -246,6 +246,14 @@ public class ApplicationProperties {
* and paused runs are kept regardless of age.
*/
private int runExpiryMinutes = 30;
/**
* Whether a policy S3 source's custom endpoint may resolve to a loopback, link-local, or
* private address. Off by default so a user-supplied endpoint cannot be pointed at internal
* services (e.g. the cloud metadata address); enable for a self-hosted MinIO or other
* in-network object store.
*/
private boolean allowPrivateS3Endpoints = false;
}
@Data
@@ -0,0 +1,56 @@
package stirling.software.common.service;
/**
* Thread-scoped correlation id for one automation run — a single pipeline, policy, or AI-workflow
* execution over its input file(s).
*
* <p>Automations dispatch each tool step as a separate internal loopback POST via {@link
* InternalApiClient}. The orchestrator opens a run scope around its dispatch loop; {@code
* InternalApiClient} reads {@link #current()} and stamps it on every sub-step request as {@link
* #RUN_ID_HEADER}. The SaaS PAYG interceptor uses that header so all sub-steps of ONE run group
* into a single charge, while two <em>separate</em> runs that happen to touch identical bytes stay
* distinct charges (the old content+time-window grouping merged them).
*
* <p>Sub-steps dispatch synchronously on the orchestrator's own thread (loopback {@code
* RestTemplate}), so this ThreadLocal is visible to {@code InternalApiClient}. The id then crosses
* to the receiving request thread via the HTTP header — never via this ThreadLocal.
*
* <p>No-op when the id is absent (a standalone tool call): the interceptor treats a missing run id
* as "its own charge", which is exactly what a one-off call should be.
*/
public final class AutomationRunContext {
/** Header carrying the run id on internal sub-step dispatches. */
public static final String RUN_ID_HEADER = "X-Stirling-Run-Id";
private static final ThreadLocal<String> CURRENT = new ThreadLocal<>();
private AutomationRunContext() {}
/**
* Opens a run scope on the current thread. Returns an {@link AutoCloseable} that restores the
* previously-active id (nesting-safe) — use in try-with-resources around the dispatch loop.
*/
public static Scope open(String runId) {
String previous = CURRENT.get();
CURRENT.set(runId);
return () -> {
if (previous == null) {
CURRENT.remove();
} else {
CURRENT.set(previous);
}
};
}
/** The run id active on this thread, or {@code null} when not inside a run scope. */
public static String current() {
return CURRENT.get();
}
/** AutoCloseable whose {@link #close()} declares no checked exception. */
public interface Scope extends AutoCloseable {
@Override
void close();
}
}
@@ -111,6 +111,14 @@ public class InternalApiClient {
// step inside a policy run must bill as AUTOMATION, not AI). Set unconditionally because
// every caller of this dispatcher is an automation surface by design.
headers.add(AUTOMATION_HEADER, "true");
// Propagate the current automation run id (set by the orchestrator around its dispatch
// loop) so the PAYG interceptor groups every sub-step of this one run into a single charge,
// and never merges two separate runs that happen to touch identical bytes. Absent → the
// receiving call is treated as standalone. See AutomationRunContext.
String runId = AutomationRunContext.current();
if (runId != null && !runId.isEmpty()) {
headers.add(AutomationRunContext.RUN_ID_HEADER, runId);
}
// A no-file ai/tools call (e.g. create-pdf-from-html-agent) sends only string params, so
// without this RestTemplate would use urlencoded instead of the multipart the controller
@@ -10,6 +10,7 @@ import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Map.Entry;
import java.util.UUID;
import org.springframework.core.io.FileSystemResource;
import org.springframework.core.io.Resource;
@@ -27,6 +28,7 @@ import stirling.software.SPDF.model.PipelineConfig;
import stirling.software.SPDF.model.PipelineOperation;
import stirling.software.SPDF.model.PipelineResult;
import stirling.software.SPDF.service.ApiDocService;
import stirling.software.common.service.AutomationRunContext;
import stirling.software.common.service.InternalApiClient;
import stirling.software.common.util.TempFileManager;
import stirling.software.common.util.ZipExtractionUtils;
@@ -71,6 +73,17 @@ public class PipelineProcessor {
PipelineResult runPipelineAgainstFiles(List<Resource> outputFiles, PipelineConfig config)
throws Exception {
// One pipeline execution = one automation run. Scope a run id so every tool sub-step
// dispatched via InternalApiClient groups into a single charge on the SaaS billing side
// (see AutomationRunContext); pipeline steps run synchronously on this thread.
try (AutomationRunContext.Scope ignored =
AutomationRunContext.open(UUID.randomUUID().toString())) {
return runPipelineAgainstFilesInternal(outputFiles, config);
}
}
private PipelineResult runPipelineAgainstFilesInternal(
List<Resource> outputFiles, PipelineConfig config) throws Exception {
PipelineResult result = new PipelineResult();
ByteArrayOutputStream logStream = new ByteArrayOutputStream();
@@ -133,29 +133,45 @@ public final class S3Clients {
* storage.s3.allow-private-endpoints=true}.
*/
static void validateEndpointHost(URI endpoint, boolean allowPrivate) {
validateEndpointHost(
endpoint,
allowPrivate,
"storage.s3.endpoint",
"set storage.s3.allow-private-endpoints=true to opt in"
+ " (e.g. for MinIO or in-cluster S3).");
}
/**
* The same private-address guard for S3 endpoints configured outside the {@code storage.s3.*}
* block (e.g. per-source policy config), with the setting named in messages supplied by the
* caller.
*/
public static void validateEndpointHost(
URI endpoint, boolean allowPrivate, String settingName, String optInHint) {
if (allowPrivate) {
return;
}
String host = endpoint.getHost();
if (host == null || host.isBlank()) {
throw new IllegalStateException("storage.s3.endpoint must include a host: " + endpoint);
throw new IllegalStateException(settingName + " must include a host: " + endpoint);
}
InetAddress[] addresses;
try {
addresses = InetAddress.getAllByName(host);
} catch (UnknownHostException e) {
throw new IllegalStateException(
"Unable to resolve storage.s3.endpoint host '" + host + "'", e);
"Unable to resolve " + settingName + " host '" + host + "'", e);
}
for (InetAddress address : addresses) {
if (isPrivateOrLocal(address)) {
throw new IllegalStateException(
"storage.s3.endpoint host '"
settingName
+ " host '"
+ host
+ "' resolves to private/link-local address "
+ address.getHostAddress()
+ "; set storage.s3.allow-private-endpoints=true to opt in"
+ " (e.g. for MinIO or in-cluster S3).");
+ "; "
+ optInHint);
}
}
}
@@ -4,6 +4,7 @@ import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.List;
import org.springframework.context.annotation.Profile;
import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
@@ -38,6 +39,9 @@ import stirling.software.proprietary.security.database.repository.UserRepository
@PreAuthorize("hasRole('ADMIN')")
@RequiredArgsConstructor
@EnterpriseEndpoint
// Self-hosted only: counts are server-wide. On SaaS this endpoint is owned by the team-scoped
// SaasFleetUsageController (@Profile("saas")) so one backend can't leak another tenant's usage.
@Profile("!saas")
public class FleetUsageController {
private final PersistentAuditEventRepository auditRepository;
@@ -0,0 +1,31 @@
package stirling.software.proprietary.integration.crypto;
import jakarta.persistence.AttributeConverter;
import jakarta.persistence.Converter;
/**
* {@link EncryptedStringConverter} for columns that held plaintext before encryption shipped:
* writes are always encrypted, but a stored value that is not valid ciphertext is returned as-is,
* so pre-encryption rows keep loading and become encrypted on their next save. The discrimination
* is exact for JSON payloads, which can never be mistaken for ciphertext ('{' is not in the Base64
* alphabet). The trade-off is that a genuinely corrupted ciphertext surfaces as garbage to the
* caller's parser instead of failing here.
*/
@Converter
public class LenientEncryptedStringConverter implements AttributeConverter<String, String> {
@Override
public String convertToDatabaseColumn(String attribute) {
return CredentialEncryption.encrypt(attribute);
}
@Override
public String convertToEntityAttribute(String dbData) {
try {
return CredentialEncryption.decrypt(dbData);
} catch (IllegalArgumentException | IllegalStateException e) {
// Not ciphertext: legacy plaintext from before encryption shipped.
return dbData;
}
}
}
@@ -48,7 +48,9 @@ import stirling.software.proprietary.policy.engine.PolicyRunHandle;
import stirling.software.proprietary.policy.engine.PolicyRunRegistry;
import stirling.software.proprietary.policy.engine.PolicyRunner;
import stirling.software.proprietary.policy.engine.PolicyValidator;
import stirling.software.proprietary.policy.engine.SweepOutcome;
import stirling.software.proprietary.policy.ledger.ProcessedLedger;
import stirling.software.proprietary.policy.model.OutputSpec;
import stirling.software.proprietary.policy.model.PipelineDefinition;
import stirling.software.proprietary.policy.model.Policy;
import stirling.software.proprietary.policy.model.PolicyInputs;
@@ -58,12 +60,15 @@ import stirling.software.proprietary.policy.model.PolicyRunView;
import stirling.software.proprietary.policy.overview.PoliciesOverviewResponse;
import stirling.software.proprietary.policy.overview.PolicyOverviewService;
import stirling.software.proprietary.policy.progress.PolicyProgressListener;
import stirling.software.proprietary.policy.source.EditorSource;
import stirling.software.proprietary.policy.source.SourceAccessGuard;
import stirling.software.proprietary.policy.source.SourceDocCounter;
import stirling.software.proprietary.policy.source.SourceStore;
import stirling.software.proprietary.policy.store.PolicyStore;
import stirling.software.proprietary.policy.trigger.PolicyTrigger;
import stirling.software.proprietary.policy.trigger.PolicyTriggerManager;
import stirling.software.proprietary.policy.trigger.TriggerInfo;
import stirling.software.proprietary.util.SecretMasker;
/**
* Policy CRUD plus pipeline runs (stored or ad-hoc). Runs are async: returns a run id, poll {@code
@@ -83,6 +88,7 @@ public class PolicyController {
private final PolicyStore policyStore;
private final SourceStore sourceStore;
private final SourceAccessGuard sourceAccessGuard;
private final SourceDocCounter docCounter;
private final PolicyValidator policyValidator;
private final PolicyAccessGuard policyAccessGuard;
private final PolicyManagementAuthority policyManagementAuthority;
@@ -109,9 +115,10 @@ public class PolicyController {
throws IOException {
requireRunnable(definition);
PolicyInputs inputs = toInputs(files);
String runId =
policyRunner.runAdHoc(definition, inputs, PolicyProgressListener.NOOP).runId();
return ResponseEntity.accepted().body(new JobResponse<>(true, runId, null));
PolicyRunHandle handle =
policyRunner.runAdHoc(definition, inputs, PolicyProgressListener.NOOP);
recordEditorDocs(inputs);
return ResponseEntity.accepted().body(new JobResponse<>(true, handle.runId(), null));
}
@PostMapping(value = "/run/stream", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
@@ -133,6 +140,7 @@ public class PolicyController {
emitter.onError(e -> log.warn("Policy run SSE emitter error", e));
PolicyRunHandle handle = policyRunner.runAdHoc(definition, inputs, streamListener(emitter));
recordEditorDocs(inputs);
// whenComplete runs on the worker thread after the run finishes, so the terminal event
// never races the step events.
handle.completion()
@@ -202,7 +210,7 @@ public class PolicyController {
+ " assigned; returns the stored policy with its id.")
public ResponseEntity<Policy> savePolicy(@RequestBody Policy policy) {
requirePolicyEditingAllowed();
Policy owned = resolveOwnership(policy);
Policy owned = withStoredOutputSecrets(resolveOwnership(policy));
requireAccessibleSources(owned);
try {
policyValidator.validate(owned);
@@ -213,7 +221,7 @@ public class PolicyController {
// Re-sync trigger registrations now so a new/changed folder-watch policy starts being
// watched immediately instead of after the next reconcile sweep.
policyTriggerManager.notifyPoliciesChanged();
return ResponseEntity.ok(saved);
return ResponseEntity.ok(withMaskedOutputSecrets(saved));
}
@PutMapping("/order")
@@ -282,6 +290,50 @@ public class PolicyController {
teamId);
}
/** Output secrets never leave the server: reads return the redaction sentinel instead. */
private static Policy withMaskedOutputSecrets(Policy policy) {
return withOutput(
policy,
new OutputSpec(
policy.output().type(), SecretMasker.mask(policy.output().options())));
}
/**
* An edit that round-trips a masked read sends output secrets back as the sentinel; restore
* them from the stored policy so saving without re-typing keeps them (validation then runs
* against the real values).
*/
private Policy withStoredOutputSecrets(Policy incoming) {
if (incoming.id() == null || incoming.id().isBlank()) {
return incoming;
}
return policyStore
.get(incoming.id())
.map(
existing ->
withOutput(
incoming,
new OutputSpec(
incoming.output().type(),
SecretMasker.restoreRedacted(
incoming.output().options(),
existing.output().options()))))
.orElse(incoming);
}
private static Policy withOutput(Policy policy, OutputSpec output) {
return new Policy(
policy.id(),
policy.name(),
policy.owner(),
policy.enabled(),
policy.trigger(),
policy.sourceIds(),
policy.steps(),
output,
policy.teamId());
}
/**
* Creating, editing, pausing/resuming, and deleting policies requires the editor role for the
* caller's team — a team leader on SaaS (see {@link PolicyManagementAuthority}); the global
@@ -306,9 +358,14 @@ public class PolicyController {
@GetMapping
@Operation(
summary = "List policies",
description = "Lists the policies belonging to the caller's team.")
description =
"Lists the policies belonging to the caller's team. Secret-bearing output"
+ " options are returned as a redaction sentinel, never their stored"
+ " values.")
public List<Policy> listPolicies() {
return policyAccessGuard.visibleFrom(policyStore);
return policyAccessGuard.visibleFrom(policyStore).stream()
.map(PolicyController::withMaskedOutputSecrets)
.toList();
}
@GetMapping("/overview")
@@ -337,11 +394,17 @@ public class PolicyController {
}
@GetMapping("/{policyId}")
@Operation(summary = "Get a policy by id")
@Operation(
summary = "Get a policy by id",
description =
"Secret-bearing output options are returned as a redaction sentinel, never"
+ " their stored values; an edit that sends the sentinel back keeps"
+ " them.")
public ResponseEntity<Policy> getPolicy(@PathVariable String policyId) {
return policyStore
.get(policyId)
.filter(policyAccessGuard::canAccess)
.map(PolicyController::withMaskedOutputSecrets)
.map(ResponseEntity::ok)
.orElseGet(() -> ResponseEntity.notFound().build());
}
@@ -412,9 +475,10 @@ public class PolicyController {
description =
"Pulls the policy's configured sources and runs the pipeline now, regardless of"
+ " the enabled flag (which only gates automatic triggering). Returns"
+ " the ids of the runs started; poll the run-status endpoint for each."
+ " Empty when the sources yielded no work to do.")
public ResponseEntity<List<String>> trigger(@PathVariable String policyId) {
+ " the ids of the runs started (poll the run-status endpoint for each)"
+ " plus what the sweep skipped - already-processed, parked-by-failure,"
+ " and in-flight counts - so an empty result explains itself.")
public ResponseEntity<SweepOutcome> trigger(@PathVariable String policyId) {
Policy policy =
policyStore
.get(policyId)
@@ -433,6 +497,17 @@ public class PolicyController {
}
}
/**
* Ad-hoc runs (AI / one-off pipelines) are still editor activity, so their supplied documents
* feed the same virtual editor source as stored editor policies, counted against the caller's
* team. A run with no primary documents (generator pipeline) records nothing.
*/
private void recordEditorDocs(PolicyInputs inputs) {
docCounter.record(
EditorSource.counterKey(sourceAccessGuard.currentTeamId()),
inputs.primary().size());
}
/**
* Turn the typed run files into engine {@link PolicyInputs}: the primary documents plus the
* named supporting-file store, where each asset's {@code key} is the name a step references
@@ -21,6 +21,7 @@ import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.model.job.ResultFile;
import stirling.software.common.service.AutomationRunContext;
import stirling.software.common.service.FileStorage;
import stirling.software.common.service.InternalApiTimeoutException;
import stirling.software.common.service.JobOwnershipService;
@@ -199,63 +200,78 @@ public class PolicyEngine {
PolicyProgressListener listener,
CompletableFuture<PolicyRun> completion) {
String runId = run.getRunId();
try {
run.markRunning();
PolicyExecutionResult result =
stepExecutor.execute(run.getDefinition(), inputs, listener);
OutputSpec output = run.getDefinition().output();
List<ResultFile> outputs =
sinkFor(output)
.deliver(
new OutputDelivery(runId, run.getPolicyId()),
result.files(),
output);
taskManager.setMultipleFileResults(runId, outputs);
taskManager.setComplete(runId);
run.complete(outputs);
} catch (PolicyInputRequiredException e) {
// Expected path: suspend rather than fail. Persist intermediates as fileIds so the run
// can resume after this worker thread is gone.
WaitState wait = suspend(e);
run.waitForInput(wait);
taskManager.addNote(runId, "Waiting for input: " + e.getMessage());
} catch (InternalApiTimeoutException e) {
String message = toolTimeoutMessage(e);
log.error(
"Policy run {} timed out on {}: {}",
runId,
e.getEndpointPath(),
e.getMessage());
run.fail(message);
taskManager.setError(runId, message);
} catch (RestClientResponseException e) {
// A downstream tool call returned an error status. When it's a structured entitlement
// response (401/402 with a JSON `error` sentinel), surface that code onto the run so
// the
// client can react — e.g. pop the usage-limit modal — instead of only seeing a generic
// failure. We don't interpret the code here (that would couple this module to the saas
// billing layer); we just pass it through for the client to map. Other statuses fall
// through to the generic failure below.
String code = DownstreamEntitlementError.extractCode(e);
if (code != null) {
log.info("Policy run {} blocked by downstream entitlement gate ({})", runId, code);
String message = "Usage limit reached";
run.failWithCode(message, code, DownstreamEntitlementError.extractSubscribed(e));
taskManager.setError(runId, message);
} else {
String message = "Policy run failed: " + e.getMessage();
log.error("Policy run {} failed (downstream HTTP error)", runId, e);
// One policy run = one automation run. Scope the run id on this worker thread (the async
// hop already happened) so every tool sub-step dispatched via InternalApiClient groups into
// a single charge, and two separate policy runs on the same document stay distinct charges.
try (AutomationRunContext.Scope runScope = AutomationRunContext.open(runId)) {
try {
run.markRunning();
PolicyExecutionResult result =
stepExecutor.execute(run.getDefinition(), inputs, listener);
OutputSpec output = run.getDefinition().output();
List<ResultFile> outputs =
sinkFor(output)
.deliver(
new OutputDelivery(runId, run.getPolicyId()),
result.files(),
output);
taskManager.setMultipleFileResults(runId, outputs);
taskManager.setComplete(runId);
run.complete(outputs);
} catch (PolicyInputRequiredException e) {
// Expected path: suspend rather than fail. Persist intermediates as fileIds so the
// run
// can resume after this worker thread is gone.
WaitState wait = suspend(e);
run.waitForInput(wait);
taskManager.addNote(runId, "Waiting for input: " + e.getMessage());
} catch (InternalApiTimeoutException e) {
String message = toolTimeoutMessage(e);
log.error(
"Policy run {} timed out on {}: {}",
runId,
e.getEndpointPath(),
e.getMessage());
run.fail(message);
taskManager.setError(runId, message);
} catch (RestClientResponseException e) {
// A downstream tool call returned an error status. When it's a structured
// entitlement
// response (401/402 with a JSON `error` sentinel), surface that code onto the run
// so
// the
// client can react — e.g. pop the usage-limit modal — instead of only seeing a
// generic
// failure. We don't interpret the code here (that would couple this module to the
// saas
// billing layer); we just pass it through for the client to map. Other statuses
// fall
// through to the generic failure below.
String code = DownstreamEntitlementError.extractCode(e);
if (code != null) {
log.info(
"Policy run {} blocked by downstream entitlement gate ({})",
runId,
code);
String message = "Usage limit reached";
run.failWithCode(
message, code, DownstreamEntitlementError.extractSubscribed(e));
taskManager.setError(runId, message);
} else {
String message = "Policy run failed: " + e.getMessage();
log.error("Policy run {} failed (downstream HTTP error)", runId, e);
run.fail(message);
taskManager.setError(runId, message);
}
} catch (Exception e) {
String message = "Policy run failed: " + e.getMessage();
log.error("Policy run {} failed", runId, e);
run.fail(message);
taskManager.setError(runId, message);
} finally {
// Always resolve so stream/await callers unblock.
completion.complete(run);
}
} catch (Exception e) {
String message = "Policy run failed: " + e.getMessage();
log.error("Policy run {} failed", runId, e);
run.fail(message);
taskManager.setError(runId, message);
} finally {
// Always resolve so stream/await callers unblock.
completion.complete(run);
}
}
@@ -21,6 +21,7 @@ import stirling.software.proprietary.policy.model.PolicyInputs;
import stirling.software.proprietary.policy.model.PolicyRun;
import stirling.software.proprietary.policy.model.PolicyRunStatus;
import stirling.software.proprietary.policy.progress.PolicyProgressListener;
import stirling.software.proprietary.policy.source.EditorSource;
import stirling.software.proprietary.policy.source.Source;
import stirling.software.proprietary.policy.source.SourceDocCounter;
import stirling.software.proprietary.policy.source.SourceStore;
@@ -44,7 +45,7 @@ public class PolicyRunner {
private final ProcessedLedger processedLedger;
/** Full-listing sweep: resolve every source, then reconcile the ledger. */
public List<String> run(Policy policy) {
public SweepOutcome run(Policy policy) {
return run(policy, SweepKind.FULL);
}
@@ -52,10 +53,10 @@ public class PolicyRunner {
* Trigger entry point. Pulls every referenced source; each yielded unit becomes its own run so
* one failure does not affect the others. No sources means one run with no input (generator
* pipeline). Missing or disabled sources are skipped so one broken reference does not stop the
* rest. Returns the ids of the runs it started (empty when sources yielded no work), so a
* manual trigger can report back which runs to follow.
* rest. Returns the ids of the runs it started plus what the sweep skipped, so a manual trigger
* can report which runs to follow or why nothing ran.
*/
public List<String> run(Policy policy, SweepKind sweep) {
public SweepOutcome run(Policy policy, SweepKind sweep) {
long sweepStart = System.currentTimeMillis();
PolicySweep context = new PolicySweep(policy.id(), sweep, processedLedger);
List<String> runIds = new ArrayList<>();
@@ -95,13 +96,19 @@ public class PolicyRunner {
policy.id());
}
}
return runIds;
return context.outcome(runIds);
}
/** Run a stored policy on caller-supplied files (e.g. manual upload), bypassing its sources. */
/**
* Run a stored policy on caller-supplied files (e.g. an editor upload), bypassing its sources.
* The supplied documents are still counted against the virtual {@link EditorSource}, scoped to
* the policy's team, so the Sources overview reports the whole team's editor throughput.
*/
public PolicyRunHandle runWith(
Policy policy, PolicyInputs inputs, PolicyProgressListener listener) {
return policyEngine.runPolicy(policy, inputs, listener);
PolicyRunHandle handle = policyEngine.runPolicy(policy, inputs, listener);
docCounter.record(EditorSource.counterKey(policy.teamId()), inputs.primary().size());
return handle;
}
/** Run an ad-hoc pipeline with no stored policy (AI/Automate one-offs). */
@@ -86,4 +86,32 @@ final class PolicySweep implements ResolveContext {
synchronized Set<String> presentIdentities() {
return Set.copyOf(present);
}
/**
* Summarise the sweep from state already in hand (no extra ledger reads): the prefetched rows
* were loaded before claiming, and successful claims flipped their entries to PROCESSING, so
* what remains DONE or ERROR is exactly what this sweep skipped.
*/
synchronized SweepOutcome outcome(List<String> runIds) {
int alreadyProcessed = 0;
int parked = 0;
int processing = 0;
for (String identity : present) {
ClaimState state = prefetched.get(identity);
if (state == null) {
continue;
}
switch (state.status()) {
case DONE -> alreadyProcessed++;
case ERROR -> parked++;
case PROCESSING, INTERRUPTED -> processing++;
}
}
return new SweepOutcome(
runIds,
present.size(),
alreadyProcessed,
parked,
Math.max(0, processing - runIds.size()));
}
}
@@ -0,0 +1,19 @@
package stirling.software.proprietary.policy.engine;
import java.util.List;
/**
* What one policy sweep found and started, so a manual trigger can explain an empty result instead
* of a blanket "nothing to do": how many files the sources listed, how many were skipped because
* they are already processed at their current version, how many are parked by a failed run (not
* retried until they change or history is cleared), and how many are still in flight from an
* earlier sweep. Counts are zero for {@link SweepKind#LIGHT} sweeps, which do not take a full
* listing.
*/
public record SweepOutcome(
List<String> runIds, int filesListed, int alreadyProcessed, int parked, int inFlight) {
public SweepOutcome {
runIds = runIds == null ? List.of() : List.copyOf(runIds);
}
}
@@ -0,0 +1,286 @@
package stirling.software.proprietary.policy.input;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty;
import org.springframework.core.io.AbstractResource;
import org.springframework.core.io.Resource;
import org.springframework.stereotype.Service;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.proprietary.policy.model.InputSpec;
import stirling.software.proprietary.policy.model.PolicyInputs;
import stirling.software.proprietary.policy.s3.S3Config;
import stirling.software.proprietary.policy.s3.S3ConnectionPool;
import stirling.software.proprietary.policy.s3.S3Identities;
import software.amazon.awssdk.core.exception.SdkException;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.DeleteObjectRequest;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
import software.amazon.awssdk.services.s3.model.HeadObjectResponse;
import software.amazon.awssdk.services.s3.model.ListObjectsV2Request;
import software.amazon.awssdk.services.s3.model.ListObjectsV2Response;
import software.amazon.awssdk.services.s3.model.NoSuchKeyException;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.s3.model.S3Object;
/**
* Reads input files from an Amazon S3 (or S3-compatible) bucket; each listed object is its own unit
* of work, claimed through the {@link ResolveContext} ledger and tracked in place. Identity and
* version gate come from {@link S3Identities}, so the steady-state sweep never downloads content.
* Options (see {@link S3Config}): "bucket" (required), "region" (default us-east-1), "prefix" (only
* keys starting with it are read), "endpoint" (S3-compatible stores such as MinIO; path-style
* addressing is used automatically), "accessKeyId" and "secretAccessKey" (required; requests are
* never signed with the server's own AWS identity), and "mode" which is "consume" (default: a
* processed object is deleted once every policy that claimed it has settled successfully and it is
* still the version that ran; failures stay in place and are not retried until they change) or
* "snapshot" (stateless, every run sees the full set). Keys ending in "/" (folder placeholders) and
* keys with a dot-prefixed path segment are never picked up, mirroring the folder source's
* hidden-file rule.
*/
@Slf4j
@Service
@RequiredArgsConstructor
@ConditionalOnBooleanProperty(name = "policies.enabled")
public class S3InputSource implements InputSource {
private static final String TYPE = "s3";
private final S3ConnectionPool connectionPool;
@Override
public String type() {
return TYPE;
}
@Override
public boolean supports(InputSpec spec) {
return spec != null && TYPE.equals(spec.type());
}
/**
* Fails fast at save time: bad config shape, a private endpoint without the operator opt-in, or
* a bucket the supplied credentials cannot list.
*/
@Override
public void validate(InputSpec spec) {
S3Config config = S3Config.from(spec.options());
try {
connectionPool.clientFor(config).listObjectsV2(listRequest(config).maxKeys(1).build());
} catch (SdkException e) {
throw new IllegalArgumentException(
"cannot access s3://"
+ config.bucket()
+ "/"
+ config.prefix()
+ ": "
+ e.getMessage(),
e);
}
}
@Override
public List<ResolvedInput> resolve(InputSpec spec, ResolveContext ctx) throws IOException {
S3Config config = S3Config.from(spec.options());
S3Client client = connectionPool.clientFor(config);
// A listing failure propagates so the sweep reads it as "could not list" (which vetoes
// presence cleanup), never as "verifiably no objects".
List<S3Object> objects = listObjects(client, config);
if (config.snapshot()) {
return objects.stream()
.map(
object ->
ResolvedInput.of(
PolicyInputs.of(
List.of(
objectResource(
client, config, object)))))
.toList();
}
ctx.reportPresent(
objects.stream()
.map(object -> S3Identities.identity(config.bucket(), object.key()))
.toList());
List<ResolvedInput> work = new ArrayList<>();
for (S3Object object : objects) {
String identity = S3Identities.identity(config.bucket(), object.key());
String gate = S3Identities.gate(object.eTag(), object.size(), object.lastModified());
if (!ctx.claim(identity, gate, null)) {
continue;
}
work.add(
new ResolvedInput(
PolicyInputs.of(List.of(objectResource(client, config, object))),
success ->
completeConsumed(
ctx,
client,
config,
object.key(),
identity,
gate,
success)));
}
return work;
}
/**
* Settle at the version this run claimed, then remove the object only when it still carries
* that version and every policy that claimed it has settled DONE, mirroring the folder source's
* consensus delete. A failed run settles ERROR and never deletes; the DONE row of an object
* that could not be deleted still stops reprocessing.
*/
private void completeConsumed(
ResolveContext ctx,
S3Client client,
S3Config config,
String key,
String identity,
String claimGate,
boolean success) {
ctx.settle(identity, claimGate, null, success);
if (!success) {
return;
}
try {
HeadObjectResponse head =
client.headObject(
HeadObjectRequest.builder().bucket(config.bucket()).key(key).build());
String currentGate =
S3Identities.gate(head.eTag(), head.contentLength(), head.lastModified());
if (currentGate.equals(claimGate) && ctx.allSettledDone(identity)) {
client.deleteObject(
DeleteObjectRequest.builder().bucket(config.bucket()).key(key).build());
}
} catch (NoSuchKeyException alreadyGone) {
// Removed by the user or a co-watching policy's own consensus delete: nothing to do.
} catch (S3Exception e) {
if (e.statusCode() == 404) {
return;
}
log.warn("Could not remove consumed S3 object {}: {}", identity, e.getMessage());
} catch (SdkException e) {
log.warn("Could not remove consumed S3 object {}: {}", identity, e.getMessage());
}
}
/** Every ingestible object under the configured prefix, across all listing pages. */
private static List<S3Object> listObjects(S3Client client, S3Config config) {
List<S3Object> objects = new ArrayList<>();
String continuationToken = null;
do {
ListObjectsV2Request.Builder request = listRequest(config);
if (continuationToken != null) {
request.continuationToken(continuationToken);
}
ListObjectsV2Response page = client.listObjectsV2(request.build());
for (S3Object object : page.contents()) {
if (ingestible(object)) {
objects.add(object);
}
}
continuationToken = page.nextContinuationToken();
} while (continuationToken != null);
return objects;
}
private static ListObjectsV2Request.Builder listRequest(S3Config config) {
ListObjectsV2Request.Builder request =
ListObjectsV2Request.builder().bucket(config.bucket());
if (!config.prefix().isEmpty()) {
request.prefix(config.prefix());
}
return request;
}
/**
* Folder-placeholder keys (ending "/") and keys with a dot-prefixed segment are skipped, so a
* hidden convention (e.g. a future output sink's staging prefix) is never re-ingested.
*/
private static boolean ingestible(S3Object object) {
String key = object.key();
if (key.isEmpty() || key.endsWith("/")) {
return false;
}
for (String segment : key.split("/")) {
if (segment.startsWith(".")) {
return false;
}
}
return true;
}
private static Resource objectResource(S3Client client, S3Config config, S3Object object) {
return new S3ObjectResource(client, config.bucket(), object);
}
/**
* Streams the object on demand, pinned to the ETag observed at listing time so a run never
* reads a different version than the sweep claimed (a swapped object fails the read with a
* precondition error and the new version is claimed by a later sweep).
*/
private static final class S3ObjectResource extends AbstractResource {
private final S3Client client;
private final String bucket;
private final String key;
private final String eTag;
private final Long size;
private S3ObjectResource(S3Client client, String bucket, S3Object object) {
this.client = client;
this.bucket = bucket;
this.key = object.key();
this.eTag = object.eTag();
this.size = object.size();
}
@Override
public InputStream getInputStream() throws IOException {
GetObjectRequest.Builder request = GetObjectRequest.builder().bucket(bucket).key(key);
if (eTag != null && !eTag.isBlank()) {
request.ifMatch(eTag);
}
try {
return client.getObject(request.build());
} catch (NoSuchKeyException e) {
throw new FileNotFoundException(getDescription() + " no longer exists");
} catch (SdkException e) {
throw new IOException(
"Could not read " + getDescription() + ": " + e.getMessage(), e);
}
}
/** Listed just now; readers get a precise error from {@link #getInputStream} instead. */
@Override
public boolean exists() {
return true;
}
@Override
public long contentLength() {
return size == null ? -1 : size;
}
@Override
public String getFilename() {
return key.substring(key.lastIndexOf('/') + 1);
}
@Override
public String getDescription() {
return "S3 object " + S3Identities.identity(bucket, key);
}
}
}
@@ -15,7 +15,6 @@ import java.util.List;
import java.util.UUID;
import java.util.stream.Stream;
import org.apache.commons.io.FilenameUtils;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty;
import org.springframework.core.io.Resource;
import org.springframework.http.MediaType;
@@ -82,7 +81,7 @@ public class FolderOutputSink implements PolicyOutputSink {
List<ResultFile> results = new ArrayList<>();
for (int i = 0; i < outputs.size(); i++) {
Resource resource = outputs.get(i);
String name = safeName(resource.getFilename(), i);
String name = OutputNames.safeName(resource.getFilename(), i);
Path staged = tmpDir.resolve(UUID.randomUUID().toString());
String contentHash = stage(resource, staged, delivery.policyId() != null);
long size = Files.size(staged);
@@ -198,29 +197,14 @@ public class FolderOutputSink implements PolicyOutputSink {
return Path.of(directory.toString());
}
// Strip any directory component / "../" so a crafted output name cannot escape targetDir.
private static String safeName(String filename, int index) {
if (filename == null || filename.isBlank()) {
return "output-" + index;
}
String name = FilenameUtils.getName(filename);
if (name.isBlank() || ".".equals(name) || "..".equals(name)) {
return "output-" + index;
}
return name;
}
// Non-colliding path, appending " (n)" before the extension.
private static Path uniqueTarget(Path dir, String filename) {
Path candidate = dir.resolve(filename);
if (!Files.exists(candidate)) {
return candidate;
}
String base = FilenameUtils.getBaseName(filename);
String ext = FilenameUtils.getExtension(filename);
String suffix = ext.isEmpty() ? "" : "." + ext;
for (int n = 1; ; n++) {
Path next = dir.resolve(base + " (" + n + ")" + suffix);
Path next = dir.resolve(OutputNames.numbered(filename, n));
if (!Files.exists(next)) {
return next;
}
@@ -0,0 +1,29 @@
package stirling.software.proprietary.policy.output;
import org.apache.commons.io.FilenameUtils;
/** Output file naming shared by the sinks: sanitised base names and collision suffixes. */
final class OutputNames {
private OutputNames() {}
/** Strip any directory component / "../" so a crafted output name cannot escape the target. */
static String safeName(String filename, int index) {
if (filename == null || filename.isBlank()) {
return "output-" + index;
}
String name = FilenameUtils.getName(filename);
if (name.isBlank() || ".".equals(name) || "..".equals(name)) {
return "output-" + index;
}
return name;
}
/** The nth alternative for a taken name, appending " (n)" before the extension. */
static String numbered(String filename, int n) {
String base = FilenameUtils.getBaseName(filename);
String ext = FilenameUtils.getExtension(filename);
String suffix = ext.isEmpty() ? "" : "." + ext;
return base + " (" + n + ")" + suffix;
}
}
@@ -0,0 +1,270 @@
package stirling.software.proprietary.policy.output;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.DigestOutputStream;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.ArrayList;
import java.util.HexFormat;
import java.util.List;
import java.util.UUID;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty;
import org.springframework.core.io.Resource;
import org.springframework.http.MediaType;
import org.springframework.http.MediaTypeFactory;
import org.springframework.stereotype.Service;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.model.job.ResultFile;
import stirling.software.proprietary.policy.ledger.ProcessedLedger;
import stirling.software.proprietary.policy.model.OutputSpec;
import stirling.software.proprietary.policy.s3.S3Config;
import stirling.software.proprietary.policy.s3.S3ConnectionPool;
import stirling.software.proprietary.policy.s3.S3Identities;
import software.amazon.awssdk.core.exception.SdkException;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
import software.amazon.awssdk.services.s3.model.NoSuchKeyException;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectResponse;
import software.amazon.awssdk.services.s3.model.S3Exception;
/**
* Uploads a run's outputs to the bucket and key prefix given in the {@link OutputSpec} (same
* connection options as the S3 input source; "prefix" is the destination folder). The
* record-before-visible obligation is met without a rename step: a single-part PUT's ETag is the
* MD5 of its content on plain and SSE-S3 buckets, so the ledger row is recorded at that predicted
* gate BEFORE the upload, and the object is claimed under exactly the gate the next listing
* returns. Stores where the returned ETag differs (e.g. SSE-KMS) are re-recorded at the actual gate
* immediately after the PUT - a narrow race those buckets accept rather than a broken loop. Names
* never overwrite: uploads are conditional on the key not existing ({@code If-None-Match: *}),
* re-picking "name (n).ext" on collision exactly like the folder sink; stores without
* conditional-write support fall back to an existence check per candidate.
*/
@Slf4j
@Service
@RequiredArgsConstructor
@ConditionalOnBooleanProperty(name = "policies.enabled")
public class S3OutputSink implements PolicyOutputSink {
private static final String TYPE = "s3";
private final S3ConnectionPool connectionPool;
private final ProcessedLedger processedLedger;
@Override
public String type() {
return TYPE;
}
@Override
public boolean supports(OutputSpec spec) {
return spec != null && TYPE.equals(spec.type());
}
/**
* Config shape and endpoint guard only - no network probe, since write-only credentials
* (s3:PutObject without s3:ListBucket) are a legitimate setup for an output bucket and a
* listing probe would wrongly reject them.
*/
@Override
public void validate(OutputSpec spec) {
connectionPool.clientFor(S3Config.from(spec.options()));
}
@Override
public List<ResultFile> deliver(
OutputDelivery delivery, List<Resource> outputs, OutputSpec spec) throws IOException {
S3Config config = S3Config.from(spec.options());
S3Client client = connectionPool.clientFor(config);
List<ResultFile> results = new ArrayList<>();
for (int i = 0; i < outputs.size(); i++) {
Resource resource = outputs.get(i);
String name = OutputNames.safeName(resource.getFilename(), i);
Path staged = Files.createTempFile("s3-output-", ".tmp");
try {
String predictedGate = stage(resource, staged, delivery.policyId() != null);
long size = Files.size(staged);
String key = upload(delivery, client, config, name, staged, predictedGate);
String contentType =
MediaTypeFactory.getMediaType(name)
.orElse(MediaType.APPLICATION_OCTET_STREAM)
.toString();
results.add(
ResultFile.builder()
.fileId(UUID.randomUUID().toString())
.fileName(S3Identities.identity(config.bucket(), key))
.contentType(contentType)
.fileSize(size)
.build());
log.debug(
"Wrote policy run {} output to {}",
delivery.runId(),
S3Identities.identity(config.bucket(), key));
} finally {
try {
Files.deleteIfExists(staged);
} catch (IOException e) {
log.warn("Could not remove S3 staging file {}: {}", staged, e.getMessage());
}
}
}
return results;
}
/**
* Spool the output to a local staging file (S3 needs a known content length, and the body must
* be re-readable across collision retries). For a recorded delivery the MD5 - the predicted
* single-part ETag - is digested in the same pass; ad-hoc runs record nothing and skip it.
*/
private static String stage(Resource resource, Path staged, boolean recorded)
throws IOException {
if (!recorded) {
try (InputStream is = resource.getInputStream();
OutputStream out = Files.newOutputStream(staged)) {
is.transferTo(out);
}
return null;
}
MessageDigest digest = newMd5();
try (InputStream is = resource.getInputStream();
DigestOutputStream out =
new DigestOutputStream(Files.newOutputStream(staged), digest)) {
is.transferTo(out);
}
return HexFormat.of().formatHex(digest.digest());
}
/**
* The S3 shape of the folder sink's record-then-rename loop. The ledger row must exist before
* the object is visible, so it is recorded at the predicted gate before the PUT; losing the
* chosen key to a concurrent writer (the conditional PUT fails) forgets the just-recorded row -
* whatever object actually owns that key must stay claimable at any version - then re-picks. A
* PUT that never made the object visible also forgets its row.
*/
private String upload(
OutputDelivery delivery,
S3Client client,
S3Config config,
String name,
Path staged,
String predictedGate)
throws IOException {
String keyPrefix = keyPrefix(config);
boolean conditionalPuts = true;
for (int attempt = 0; ; attempt++) {
String key = keyPrefix + (attempt == 0 ? name : OutputNames.numbered(name, attempt));
String identity = S3Identities.identity(config.bucket(), key);
if (!conditionalPuts && exists(client, config.bucket(), key)) {
continue;
}
if (delivery.policyId() != null) {
processedLedger.recordOutput(delivery.policyId(), identity, predictedGate, null);
}
PutObjectRequest.Builder put =
PutObjectRequest.builder().bucket(config.bucket()).key(key);
if (conditionalPuts) {
put.ifNoneMatch("*");
}
try {
PutObjectResponse response =
client.putObject(put.build(), RequestBody.fromFile(staged));
reRecordIfGateDiffers(delivery, identity, predictedGate, response);
return key;
} catch (S3Exception e) {
forgetRecorded(delivery, identity, predictedGate);
if (conditionalPuts && e.statusCode() == 412) {
// Known edge: if our own PUT succeeded server-side but the response was lost
// and the SDK retried, that retry 412s here too - we then upload under the
// next name, leaving the first object row-less (claimable, single duplicate).
// Requires a response-lost network flake at exactly this moment; accepted.
log.debug("Output key {} taken concurrently; re-picking", identity);
continue;
}
if (conditionalPuts && e.statusCode() == 501) {
// Store without conditional-write support: retry this candidate with a plain
// existence check instead.
log.debug(
"Conditional PUT unsupported by {}; falling back to existence checks",
config.bucket());
conditionalPuts = false;
attempt--;
continue;
}
throw new IOException("Could not upload " + identity + ": " + e.getMessage(), e);
} catch (SdkException e) {
forgetRecorded(delivery, identity, predictedGate);
throw new IOException("Could not upload " + identity + ": " + e.getMessage(), e);
}
}
}
/**
* On buckets where a PUT's ETag is not the content MD5 (e.g. SSE-KMS), re-record at the gate
* listings will actually return. The row is briefly at the wrong gate while the object is
* already visible - the narrow race such stores trade for a working self-output skip.
*/
private void reRecordIfGateDiffers(
OutputDelivery delivery,
String identity,
String predictedGate,
PutObjectResponse response) {
if (delivery.policyId() == null) {
return;
}
String actualGate = S3Identities.gate(response.eTag(), null, null);
if (!actualGate.equals(predictedGate)) {
log.debug(
"PUT ETag for {} differs from content MD5 (encrypted bucket?); re-recording",
identity);
processedLedger.recordOutput(delivery.policyId(), identity, actualGate, null);
}
}
private void forgetRecorded(OutputDelivery delivery, String identity, String predictedGate) {
if (delivery.policyId() != null) {
processedLedger.forgetOutput(delivery.policyId(), identity, predictedGate);
}
}
private static boolean exists(S3Client client, String bucket, String key) {
try {
client.headObject(HeadObjectRequest.builder().bucket(bucket).key(key).build());
return true;
} catch (NoSuchKeyException e) {
return false;
} catch (S3Exception e) {
if (e.statusCode() == 404) {
return false;
}
throw e;
}
}
/** The configured prefix as a key-path prefix: "processed" and "processed/" mean the same. */
private static String keyPrefix(S3Config config) {
String prefix = config.prefix();
if (prefix.isEmpty() || prefix.endsWith("/")) {
return prefix;
}
return prefix + "/";
}
private static MessageDigest newMd5() {
try {
return MessageDigest.getInstance("MD5");
} catch (NoSuchAlgorithmException e) {
throw new IllegalStateException("MD5 unavailable", e);
}
}
}
@@ -0,0 +1,103 @@
package stirling.software.proprietary.policy.s3;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.Map;
/**
* Connection settings shared by the S3 input source and output sink, parsed from a spec's options
* map. Credentials are required: there is deliberately no fallback to the server's own AWS
* credential chain, so user-supplied config can never borrow the host's identity. {@code snapshot}
* is input-only and ignored by the sink.
*/
public record S3Config(
String bucket,
String region,
String prefix,
String endpoint,
String accessKeyId,
String secretAccessKey,
boolean snapshot) {
private static final String BUCKET_OPTION = "bucket";
private static final String REGION_OPTION = "region";
private static final String PREFIX_OPTION = "prefix";
private static final String ENDPOINT_OPTION = "endpoint";
private static final String ACCESS_KEY_ID_OPTION = "accessKeyId";
private static final String SECRET_ACCESS_KEY_OPTION = "secretAccessKey";
private static final String MODE_OPTION = "mode";
private static final String MODE_CONSUME = "consume";
private static final String MODE_SNAPSHOT = "snapshot";
public static S3Config from(Map<String, Object> options) {
String bucket = trimmed(options.get(BUCKET_OPTION));
if (bucket == null) {
throw new IllegalArgumentException("s3 config requires a 'bucket' option");
}
String region = trimmed(options.get(REGION_OPTION));
String prefix = trimmed(options.get(PREFIX_OPTION));
if (prefix != null && prefix.startsWith("/")) {
prefix = prefix.substring(1);
}
String endpoint = validEndpoint(trimmed(options.get(ENDPOINT_OPTION)));
String accessKeyId = trimmed(options.get(ACCESS_KEY_ID_OPTION));
String secretAccessKey = trimmed(options.get(SECRET_ACCESS_KEY_OPTION));
if (accessKeyId == null || secretAccessKey == null) {
throw new IllegalArgumentException(
"s3 config requires an 'accessKeyId' and 'secretAccessKey'");
}
String mode = trimmed(options.get(MODE_OPTION));
if (mode != null && !MODE_CONSUME.equals(mode) && !MODE_SNAPSHOT.equals(mode)) {
throw new IllegalArgumentException("s3 config 'mode' must be 'consume' or 'snapshot'");
}
return new S3Config(
bucket,
region == null ? "us-east-1" : region,
prefix == null ? "" : prefix,
endpoint,
accessKeyId,
secretAccessKey,
MODE_SNAPSHOT.equals(mode));
}
private static String validEndpoint(String endpoint) {
if (endpoint == null) {
return null;
}
URI uri;
try {
uri = new URI(endpoint);
} catch (URISyntaxException e) {
throw new IllegalArgumentException("s3 config 'endpoint' is not a valid URL", e);
}
if (!"http".equals(uri.getScheme()) && !"https".equals(uri.getScheme())) {
throw new IllegalArgumentException(
"s3 config 'endpoint' must be an http(s) URL, e.g. https://s3.example.com");
}
return endpoint;
}
private static String trimmed(Object value) {
if (value == null) {
return null;
}
String text = value.toString().trim();
return text.isEmpty() ? null : text;
}
/** Never prints the credentials, so an accidental log line cannot leak them. */
@Override
public String toString() {
return "S3Config[bucket="
+ bucket
+ ", region="
+ region
+ ", prefix="
+ prefix
+ ", endpoint="
+ endpoint
+ ", snapshot="
+ snapshot
+ "]";
}
}
@@ -0,0 +1,110 @@
package stirling.software.proprietary.policy.s3;
import java.net.URI;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.function.Function;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty;
import org.springframework.stereotype.Service;
import jakarta.annotation.PreDestroy;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.proprietary.cluster.s3.S3Clients;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3ClientBuilder;
import software.amazon.awssdk.services.s3.S3Configuration;
/**
* Long-lived {@link S3Client}s for policy S3 sources and sinks, one per distinct {@link S3Config},
* closed at shutdown. An edited spec simply maps to a new entry, and a stale entry costs nothing
* (the URL-connection HTTP client holds no pooled sockets or threads). Clients sign exclusively
* with the spec's own credentials - there is deliberately no fallback to the server's AWS
* credential chain, so user-supplied config can never borrow the host's identity. Endpoints are
* guarded against private addresses before a client is ever built, since they come from portal
* users rather than the operator.
*/
@Service
@ConditionalOnBooleanProperty(name = "policies.enabled")
public class S3ConnectionPool {
private final ApplicationProperties applicationProperties;
private final Function<S3Config, S3Client> clientFactory;
private final Map<S3Config, S3Client> clients = new ConcurrentHashMap<>();
@Autowired
public S3ConnectionPool(ApplicationProperties applicationProperties) {
this(applicationProperties, S3ConnectionPool::buildClient);
}
/** Factory-injecting constructor for tests. */
public S3ConnectionPool(
ApplicationProperties applicationProperties,
Function<S3Config, S3Client> clientFactory) {
this.applicationProperties = applicationProperties;
this.clientFactory = clientFactory;
}
public S3Client clientFor(S3Config config) {
return clients.computeIfAbsent(
config,
c -> {
requirePermittedEndpoint(c);
return clientFactory.apply(c);
});
}
/**
* A user-supplied endpoint must not reach loopback, link-local, or private addresses unless the
* operator has opted in via {@code policies.allowPrivateS3Endpoints}.
*/
private void requirePermittedEndpoint(S3Config config) {
if (config.endpoint() == null) {
return;
}
try {
S3Clients.validateEndpointHost(
URI.create(config.endpoint()),
applicationProperties.getPolicies().isAllowPrivateS3Endpoints(),
"S3 source endpoint",
"set policies.allowPrivateS3Endpoints=true to opt in (e.g. for a local"
+ " MinIO).");
} catch (IllegalStateException e) {
throw new IllegalArgumentException(e.getMessage(), e);
}
}
private static S3Client buildClient(S3Config config) {
S3ClientBuilder builder =
S3Client.builder()
.httpClient(UrlConnectionHttpClient.create())
.region(Region.of(config.region()))
// Path-style addressing whenever a custom endpoint is set: S3-compatible
// stores rarely support virtual-hosted bucket DNS.
.serviceConfiguration(
S3Configuration.builder()
.pathStyleAccessEnabled(config.endpoint() != null)
.build())
.credentialsProvider(
StaticCredentialsProvider.create(
AwsBasicCredentials.create(
config.accessKeyId(), config.secretAccessKey())));
if (config.endpoint() != null) {
builder.endpointOverride(URI.create(config.endpoint()));
}
return builder.build();
}
@PreDestroy
void closeClients() {
clients.values().forEach(S3Client::close);
clients.clear();
}
}
@@ -0,0 +1,28 @@
package stirling.software.proprietary.policy.s3;
import java.time.Instant;
/**
* The S3 backend's identity and version scheme, shared by {@code S3InputSource} and {@code
* S3OutputSink} so outputs are recorded under exactly the identity and gate the next listing
* derives. Identity is {@code s3://bucket/key}; the gate is the ETag every listing returns for free
* (multipart ETags are not content hashes, so any ETag change simply reads as a new version).
*/
public final class S3Identities {
private S3Identities() {}
public static String identity(String bucket, String key) {
return "s3://" + bucket + "/" + key;
}
/** ETag stripped of its quotes; falls back to size:lastModified for stores that omit it. */
public static String gate(String eTag, Long size, Instant lastModified) {
if (eTag != null && !eTag.isBlank()) {
return eTag.replace("\"", "");
}
return (size == null ? -1 : size)
+ ":"
+ (lastModified == null ? 0 : lastModified.toEpochMilli());
}
}
@@ -0,0 +1,27 @@
package stirling.software.proprietary.policy.source;
/**
* The Editor as a virtual, always-present source. Unlike a persisted {@link Source} it is neither
* stored nor configurable: it stands for the documents a team processes by running policies from
* the in-app editor (the {@code POST /api/v1/policies/{id}/run} path). Its throughput is tracked
* through {@link SourceDocCounter} under a synthetic, team-scoped key, so each team sees only its
* own editor activity and the client is only ever handed the opaque {@link #ID}, never a team.
*/
public final class EditorSource {
/** The single, stable id and type the client sees for the editor row. */
public static final String ID = "editor";
public static final String TYPE = "editor";
private EditorSource() {}
/**
* The per-team {@link SourceDocCounter} key. A {@code null} team (login disabled / self-hosted
* single user) shares one global bucket; otherwise counts are partitioned by team so a team's
* total aggregates every member's editor runs and no other team's.
*/
public static String counterKey(Long teamId) {
return teamId == null ? ID : ID + ":" + teamId;
}
}
@@ -34,6 +34,13 @@ public class SourceAccessGuard {
/** Team a new source is stamped with: the creator's team. {@code null} when login disabled. */
public Long teamForNewSource() {
return currentTeamId();
}
/**
* The current user's team (what scopes their sources), or {@code null} when login is disabled.
*/
public Long currentTeamId() {
return enforced() ? policyManagementAuthority.currentUserTeamId() : null;
}
@@ -1,6 +1,7 @@
package stirling.software.proprietary.policy.source;
import java.util.List;
import java.util.Map;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBooleanProperty;
import org.springframework.http.HttpStatus;
@@ -29,6 +30,7 @@ import stirling.software.proprietary.policy.model.InputSpec;
import stirling.software.proprietary.policy.model.Policy;
import stirling.software.proprietary.policy.store.PolicyStore;
import stirling.software.proprietary.policy.trigger.PolicyTriggerManager;
import stirling.software.proprietary.util.SecretMasker;
/**
* CRUD for persisted, reusable input connections plus the Sources overview for the admin portal. A
@@ -65,11 +67,16 @@ public class SourceController {
}
@GetMapping("/{sourceId}")
@Operation(summary = "Get a source by id")
@Operation(
summary = "Get a source by id",
description =
"Secret-bearing options are returned as a redaction sentinel, never their"
+ " stored values; an edit that sends the sentinel back keeps them.")
public ResponseEntity<Source> get(@PathVariable String sourceId) {
return sourceStore
.get(sourceId)
.filter(sourceAccessGuard::canAccess)
.map(SourceController::withMaskedSecrets)
.map(ResponseEntity::ok)
.orElseGet(() -> ResponseEntity.notFound().build());
}
@@ -81,6 +88,10 @@ public class SourceController {
"The trailing 30-day per-day document series (oldest first) for the source's"
+ " sparkline.")
public ResponseEntity<List<Long>> documentCounts(@PathVariable String sourceId) {
// The editor is virtual: its series is tracked per team, not against a persisted source.
if (EditorSource.ID.equals(sourceId)) {
return ResponseEntity.ok(overviewService.editorDailySeries());
}
return sourceStore
.get(sourceId)
.filter(sourceAccessGuard::canAccess)
@@ -97,7 +108,8 @@ public class SourceController {
+ " matching source type.")
public ResponseEntity<Source> save(@RequestBody Source source) {
requireSourceEditingAllowed();
Source owned = resolveOwnership(source);
requireNotEditor(source.id(), source.type());
Source owned = withStoredSecrets(resolveOwnership(source));
try {
validateConfig(owned);
} catch (IllegalArgumentException e) {
@@ -107,7 +119,7 @@ public class SourceController {
// An edited folder source can change which directory needs watching, so re-sync trigger
// registrations now instead of waiting for the next reconcile.
policyTriggerManager.notifyPoliciesChanged();
return ResponseEntity.ok(saved);
return ResponseEntity.ok(withMaskedSecrets(saved));
}
@DeleteMapping("/{sourceId}")
@@ -118,6 +130,7 @@ public class SourceController {
+ " so the connection can't be pulled out from under a live policy.")
public ResponseEntity<Void> delete(@PathVariable String sourceId) {
requireSourceEditingAllowed();
requireNotEditor(sourceId, null);
Source source = sourceStore.get(sourceId).filter(sourceAccessGuard::canAccess).orElse(null);
if (source == null) {
return ResponseEntity.notFound().build();
@@ -169,6 +182,42 @@ public class SourceController {
teamId);
}
private static Source withOptions(Source source, Map<String, Object> options) {
return new Source(
source.id(),
source.name(),
source.type(),
options,
source.enabled(),
source.owner(),
source.teamId());
}
/** Secrets never leave the server: reads return the redaction sentinel in their place. */
private static Source withMaskedSecrets(Source source) {
return withOptions(source, SecretMasker.mask(source.options()));
}
/**
* An edit that round-trips a masked read sends secrets back as the sentinel; restore them from
* the stored source so saving without re-typing keeps them (validation then runs against the
* real values).
*/
private Source withStoredSecrets(Source incoming) {
if (incoming.id() == null || incoming.id().isBlank()) {
return incoming;
}
return sourceStore
.get(incoming.id())
.map(
existing ->
withOptions(
incoming,
SecretMasker.restoreRedacted(
incoming.options(), existing.options())))
.orElse(incoming);
}
/** Validate the config against the bean that handles the source's type, as the engine will. */
private void validateConfig(Source source) {
InputSpec spec = source.toInputSpec();
@@ -195,6 +244,18 @@ public class SourceController {
}
}
/**
* The editor is a built-in, virtual source: it is always present and cannot be created, edited,
* or deleted like a persisted connection. Reject any attempt to touch it by id or type.
*/
private static void requireNotEditor(String id, String type) {
if (EditorSource.ID.equals(id) || EditorSource.TYPE.equals(type)) {
throw new ResponseStatusException(
HttpStatus.BAD_REQUEST,
"The editor is a built-in source and cannot be created, edited, or deleted");
}
}
/** Names of the caller's visible policies that reference the given source. */
private List<String> referencingPolicyNames(String sourceId) {
return policyAccessGuard.visibleFrom(policyStore).stream()
@@ -3,6 +3,7 @@ package stirling.software.proprietary.policy.source;
import java.io.Serializable;
import jakarta.persistence.Column;
import jakarta.persistence.Convert;
import jakarta.persistence.Entity;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
@@ -11,6 +12,8 @@ import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import stirling.software.proprietary.integration.crypto.LenientEncryptedStringConverter;
/**
* JPA row for a {@link Source}. The whole source lives as JSON in {@code sourceJson} (authoritative
* on read); the scalar columns are denormalized copies for querying. {@code owner} and {@code
@@ -45,6 +48,9 @@ public class SourceEntity implements Serializable {
@Column(name = "enabled")
private boolean enabled;
// Encrypted at rest: source options carry user-supplied credentials (e.g. an S3 secret
// access key). Lenient so rows written before encryption shipped still load.
@Convert(converter = LenientEncryptedStringConverter.class)
@Column(name = "source_json", columnDefinition = "text")
private String sourceJson;
}
@@ -14,6 +14,7 @@ import lombok.RequiredArgsConstructor;
import stirling.software.proprietary.policy.config.PolicyAccessGuard;
import stirling.software.proprietary.policy.model.Policy;
import stirling.software.proprietary.policy.store.PolicyStore;
import stirling.software.proprietary.util.SecretMasker;
/**
* Builds the Sources overview: every persisted source the caller's team owns, shown exactly once,
@@ -40,7 +41,7 @@ public class SourceOverviewService {
Map<String, DocStats> docStats =
docCounter.statsFor(sources.stream().map(Source::id).toList());
List<SourceView> views =
List<SourceView> persisted =
sources.stream()
.map(
source ->
@@ -55,7 +56,13 @@ public class SourceOverviewService {
.thenComparing(SourceView::name))
.toList();
return new SourcesResponse(buildKpis(views), views);
// The editor is a built-in source: always present and pinned first. The KPI strip counts
// only the connections a team configures, so the editor is left out of the KPIs.
List<SourceView> views = new ArrayList<>();
views.add(editorView(policies));
views.addAll(persisted);
return new SourcesResponse(buildKpis(persisted), views);
}
/**
@@ -66,6 +73,49 @@ public class SourceOverviewService {
return docCounter.dailySeriesFor(sourceId);
}
/** The 30-day daily editor document series (oldest first) for the caller's team. */
public List<Long> editorDailySeries() {
return docCounter.dailySeriesFor(
EditorSource.counterKey(sourceAccessGuard.currentTeamId()));
}
/**
* The always-present editor row. It has no stored config; its documents are those the team has
* processed by running policies from the editor, and it is "used by" every policy that targets
* the editor as its source.
*/
private SourceView editorView(List<Policy> policies) {
String key = EditorSource.counterKey(sourceAccessGuard.currentTeamId());
DocStats docs = docCounter.statsFor(List.of(key)).getOrDefault(key, DocStats.ZERO);
List<SourceView.PolicyRef> refs =
policies.stream()
.filter(SourceOverviewService::runsFromEditor)
.map(policy -> new SourceView.PolicyRef(policy.id(), policy.name()))
.toList();
return new SourceView(
EditorSource.ID,
"Editor",
EditorSource.TYPE,
"active",
refs.size(),
refs,
List.of(),
docs.total(),
docs.last24h(),
docs.last30d());
}
/**
* Whether a policy runs from the editor. Editor membership is carried in the policy's output
* metadata ({@code output.options.sources}) - a client-side list the editor writes when a
* policy targets it - rather than as a persisted {@code sourceId}, because the editor is
* virtual and has no stored source to reference.
*/
private static boolean runsFromEditor(Policy policy) {
Object sources = policy.output().options().get("sources");
return sources instanceof List<?> list && list.contains(EditorSource.ID);
}
/** Policies referencing each source id, across the caller's visible policies. */
private static Map<String, List<Policy>> referencesBySource(List<Policy> policies) {
Map<String, List<Policy>> bySource = new HashMap<>();
@@ -104,13 +154,18 @@ public class SourceOverviewService {
return referenceCount == 0 ? "unused" : "active";
}
/** Generic key/value view of the source's config - works for any source type. */
/**
* Generic key/value view of the source's config - works for any source type. Secret-bearing
* options (e.g. an S3 secret access key) are redacted, not omitted, so the overview still shows
* that a credential is configured.
*/
private static List<SourceView.DetailRow> configRows(Source source) {
return source.options().entrySet().stream()
Map<String, Object> masked = SecretMasker.mask(source.options());
return source.options().keySet().stream()
.map(
entry ->
key ->
new SourceView.DetailRow(
humanize(entry.getKey()), String.valueOf(entry.getValue())))
humanize(key), String.valueOf(masked.get(key))))
.toList();
}
@@ -3,6 +3,7 @@ package stirling.software.proprietary.policy.store;
import java.io.Serializable;
import jakarta.persistence.Column;
import jakarta.persistence.Convert;
import jakarta.persistence.Entity;
import jakarta.persistence.Id;
import jakarta.persistence.Table;
@@ -11,6 +12,8 @@ import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import stirling.software.proprietary.integration.crypto.LenientEncryptedStringConverter;
/**
* JPA row for a {@link stirling.software.proprietary.policy.model.Policy}. The whole policy lives
* as JSON in {@code policyJson} (authoritative on read); the scalar columns are denormalized copies
@@ -55,6 +58,9 @@ public class PolicyEntity implements Serializable {
@Column(name = "sort_order")
private Integer sortOrder;
// Encrypted at rest: output options carry user-supplied credentials (e.g. an S3 secret
// access key). Lenient so rows written before encryption shipped still load.
@Convert(converter = LenientEncryptedStringConverter.class)
@Column(name = "policy_json", columnDefinition = "text")
private String policyJson;
}
@@ -99,11 +99,17 @@ public class ScheduleTrigger implements PolicyTrigger {
// Baseline a newly-seen policy to now so it does not fire immediately.
Instant last = lastFiredByPolicy.computeIfAbsent(policy.id(), id -> now);
ZonedDateTime next = config.schedule().nextAfter(last.atZone(config.zone()));
if (!next.toInstant().isAfter(now)) {
lastFiredByPolicy.put(policy.id(), now);
log.info("Scheduled policy {} ({}) is due", policy.id(), policy.name());
policyRunner.run(policy);
if (next.toInstant().isAfter(now)) {
continue;
}
ZonedDateTime later = config.schedule().nextAfter(next);
while (!later.toInstant().isAfter(now)) {
next = later;
later = config.schedule().nextAfter(later);
}
lastFiredByPolicy.put(policy.id(), next.toInstant());
log.info("Scheduled policy {} ({}) is due", policy.id(), policy.name());
policyRunner.run(policy);
}
}
@@ -276,4 +276,25 @@ public interface PersistentAuditEventRepository extends JpaRepository<Persistent
@Param("source") String source,
@Param("excludeType") String excludeType,
@Param("since") Instant since);
// Team-scoped (SaaS) variants: same free-UI counts, constrained to a team's member principals.
@Query(
"SELECT COUNT(e) FROM PersistentAuditEvent e "
+ "WHERE e.type IN :types AND e.source = :source "
+ "AND e.principal IN :principals AND e.timestamp > :since")
long countByTypeInAndSourceAndPrincipalInAndTimestampAfter(
@Param("types") List<String> types,
@Param("source") String source,
@Param("principals") List<String> principals,
@Param("since") Instant since);
@Query(
"SELECT COUNT(DISTINCT e.principal) FROM PersistentAuditEvent e "
+ "WHERE e.source = :source AND e.type <> :excludeType "
+ "AND e.principal IN :principals AND e.timestamp > :since")
long countDistinctPrincipalsBySourceExcludingTypeAndPrincipalInAfter(
@Param("source") String source,
@Param("excludeType") String excludeType,
@Param("principals") List<String> principals,
@Param("since") Instant since);
}
@@ -7,6 +7,7 @@ import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.stream.Collectors;
import org.apache.commons.io.FilenameUtils;
@@ -27,6 +28,7 @@ import lombok.Data;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.common.service.AutomationRunContext;
import stirling.software.common.service.CustomPDFDocumentFactory;
import stirling.software.common.service.FileStorage;
import stirling.software.common.service.InternalApiTimeoutException;
@@ -156,33 +158,41 @@ public class AiWorkflowService {
throws IOException {
validateRequest(request);
// Key by opaque file id, not filename. Filenames aren't guaranteed unique across an
// upload (users can rotate the same 'scan.pdf' twice), and the engine identifies files
// by id in every response shape that asks Java to look a file up again.
Map<String, MultipartFile> filesById = new LinkedHashMap<>();
List<AiFile> files = new ArrayList<>();
for (AiWorkflowFileInput fileInput : request.getFileInputs()) {
MultipartFile multipartFile = fileInput.getFileInput();
AiFile aiFile =
new AiFile(
fileIdStrategy.idFor(multipartFile),
multipartFile.getOriginalFilename());
filesById.put(aiFile.getId(), multipartFile);
files.add(aiFile);
}
// One AI orchestration = one automation run. Scope a run id (on whichever thread runs
// orchestrate — request thread for sync, stream-executor for streaming) so every tool
// sub-step it dispatches via PolicyExecutor → InternalApiClient groups into one charge.
try (AutomationRunContext.Scope ignored =
AutomationRunContext.open(UUID.randomUUID().toString())) {
WorkflowTurnRequest initialRequest = new WorkflowTurnRequest();
initialRequest.setUserMessage(request.getUserMessage().trim());
initialRequest.setFiles(files);
initialRequest.setConversationHistory(new ArrayList<>(request.getConversationHistory()));
initialRequest.setEnabledEndpoints(endpointResolver.getEnabledEndpointUrls());
listener.onProgress(AiWorkflowProgressEvent.of(AiWorkflowPhase.ANALYZING));
// Key by opaque file id, not filename. Filenames aren't guaranteed unique across an
// upload (users can rotate the same 'scan.pdf' twice), and the engine identifies files
// by id in every response shape that asks Java to look a file up again.
Map<String, MultipartFile> filesById = new LinkedHashMap<>();
List<AiFile> files = new ArrayList<>();
for (AiWorkflowFileInput fileInput : request.getFileInputs()) {
MultipartFile multipartFile = fileInput.getFileInput();
AiFile aiFile =
new AiFile(
fileIdStrategy.idFor(multipartFile),
multipartFile.getOriginalFilename());
filesById.put(aiFile.getId(), multipartFile);
files.add(aiFile);
}
WorkflowState state = new WorkflowState.Pending(initialRequest);
while (state instanceof WorkflowState.Pending pending) {
state = advance(pending.request(), filesById, listener);
WorkflowTurnRequest initialRequest = new WorkflowTurnRequest();
initialRequest.setUserMessage(request.getUserMessage().trim());
initialRequest.setFiles(files);
initialRequest.setConversationHistory(
new ArrayList<>(request.getConversationHistory()));
initialRequest.setEnabledEndpoints(endpointResolver.getEnabledEndpointUrls());
listener.onProgress(AiWorkflowProgressEvent.of(AiWorkflowPhase.ANALYZING));
WorkflowState state = new WorkflowState.Pending(initialRequest);
while (state instanceof WorkflowState.Pending pending) {
state = advance(pending.request(), filesById, listener);
}
return ((WorkflowState.Terminal) state).response();
}
return ((WorkflowState.Terminal) state).response();
}
private WorkflowState advance(
@@ -1,5 +1,6 @@
package stirling.software.proprietary.util;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.regex.Pattern;
@@ -13,10 +14,15 @@ import stirling.software.common.util.RegexPatternUtils;
@Slf4j
public final class SecretMasker {
/** The placeholder masked values are replaced with; reads as "a secret is set". */
public static final String REDACTED = "********";
private static final Pattern SENSITIVE =
RegexPatternUtils.getInstance()
.getPattern(
"(?i)\\b(password|token|secret|api[_-]?key|authorization|auth|jwt|cred|cert)\\b");
// secret[_-]?access[_-]?key precedes plain secret so camelCase keys
// like secretAccessKey (no word boundary after "secret") still match.
"(?i)\\b(password|token|secret[_-]?access[_-]?key|secret|api[_-]?key|authorization|auth|jwt|cred|cert)\\b");
private SecretMasker() {}
@@ -47,8 +53,28 @@ public final class SecretMasker {
private static Object deepMaskValue(String key, Object value) {
if (key != null && SENSITIVE.matcher(key).find()) {
return "***REDACTED***";
return REDACTED;
}
return deepMask(value);
}
/**
* Restore top-level values the caller sent back as the {@link #REDACTED} sentinel from the
* stored map, so a masked read can round-trip through an edit without re-typing secrets. A
* sentinel with no stored counterpart is left as-is (it fails whatever validates it, rather
* than silently passing an unset secret).
*/
public static Map<String, Object> restoreRedacted(
Map<String, Object> incoming, Map<String, Object> stored) {
if (incoming == null || stored == null) {
return incoming;
}
Map<String, Object> merged = new LinkedHashMap<>(incoming);
merged.replaceAll(
(key, value) ->
REDACTED.equals(value) && stored.containsKey(key)
? stored.get(key)
: value);
return merged;
}
}
@@ -0,0 +1,45 @@
package stirling.software.proprietary.integration.crypto;
import static org.assertj.core.api.Assertions.assertThat;
import javax.crypto.KeyGenerator;
import javax.crypto.SecretKey;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
class LenientEncryptedStringConverterTest {
private final LenientEncryptedStringConverter converter = new LenientEncryptedStringConverter();
@BeforeAll
static void initKey() throws Exception {
KeyGenerator generator = KeyGenerator.getInstance("AES");
generator.init(256);
SecretKey key = generator.generateKey();
CredentialEncryption.initialiseForTesting(key);
}
@Test
void roundTripsThroughCiphertext() {
String json = "{\"bucket\":\"inbox\",\"secretAccessKey\":\"shh\"}";
String stored = converter.convertToDatabaseColumn(json);
assertThat(stored).isNotEqualTo(json).doesNotContain("shh");
assertThat(converter.convertToEntityAttribute(stored)).isEqualTo(json);
}
@Test
void legacyPlaintextRowsPassThroughOnRead() {
String legacy = "{\"bucket\":\"inbox\",\"mode\":\"consume\"}";
assertThat(converter.convertToEntityAttribute(legacy)).isEqualTo(legacy);
}
@Test
void nullsPassThrough() {
assertThat(converter.convertToDatabaseColumn(null)).isNull();
assertThat(converter.convertToEntityAttribute(null)).isNull();
}
}
@@ -9,6 +9,7 @@ import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.concurrent.CompletableFuture;
@@ -17,6 +18,7 @@ import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Nested;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.ArgumentCaptor;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.http.HttpStatus;
@@ -34,16 +36,21 @@ import stirling.software.proprietary.policy.engine.PolicyRunHandle;
import stirling.software.proprietary.policy.engine.PolicyRunRegistry;
import stirling.software.proprietary.policy.engine.PolicyRunner;
import stirling.software.proprietary.policy.engine.PolicyValidator;
import stirling.software.proprietary.policy.engine.SweepOutcome;
import stirling.software.proprietary.policy.ledger.ProcessedLedger;
import stirling.software.proprietary.policy.model.OutputSpec;
import stirling.software.proprietary.policy.model.PipelineDefinition;
import stirling.software.proprietary.policy.model.PipelineStep;
import stirling.software.proprietary.policy.model.Policy;
import stirling.software.proprietary.policy.model.PolicyRun;
import stirling.software.proprietary.policy.model.PolicyRunView;
import stirling.software.proprietary.policy.progress.PolicyProgressListener;
import stirling.software.proprietary.policy.source.EditorSource;
import stirling.software.proprietary.policy.source.SourceAccessGuard;
import stirling.software.proprietary.policy.source.SourceDocCounter;
import stirling.software.proprietary.policy.source.SourceStore;
import stirling.software.proprietary.policy.trigger.PolicyTriggerManager;
import stirling.software.proprietary.util.SecretMasker;
@ExtendWith(MockitoExtension.class)
@DisplayName("PolicyController")
@@ -54,6 +61,7 @@ class PolicyControllerTest {
@Mock private stirling.software.proprietary.policy.store.PolicyStore policyStore;
@Mock private SourceStore sourceStore;
@Mock private SourceAccessGuard sourceAccessGuard;
@Mock private SourceDocCounter docCounter;
@Mock private PolicyValidator policyValidator;
@Mock private PolicyAccessGuard policyAccessGuard;
@Mock private PolicyManagementAuthority policyManagementAuthority;
@@ -87,6 +95,7 @@ class PolicyControllerTest {
policyStore,
sourceStore,
sourceAccessGuard,
docCounter,
policyValidator,
policyAccessGuard,
policyManagementAuthority,
@@ -128,6 +137,17 @@ class PolicyControllerTest {
return new Policy(id, "name", "owner", true, null, List.of(), List.of(), null, teamId);
}
private static Policy s3OutputPolicy(String id, String secret) {
OutputSpec output =
new OutputSpec(
"s3",
Map.of(
"bucket", "outbox",
"accessKeyId", "AKIAEXAMPLE",
"secretAccessKey", secret));
return new Policy(id, "name", "owner", true, null, List.of(), List.of(), output, 1L);
}
private static PolicyRunHandle handle(String runId) {
PolicyRun run = new PolicyRun(runId, null, definitionWithStep());
return new PolicyRunHandle(runId, CompletableFuture.completedFuture(run));
@@ -150,6 +170,18 @@ class PolicyControllerTest {
assertThat(response.getBody().getJobId()).isEqualTo("run-1");
}
@Test
@DisplayName("feeds the editor source, scoped to the caller's team")
void adHocRunFeedsTheEditorSource() throws Exception {
when(policyRunner.runAdHoc(any(), any(), eq(PolicyProgressListener.NOOP)))
.thenReturn(handle("run-1"));
when(sourceAccessGuard.currentTeamId()).thenReturn(3L);
controller.run(definitionWithStep(), new PolicyRunFiles());
verify(docCounter).record(EditorSource.counterKey(3L), 0L);
}
@Test
@DisplayName("rejects a pipeline with no steps")
void rejectsEmptyPipeline() {
@@ -263,6 +295,27 @@ class PolicyControllerTest {
verify(policyTriggerManager).notifyPoliciesChanged();
}
@Test
@DisplayName("saving the sentinel back keeps the stored output secret")
void saveRestoresOutputSecrets() {
applicationProperties.getSecurity().setEnableLogin(false);
Policy existing = s3OutputPolicy("p1", "shh");
when(policyStore.get("p1")).thenReturn(Optional.of(existing));
when(policyAccessGuard.canAccess(existing)).thenReturn(true);
when(policyStore.save(any())).thenAnswer(i -> i.getArgument(0));
ResponseEntity<Policy> response =
controller.savePolicy(s3OutputPolicy("p1", SecretMasker.REDACTED));
ArgumentCaptor<Policy> stored = ArgumentCaptor.forClass(Policy.class);
verify(policyStore).save(stored.capture());
assertThat(stored.getValue().output().options().get("secretAccessKey"))
.isEqualTo("shh");
// The save response is masked again; only the store sees the real value.
assertThat(response.getBody().output().options().get("secretAccessKey"))
.isEqualTo(SecretMasker.REDACTED);
}
@Test
@DisplayName("forbidden when login enabled and caller cannot edit")
void forbidden() {
@@ -363,6 +416,20 @@ class PolicyControllerTest {
assertThat(response.getBody().id()).isEqualTo("a");
}
@Test
@DisplayName("getPolicy returns output secrets as the redaction sentinel")
void getMasksOutputSecrets() {
Policy p = s3OutputPolicy("a", "shh");
when(policyStore.get("a")).thenReturn(Optional.of(p));
when(policyAccessGuard.canAccess(p)).thenReturn(true);
Policy read = controller.getPolicy("a").getBody();
assertThat(read.output().options().get("secretAccessKey"))
.isEqualTo(SecretMasker.REDACTED);
assertThat(read.output().options().get("bucket")).isEqualTo("outbox");
}
@Test
@DisplayName("getPolicy returns 404 when not accessible")
void getNotAccessible() {
@@ -536,17 +603,18 @@ class PolicyControllerTest {
}
@Test
@DisplayName("trigger runs an accessible policy against its sources and returns run ids")
@DisplayName("trigger runs an accessible policy against its sources and returns the sweep")
void triggersRun() {
Policy p = policy("a", 1L);
when(policyStore.get("a")).thenReturn(Optional.of(p));
when(policyAccessGuard.canAccess(p)).thenReturn(true);
when(policyRunner.run(p)).thenReturn(List.of("run-a", "run-b"));
SweepOutcome outcome = new SweepOutcome(List.of("run-a", "run-b"), 3, 1, 0, 0);
when(policyRunner.run(p)).thenReturn(outcome);
ResponseEntity<List<String>> response = controller.trigger("a");
ResponseEntity<SweepOutcome> response = controller.trigger("a");
assertThat(response.getStatusCode()).isEqualTo(HttpStatus.ACCEPTED);
assertThat(response.getBody()).containsExactly("run-a", "run-b");
assertThat(response.getBody()).isEqualTo(outcome);
}
@Test
@@ -1,5 +1,6 @@
package stirling.software.proprietary.policy.engine;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertSame;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -26,10 +27,12 @@ import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.ArgumentCaptor;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.core.io.ByteArrayResource;
import stirling.software.proprietary.policy.input.InputSource;
import stirling.software.proprietary.policy.input.ResolveContext;
import stirling.software.proprietary.policy.input.ResolvedInput;
import stirling.software.proprietary.policy.ledger.InProcessProcessedLedger;
import stirling.software.proprietary.policy.ledger.ProcessedLedger;
import stirling.software.proprietary.policy.model.InputSpec;
import stirling.software.proprietary.policy.model.OutputSpec;
@@ -39,6 +42,7 @@ import stirling.software.proprietary.policy.model.PolicyInputs;
import stirling.software.proprietary.policy.model.PolicyRun;
import stirling.software.proprietary.policy.model.PolicyRunStatus;
import stirling.software.proprietary.policy.progress.PolicyProgressListener;
import stirling.software.proprietary.policy.source.EditorSource;
import stirling.software.proprietary.policy.source.InProcessSourceDocCounter;
import stirling.software.proprietary.policy.source.InProcessSourceStore;
import stirling.software.proprietary.policy.source.Source;
@@ -56,6 +60,7 @@ class PolicyRunnerTest {
@Mock private ProcessedLedger processedLedger;
private final SourceStore sourceStore = new InProcessSourceStore();
private final InProcessSourceDocCounter docCounter = new InProcessSourceDocCounter();
private PolicyRunner runner;
@BeforeEach
@@ -65,7 +70,7 @@ class PolicyRunnerTest {
policyEngine,
List.of(folderSource),
sourceStore,
new InProcessSourceDocCounter(),
docCounter,
processedLedger);
}
@@ -85,6 +90,42 @@ class PolicyRunnerTest {
verify(processedLedger).deleteUnseen(eq("p1"), anyLong());
}
@Test
void reportsWhatTheSweepSkippedSoAnEmptyTriggerExplainsItself() throws Exception {
InProcessProcessedLedger ledger = new InProcessProcessedLedger();
PolicyRunner reporting =
new PolicyRunner(
policyEngine,
List.of(folderSource),
sourceStore,
new InProcessSourceDocCounter(),
ledger);
InputSpec spec = InputSpec.folder("/in");
Policy policy = policy(List.of(spec));
// One file already processed at its current version, one parked by a failed run.
ledger.claim("p1", "/in/done.pdf", "g1", null);
ledger.settle("p1", "/in/done.pdf", "g1", null, true);
ledger.claim("p1", "/in/failed.pdf", "g2", null);
ledger.settle("p1", "/in/failed.pdf", "g2", null, false);
when(folderSource.supports(spec)).thenReturn(true);
when(folderSource.resolve(eq(spec), any()))
.thenAnswer(
invocation -> {
ResolveContext ctx = invocation.getArgument(1);
ctx.reportPresent(List.of("/in/done.pdf", "/in/failed.pdf"));
// Both are at their settled versions, so neither claims.
return List.of();
});
SweepOutcome outcome = reporting.run(policy);
assertTrue(outcome.runIds().isEmpty());
assertEquals(2, outcome.filesListed());
assertEquals(1, outcome.alreadyProcessed());
assertEquals(1, outcome.parked());
assertEquals(0, outcome.inFlight());
}
@Test
void pullsEverySourceAndRunsOnePerUnitOfWork() throws Exception {
InputSpec spec = InputSpec.folder("/in");
@@ -257,6 +298,33 @@ class PolicyRunnerTest {
verifyNoInteractions(folderSource);
}
@Test
void runWithRecordsSuppliedDocsAgainstTheEditorSourceForThePolicyTeam() {
Policy policy =
new Policy(
"p1",
"p",
"owner",
true,
null,
List.of(),
List.of(new PipelineStep("/api/v1/misc/compress-pdf", Map.of())),
OutputSpec.inline(),
7L);
PolicyInputs inputs =
PolicyInputs.of(
List.of(
new ByteArrayResource("a".getBytes()),
new ByteArrayResource("b".getBytes())));
when(policyEngine.runPolicy(policy, inputs, PolicyProgressListener.NOOP))
.thenReturn(new PolicyRunHandle("r", new CompletableFuture<>()));
runner.runWith(policy, inputs, PolicyProgressListener.NOOP);
String key = EditorSource.counterKey(7L);
assertEquals(2, docCounter.statsFor(List.of(key)).get(key).total());
}
/** Persists each spec as a source and returns a policy referencing them by id. */
private Policy policy(List<InputSpec> sources) {
List<String> sourceIds =
@@ -0,0 +1,231 @@
package stirling.software.proprietary.policy.input;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.function.Supplier;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.testcontainers.containers.MinIOContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.proprietary.policy.ledger.InProcessProcessedLedger;
import stirling.software.proprietary.policy.model.InputSpec;
import stirling.software.proprietary.policy.s3.S3ConnectionPool;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3Configuration;
import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
import software.amazon.awssdk.services.s3.model.NoSuchKeyException;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
/**
* End-to-end {@link S3InputSource} test against a real S3 API (MinIO), through the production
* client factory: listing, claiming, streaming, consensus delete, and save-time validation.
*/
@Testcontainers(disabledWithoutDocker = true)
class S3InputSourceMinioTest {
private static final String POLICY = "p1";
private static final String ACCESS_KEY = "minioadmin";
private static final String SECRET_KEY = "minioadmin";
@Container
static MinIOContainer minio =
new MinIOContainer("minio/minio:latest")
.withUserName(ACCESS_KEY)
.withPassword(SECRET_KEY);
private static S3Client adminClient;
private static int bucketCounter;
private String bucket;
private S3InputSource source;
private InProcessProcessedLedger ledger;
private RecordingContext ctx;
@BeforeEach
void setUp() {
if (adminClient == null) {
adminClient =
S3Client.builder()
.endpointOverride(java.net.URI.create(minio.getS3URL()))
.httpClient(UrlConnectionHttpClient.create())
.region(Region.US_EAST_1)
.credentialsProvider(
StaticCredentialsProvider.create(
AwsBasicCredentials.create(ACCESS_KEY, SECRET_KEY)))
.serviceConfiguration(
S3Configuration.builder().pathStyleAccessEnabled(true).build())
.build();
}
bucket = "policy-inbox-" + ++bucketCounter;
adminClient.createBucket(CreateBucketRequest.builder().bucket(bucket).build());
// The MinIO endpoint resolves to loopback, so the operator opt-in must be on.
ApplicationProperties properties = new ApplicationProperties();
properties.getPolicies().setAllowPrivateS3Endpoints(true);
source = new S3InputSource(new S3ConnectionPool(properties));
ledger = new InProcessProcessedLedger();
ctx = new RecordingContext();
}
@Test
void consumeListsStreamsAndDeletesByConsensus() throws IOException {
put("incoming/doc.pdf", "pdf bytes");
put("incoming/other.txt", "text");
List<ResolvedInput> work = source.resolve(spec(Map.of("prefix", "incoming/")), ctx);
assertThat(work).hasSize(2);
assertThat(ctx.present)
.containsExactlyInAnyOrder(
"s3://" + bucket + "/incoming/doc.pdf",
"s3://" + bucket + "/incoming/other.txt");
assertThat(read(work.get(0))).isIn("pdf bytes", "text");
// In flight: nothing to claim on a second sweep.
assertThat(source.resolve(spec(Map.of("prefix", "incoming/")), ctx)).isEmpty();
work.forEach(unit -> unit.onComplete().accept(true));
assertThat(exists("incoming/doc.pdf")).isFalse();
assertThat(exists("incoming/other.txt")).isFalse();
}
@Test
void aFailedObjectStaysInTheBucket() throws IOException {
put("doc.pdf", "data");
source.resolve(spec(Map.of()), ctx).get(0).onComplete().accept(false);
assertThat(exists("doc.pdf")).isTrue();
assertThat(source.resolve(spec(Map.of()), ctx)).isEmpty();
}
@Test
void anObjectOverwrittenMidRunSurvivesTheDeleteAndRunsAgain() throws IOException {
put("doc.pdf", "v1");
List<ResolvedInput> work = source.resolve(spec(Map.of()), ctx);
put("doc.pdf", "v2 with a different etag");
work.get(0).onComplete().accept(true);
assertThat(exists("doc.pdf")).isTrue();
assertThat(source.resolve(spec(Map.of()), ctx)).hasSize(1);
}
@Test
void prefixLimitsWhatIsRead() throws IOException {
put("incoming/doc.pdf", "data");
put("archive/old.pdf", "data");
List<ResolvedInput> work = source.resolve(spec(Map.of("prefix", "incoming/")), ctx);
assertThat(work).hasSize(1);
assertThat(ctx.present).containsExactly("s3://" + bucket + "/incoming/doc.pdf");
}
@Test
void validateAcceptsAReachableBucketAndRejectsBadCredentials() {
source.validate(spec(Map.of()));
Map<String, Object> wrongSecret = new HashMap<>(baseOptions());
wrongSecret.put("secretAccessKey", "not-the-secret");
assertThatThrownBy(() -> source.validate(new InputSpec("s3", wrongSecret)))
.isInstanceOf(IllegalArgumentException.class)
.hasMessageContaining("cannot access");
Map<String, Object> missingBucket = new HashMap<>(baseOptions());
missingBucket.put("bucket", "no-such-bucket-here");
assertThatThrownBy(() -> source.validate(new InputSpec("s3", missingBucket)))
.isInstanceOf(IllegalArgumentException.class)
.hasMessageContaining("cannot access");
}
@Test
void aPrivateEndpointIsRejectedWithoutTheOperatorOptIn() {
S3InputSource guarded =
new S3InputSource(new S3ConnectionPool(new ApplicationProperties()));
assertThatThrownBy(() -> guarded.validate(spec(Map.of())))
.isInstanceOf(IllegalArgumentException.class)
.hasMessageContaining("policies.allowPrivateS3Endpoints");
}
private Map<String, Object> baseOptions() {
return Map.of(
"bucket", bucket,
"endpoint", minio.getS3URL(),
"accessKeyId", ACCESS_KEY,
"secretAccessKey", SECRET_KEY);
}
private InputSpec spec(Map<String, Object> extra) {
Map<String, Object> options = new HashMap<>(baseOptions());
options.putAll(extra);
return new InputSpec("s3", options);
}
private void put(String key, String content) {
adminClient.putObject(
PutObjectRequest.builder().bucket(bucket).key(key).build(),
RequestBody.fromString(content, StandardCharsets.UTF_8));
}
private boolean exists(String key) {
try {
adminClient.headObject(HeadObjectRequest.builder().bucket(bucket).key(key).build());
return true;
} catch (NoSuchKeyException e) {
return false;
}
}
private static String read(ResolvedInput unit) throws IOException {
try (InputStream stream = unit.inputs().primary().get(0).getInputStream()) {
return new String(stream.readAllBytes(), StandardCharsets.UTF_8);
}
}
private class RecordingContext implements ResolveContext {
private final List<String> present = new ArrayList<>();
@Override
public boolean claim(String identity, String gate, Supplier<String> contentHash) {
return ledger.claim(POLICY, identity, gate, contentHash);
}
@Override
public void settle(
String identity, String finalGate, String finalContentHash, boolean success) {
ledger.settle(POLICY, identity, finalGate, finalContentHash, success);
}
@Override
public boolean allSettledDone(String identity) {
return ledger.allSettledDone(identity);
}
@Override
public void reportPresent(Collection<String> identities) {
present.addAll(identities);
}
}
}
@@ -0,0 +1,327 @@
package stirling.software.proprietary.policy.input;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.function.Supplier;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.proprietary.policy.ledger.InProcessProcessedLedger;
import stirling.software.proprietary.policy.model.InputSpec;
import stirling.software.proprietary.policy.s3.S3ConnectionPool;
import software.amazon.awssdk.core.ResponseInputStream;
import software.amazon.awssdk.core.exception.SdkClientException;
import software.amazon.awssdk.http.AbortableInputStream;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.DeleteObjectRequest;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.GetObjectResponse;
import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
import software.amazon.awssdk.services.s3.model.HeadObjectResponse;
import software.amazon.awssdk.services.s3.model.ListObjectsV2Request;
import software.amazon.awssdk.services.s3.model.ListObjectsV2Response;
import software.amazon.awssdk.services.s3.model.S3Object;
/**
* Tests for {@link S3InputSource}: consume mode tracks objects in place through the ledger and
* removes them by consensus, snapshot stays stateless, and discovery skips folder placeholders and
* dot-prefixed keys.
*/
@ExtendWith(MockitoExtension.class)
class S3InputSourceTest {
private static final String POLICY = "p1";
private static final String BUCKET = "inbox-bucket";
@Mock private S3Client s3Client;
private S3InputSource source;
private InProcessProcessedLedger ledger;
private RecordingContext ctx;
@BeforeEach
void setUp() {
source =
new S3InputSource(
new S3ConnectionPool(new ApplicationProperties(), config -> s3Client));
ledger = new InProcessProcessedLedger();
ctx = new RecordingContext();
}
@Test
void consumeRemovesTheObjectOnceProcessed() throws IOException {
listingReturns(object("doc.pdf", "\"etag-1\""));
headReturns("doc.pdf", "\"etag-1\"");
List<ResolvedInput> work = source.resolve(spec(), ctx);
assertEquals(1, work.size());
assertEquals(1, work.get(0).inputs().primary().size());
// In flight: a second sweep does not pick it up again.
assertTrue(source.resolve(spec(), ctx).isEmpty());
work.get(0).onComplete().accept(true);
verify(s3Client).deleteObject(any(DeleteObjectRequest.class));
assertTrue(source.resolve(spec(), ctx).isEmpty());
}
@Test
void anObjectReplacedMidRunSurvivesTheDelete() throws IOException {
listingReturns(object("doc.pdf", "\"etag-1\""));
// The object is overwritten while the run is executing.
headReturns("doc.pdf", "\"etag-2\"");
List<ResolvedInput> work = source.resolve(spec(), ctx);
work.get(0).onComplete().accept(true);
// The delete is version-guarded: the replacement is not the object that ran, so it stays
// and is claimed as fresh work instead of being marked processed.
verify(s3Client, never()).deleteObject(any(DeleteObjectRequest.class));
listingReturns(object("doc.pdf", "\"etag-2\""));
assertEquals(1, source.resolve(spec(), ctx).size());
}
@Test
void aSharedObjectIsRemovedOnlyOnceEveryPolicyHasProcessedIt() throws IOException {
listingReturns(object("doc.pdf", "\"etag-1\""));
headReturns("doc.pdf", "\"etag-1\"");
RecordingContext other = new RecordingContext("p2");
List<ResolvedInput> mine = source.resolve(spec(), ctx);
List<ResolvedInput> theirs = source.resolve(spec(), other);
assertEquals(1, mine.size());
assertEquals(1, theirs.size());
mine.get(0).onComplete().accept(true);
// The other policy's claim is still in flight, so the first finisher must not delete.
verify(s3Client, never()).deleteObject(any(DeleteObjectRequest.class));
theirs.get(0).onComplete().accept(true);
verify(s3Client).deleteObject(any(DeleteObjectRequest.class));
}
@Test
void aFailedObjectStaysAndIsNotRetriedUntilItChanges() throws IOException {
listingReturns(object("doc.pdf", "\"etag-1\""));
source.resolve(spec(), ctx).get(0).onComplete().accept(false);
verify(s3Client, never()).deleteObject(any(DeleteObjectRequest.class));
assertTrue(source.resolve(spec(), ctx).isEmpty());
// A new upload carries a new ETag, which reads as a new version and retries.
listingReturns(object("doc.pdf", "\"etag-2\""));
assertEquals(1, source.resolve(spec(), ctx).size());
}
@Test
void snapshotReadsStatelesslyEverySweep() throws IOException {
listingReturns(object("doc.pdf", "\"etag-1\""));
InputSpec spec = new InputSpec("s3", options(Map.of("mode", "snapshot")));
List<ResolvedInput> first = source.resolve(spec, ctx);
first.get(0).onComplete().accept(true);
List<ResolvedInput> second = source.resolve(spec, ctx);
assertEquals(1, first.size());
assertEquals(1, second.size());
verify(s3Client, never()).deleteObject(any(DeleteObjectRequest.class));
assertTrue(ctx.present.isEmpty());
}
@Test
void folderPlaceholdersAndDotPrefixedKeysAreSkipped() throws IOException {
listingReturns(
object("doc.pdf", "\"etag-1\""),
object("incoming/", "\"etag-2\""),
object(".stirling/tmp/staged.pdf", "\"etag-3\""),
object("incoming/.hidden.pdf", "\"etag-4\""));
List<ResolvedInput> work = source.resolve(spec(), ctx);
assertEquals(1, work.size());
assertEquals(List.of("s3://" + BUCKET + "/doc.pdf"), ctx.present);
}
@Test
void listingPagesAreAllRead() throws IOException {
ListObjectsV2Response firstPage =
ListObjectsV2Response.builder()
.contents(object("a.pdf", "\"etag-a\""))
.nextContinuationToken("next")
.build();
ListObjectsV2Response secondPage =
ListObjectsV2Response.builder().contents(object("b.pdf", "\"etag-b\"")).build();
when(s3Client.listObjectsV2(any(ListObjectsV2Request.class)))
.thenReturn(firstPage, secondPage);
assertEquals(2, source.resolve(spec(), ctx).size());
}
@Test
void aListingFailurePropagatesSoTheSweepVetoesCleanup() {
when(s3Client.listObjectsV2(any(ListObjectsV2Request.class)))
.thenThrow(SdkClientException.create("connection refused"));
assertThrows(SdkClientException.class, () -> source.resolve(spec(), ctx));
}
@Test
void resourceStreamsTheObjectAndNamesItByKeyBasename() throws IOException {
listingReturns(object("incoming/doc.pdf", "\"etag-1\""));
byte[] payload = "data".getBytes(StandardCharsets.UTF_8);
when(s3Client.getObject(any(GetObjectRequest.class)))
.thenReturn(
new ResponseInputStream<>(
GetObjectResponse.builder().build(),
AbortableInputStream.create(new ByteArrayInputStream(payload))));
var resource = source.resolve(spec(), ctx).get(0).inputs().primary().get(0);
assertEquals("doc.pdf", resource.getFilename());
// Content length comes from the listing, not a download.
assertEquals(4, resource.contentLength());
try (var stream = resource.getInputStream()) {
assertEquals("data", new String(stream.readAllBytes(), StandardCharsets.UTF_8));
}
}
@Test
void aMissingETagFallsBackToSizeAndLastModified() throws IOException {
Instant modified = Instant.parse("2026-01-01T00:00:00Z");
listingReturns(S3Object.builder().key("doc.pdf").size(4L).lastModified(modified).build());
assertEquals(1, source.resolve(spec(), ctx).size());
// The same gate on the next sweep reads as already claimed.
listingReturns(S3Object.builder().key("doc.pdf").size(4L).lastModified(modified).build());
assertTrue(source.resolve(spec(), ctx).isEmpty());
}
@Test
void validateRejectsBadConfig() {
// No bucket.
assertThrows(
IllegalArgumentException.class,
() -> source.validate(new InputSpec("s3", Map.of())));
// Credentials are required, never the server's own identity - together and individually.
assertThrows(
IllegalArgumentException.class,
() -> source.validate(new InputSpec("s3", Map.of("bucket", BUCKET))));
assertThrows(
IllegalArgumentException.class,
() ->
source.validate(
new InputSpec(
"s3", Map.of("bucket", BUCKET, "accessKeyId", "AKIA"))));
assertThrows(
IllegalArgumentException.class,
() -> source.validate(new InputSpec("s3", options(Map.of("mode", "sideways")))));
assertThrows(
IllegalArgumentException.class,
() ->
source.validate(
new InputSpec(
"s3", options(Map.of("endpoint", "ftp://example.com")))));
}
@Test
void validateRejectsAnUnreachableBucket() {
when(s3Client.listObjectsV2(any(ListObjectsV2Request.class)))
.thenThrow(SdkClientException.create("connection refused"));
assertThrows(IllegalArgumentException.class, () -> source.validate(spec()));
}
private static InputSpec spec() {
return new InputSpec("s3", options(Map.of()));
}
/** The required options (bucket + credentials) plus any extras under test. */
private static Map<String, Object> options(Map<String, Object> extra) {
Map<String, Object> options = new HashMap<>(extra);
options.put("bucket", BUCKET);
options.put("accessKeyId", "AKIAEXAMPLE");
options.put("secretAccessKey", "shh");
return options;
}
private static S3Object object(String key, String eTag) {
return S3Object.builder()
.key(key)
.eTag(eTag)
.size(4L)
.lastModified(Instant.parse("2026-01-01T00:00:00Z"))
.build();
}
private void listingReturns(S3Object... objects) {
when(s3Client.listObjectsV2(any(ListObjectsV2Request.class)))
.thenReturn(ListObjectsV2Response.builder().contents(objects).build());
}
private void headReturns(String key, String eTag) {
when(s3Client.headObject(any(HeadObjectRequest.class)))
.thenReturn(
HeadObjectResponse.builder()
.eTag(eTag)
.contentLength(4L)
.lastModified(Instant.parse("2026-01-01T00:00:00Z"))
.build());
}
private class RecordingContext implements ResolveContext {
private final String policyId;
private final List<String> present = new ArrayList<>();
private RecordingContext() {
this(POLICY);
}
private RecordingContext(String policyId) {
this.policyId = policyId;
}
@Override
public boolean claim(String identity, String gate, Supplier<String> contentHash) {
return ledger.claim(policyId, identity, gate, contentHash);
}
@Override
public void settle(
String identity, String finalGate, String finalContentHash, boolean success) {
ledger.settle(policyId, identity, finalGate, finalContentHash, success);
}
@Override
public boolean allSettledDone(String identity) {
return ledger.allSettledDone(identity);
}
@Override
public void reportPresent(Collection<String> identities) {
present.addAll(identities);
}
}
}
@@ -0,0 +1,211 @@
package stirling.software.proprietary.policy.output;
import static org.assertj.core.api.Assertions.assertThat;
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.function.Supplier;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.core.io.ByteArrayResource;
import org.springframework.core.io.Resource;
import org.testcontainers.containers.MinIOContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.common.model.job.ResultFile;
import stirling.software.proprietary.policy.input.ResolveContext;
import stirling.software.proprietary.policy.input.ResolvedInput;
import stirling.software.proprietary.policy.input.S3InputSource;
import stirling.software.proprietary.policy.ledger.InProcessProcessedLedger;
import stirling.software.proprietary.policy.model.InputSpec;
import stirling.software.proprietary.policy.model.OutputSpec;
import stirling.software.proprietary.policy.s3.S3ConnectionPool;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.ResponseInputStream;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3Configuration;
import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.GetObjectResponse;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
/**
* End-to-end {@link S3OutputSink} test against a real S3 API (MinIO): uploads, collision renaming,
* and - composed with {@link S3InputSource} - the loop-safety guarantee that a policy writing into
* a bucket it also watches never re-ingests its own outputs, while a second policy still can.
*/
@Testcontainers(disabledWithoutDocker = true)
class S3OutputSinkMinioTest {
private static final String POLICY = "p1";
private static final String ACCESS_KEY = "minioadmin";
private static final String SECRET_KEY = "minioadmin";
@Container
static MinIOContainer minio =
new MinIOContainer("minio/minio:latest")
.withUserName(ACCESS_KEY)
.withPassword(SECRET_KEY);
private static S3Client adminClient;
private static int bucketCounter;
private String bucket;
private S3OutputSink sink;
private S3InputSource source;
private InProcessProcessedLedger ledger;
@BeforeEach
void setUp() {
if (adminClient == null) {
adminClient =
S3Client.builder()
.endpointOverride(URI.create(minio.getS3URL()))
.httpClient(UrlConnectionHttpClient.create())
.region(Region.US_EAST_1)
.credentialsProvider(
StaticCredentialsProvider.create(
AwsBasicCredentials.create(ACCESS_KEY, SECRET_KEY)))
.serviceConfiguration(
S3Configuration.builder().pathStyleAccessEnabled(true).build())
.build();
}
bucket = "policy-outbox-" + ++bucketCounter;
adminClient.createBucket(CreateBucketRequest.builder().bucket(bucket).build());
ApplicationProperties properties = new ApplicationProperties();
properties.getPolicies().setAllowPrivateS3Endpoints(true);
S3ConnectionPool pool = new S3ConnectionPool(properties);
ledger = new InProcessProcessedLedger();
sink = new S3OutputSink(pool, ledger);
source = new S3InputSource(pool);
}
@Test
void uploadsOutputsUnderThePrefix() throws IOException {
List<ResultFile> results =
sink.deliver(
new OutputDelivery("run-1", POLICY),
List.of(output("doc.pdf", "pdf bytes")),
outputSpec("processed/"));
assertThat(results).hasSize(1);
assertThat(results.get(0).getFileName()).isEqualTo("s3://" + bucket + "/processed/doc.pdf");
assertThat(objectContent("processed/doc.pdf")).isEqualTo("pdf bytes");
}
@Test
void anExistingKeyIsNeverOverwritten() throws IOException {
adminClient.putObject(
PutObjectRequest.builder().bucket(bucket).key("doc.pdf").build(),
RequestBody.fromString("theirs", StandardCharsets.UTF_8));
List<ResultFile> results =
sink.deliver(
new OutputDelivery("run-1", POLICY),
List.of(output("doc.pdf", "ours")),
outputSpec(""));
assertThat(results.get(0).getFileName()).isEqualTo("s3://" + bucket + "/doc (1).pdf");
assertThat(objectContent("doc.pdf")).isEqualTo("theirs");
assertThat(objectContent("doc (1).pdf")).isEqualTo("ours");
}
@Test
void aPolicyWritingIntoItsWatchedBucketSkipsItsOwnOutputsButAnotherPolicyChains()
throws IOException {
sink.deliver(
new OutputDelivery("run-1", POLICY),
List.of(output("result.pdf", "produced")),
outputSpec(""));
// The producing policy's sweep sees its own output at the recorded gate and skips it.
assertThat(source.resolve(inputSpec(), new RecordingContext(POLICY))).isEmpty();
// A different policy watching the same bucket has no row and processes it - chaining.
List<ResolvedInput> chained = source.resolve(inputSpec(), new RecordingContext("p2"));
assertThat(chained).hasSize(1);
try (InputStream stream = chained.get(0).inputs().primary().get(0).getInputStream()) {
assertThat(new String(stream.readAllBytes(), StandardCharsets.UTF_8))
.isEqualTo("produced");
}
}
private OutputSpec outputSpec(String prefix) {
return new OutputSpec(
"s3",
Map.of(
"bucket", bucket,
"prefix", prefix,
"endpoint", minio.getS3URL(),
"accessKeyId", ACCESS_KEY,
"secretAccessKey", SECRET_KEY));
}
private InputSpec inputSpec() {
return new InputSpec(
"s3",
Map.of(
"bucket", bucket,
"endpoint", minio.getS3URL(),
"accessKeyId", ACCESS_KEY,
"secretAccessKey", SECRET_KEY));
}
private String objectContent(String key) throws IOException {
try (ResponseInputStream<GetObjectResponse> stream =
adminClient.getObject(GetObjectRequest.builder().bucket(bucket).key(key).build())) {
return new String(stream.readAllBytes(), StandardCharsets.UTF_8);
}
}
private static Resource output(String name, String content) {
return new ByteArrayResource(content.getBytes(StandardCharsets.UTF_8)) {
@Override
public String getFilename() {
return name;
}
};
}
private class RecordingContext implements ResolveContext {
private final String policyId;
private RecordingContext(String policyId) {
this.policyId = policyId;
}
@Override
public boolean claim(String identity, String gate, Supplier<String> contentHash) {
return ledger.claim(policyId, identity, gate, contentHash);
}
@Override
public void settle(
String identity, String finalGate, String finalContentHash, boolean success) {
ledger.settle(policyId, identity, finalGate, finalContentHash, success);
}
@Override
public boolean allSettledDone(String identity) {
return ledger.allSettledDone(identity);
}
@Override
public void reportPresent(Collection<String> identities) {}
}
}
@@ -0,0 +1,267 @@
package stirling.software.proprietary.policy.output;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.when;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.util.ArrayList;
import java.util.HexFormat;
import java.util.List;
import java.util.Map;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.core.io.ByteArrayResource;
import org.springframework.core.io.Resource;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.common.model.job.ResultFile;
import stirling.software.proprietary.policy.ledger.ClaimState;
import stirling.software.proprietary.policy.ledger.InProcessProcessedLedger;
import stirling.software.proprietary.policy.ledger.ProcessedFileStatus;
import stirling.software.proprietary.policy.model.OutputSpec;
import stirling.software.proprietary.policy.s3.S3ConnectionPool;
import software.amazon.awssdk.awscore.exception.AwsServiceException;
import software.amazon.awssdk.core.exception.SdkClientException;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectResponse;
import software.amazon.awssdk.services.s3.model.S3Exception;
/**
* Tests for {@link S3OutputSink}: the ledger row exists before the object is visible, collisions
* re-pick names, ad-hoc runs record nothing, and encrypted-bucket ETags are re-recorded.
*/
@ExtendWith(MockitoExtension.class)
class S3OutputSinkTest {
private static final String POLICY = "p1";
private static final String BUCKET = "outbox-bucket";
private static final OutputDelivery DELIVERY = new OutputDelivery("run-1", POLICY);
private static final OutputDelivery AD_HOC = new OutputDelivery("run-2", null);
@Mock private S3Client s3Client;
private S3OutputSink sink;
private InProcessProcessedLedger ledger;
private final List<PutObjectRequest> puts = new ArrayList<>();
@BeforeEach
void setUp() {
ledger = new InProcessProcessedLedger();
sink =
new S3OutputSink(
new S3ConnectionPool(new ApplicationProperties(), config -> s3Client),
ledger);
}
@Test
void recordsTheRowBeforeTheObjectBecomesVisible() throws IOException {
// The row for the exact key must already be settled DONE at the moment the PUT runs -
// record-before-visible, asserted from inside the upload itself.
List<ClaimState> stateAtPutTime = new ArrayList<>();
when(s3Client.putObject(any(PutObjectRequest.class), any(RequestBody.class)))
.thenAnswer(
invocation -> {
PutObjectRequest request = invocation.getArgument(0);
puts.add(request);
stateAtPutTime.add(stateFor(identity(request.key())));
return PutObjectResponse.builder().eTag(quotedMd5("data")).build();
});
List<ResultFile> results =
sink.deliver(DELIVERY, List.of(output("doc.pdf", "data")), spec());
assertEquals(1, results.size());
assertEquals("s3://" + BUCKET + "/processed/doc.pdf", results.get(0).getFileName());
assertEquals(4, results.get(0).getFileSize());
assertNotNull(stateAtPutTime.get(0));
assertEquals(ProcessedFileStatus.DONE, stateAtPutTime.get(0).status());
assertEquals(md5("data"), stateAtPutTime.get(0).gate());
assertTrue(puts.get(0).ifNoneMatch() != null);
}
@Test
void aTakenKeyIsForgottenAndRePicked() throws IOException {
when(s3Client.putObject(any(PutObjectRequest.class), any(RequestBody.class)))
.thenAnswer(
invocation -> {
PutObjectRequest request = invocation.getArgument(0);
puts.add(request);
if (puts.size() == 1) {
throw s3Error(412, "PreconditionFailed");
}
return PutObjectResponse.builder().eTag(quotedMd5("data")).build();
});
List<ResultFile> results =
sink.deliver(DELIVERY, List.of(output("doc.pdf", "data")), spec());
assertEquals("s3://" + BUCKET + "/processed/doc (1).pdf", results.get(0).getFileName());
// The lost candidate's row is gone; only the delivered key is recorded.
assertNull(stateFor(identity("processed/doc.pdf")));
assertNotNull(stateFor(identity("processed/doc (1).pdf")));
}
@Test
void anEncryptedBucketETagIsReRecordedAtTheActualGate() throws IOException {
when(s3Client.putObject(any(PutObjectRequest.class), any(RequestBody.class)))
.thenReturn(PutObjectResponse.builder().eTag("\"kms-opaque-etag\"").build());
sink.deliver(DELIVERY, List.of(output("doc.pdf", "data")), spec());
assertEquals("kms-opaque-etag", stateFor(identity("processed/doc.pdf")).gate());
}
@Test
void anAdHocDeliveryRecordsNothing() throws IOException {
when(s3Client.putObject(any(PutObjectRequest.class), any(RequestBody.class)))
.thenReturn(PutObjectResponse.builder().eTag(quotedMd5("data")).build());
sink.deliver(AD_HOC, List.of(output("doc.pdf", "data")), spec());
assertNull(stateFor(identity("processed/doc.pdf")));
}
@Test
void aFailedUploadForgetsItsRowAndThrows() {
when(s3Client.putObject(any(PutObjectRequest.class), any(RequestBody.class)))
.thenThrow(SdkClientException.create("connection refused"));
assertThrows(
IOException.class,
() -> sink.deliver(DELIVERY, List.of(output("doc.pdf", "data")), spec()));
assertNull(stateFor(identity("processed/doc.pdf")));
}
@Test
void aStoreWithoutConditionalPutsFallsBackToExistenceChecks() throws IOException {
when(s3Client.headObject(any(HeadObjectRequest.class))).thenThrow(s3Error(404, "NotFound"));
when(s3Client.putObject(any(PutObjectRequest.class), any(RequestBody.class)))
.thenAnswer(
invocation -> {
PutObjectRequest request = invocation.getArgument(0);
puts.add(request);
if (request.ifNoneMatch() != null) {
throw s3Error(501, "NotImplemented");
}
return PutObjectResponse.builder().eTag(quotedMd5("data")).build();
});
List<ResultFile> results =
sink.deliver(DELIVERY, List.of(output("doc.pdf", "data")), spec());
// Same key, second attempt unconditional.
assertEquals("s3://" + BUCKET + "/processed/doc.pdf", results.get(0).getFileName());
assertEquals(2, puts.size());
assertNull(puts.get(1).ifNoneMatch());
assertNotNull(stateFor(identity("processed/doc.pdf")));
}
@Test
void aBarePrefixGetsItsSlash() throws IOException {
when(s3Client.putObject(any(PutObjectRequest.class), any(RequestBody.class)))
.thenAnswer(
invocation -> {
puts.add(invocation.getArgument(0));
return PutObjectResponse.builder().eTag(quotedMd5("data")).build();
});
sink.deliver(DELIVERY, List.of(output("doc.pdf", "data")), spec("processed"));
assertEquals("processed/doc.pdf", puts.get(0).key());
}
@Test
void validateRejectsBadConfigShape() {
assertThrows(
IllegalArgumentException.class,
() -> sink.validate(new OutputSpec("s3", Map.of())));
// Credentials are required, never the server's own identity.
assertThrows(
IllegalArgumentException.class,
() -> sink.validate(new OutputSpec("s3", Map.of("bucket", BUCKET))));
assertThrows(
IllegalArgumentException.class,
() ->
sink.validate(
new OutputSpec(
"s3", Map.of("bucket", BUCKET, "accessKeyId", "AKIA"))));
}
@Test
void supportsOnlyS3Specs() {
assertTrue(sink.supports(spec()));
assertFalse(sink.supports(OutputSpec.inline()));
assertFalse(sink.supports(null));
}
private static OutputSpec spec() {
return spec("processed/");
}
private static OutputSpec spec(String prefix) {
return new OutputSpec(
"s3",
Map.of(
"bucket",
BUCKET,
"prefix",
prefix,
"accessKeyId",
"AKIAEXAMPLE",
"secretAccessKey",
"shh"));
}
private static String identity(String key) {
return "s3://" + BUCKET + "/" + key;
}
private ClaimState stateFor(String identity) {
return ledger.statesFor(POLICY, List.of(identity)).get(identity);
}
private static Resource output(String name, String content) {
return new ByteArrayResource(content.getBytes(StandardCharsets.UTF_8)) {
@Override
public String getFilename() {
return name;
}
};
}
private static String md5(String content) {
try {
return HexFormat.of()
.formatHex(
MessageDigest.getInstance("MD5")
.digest(content.getBytes(StandardCharsets.UTF_8)));
} catch (Exception e) {
throw new IllegalStateException(e);
}
}
private static String quotedMd5(String content) {
return "\"" + md5(content) + "\"";
}
private static AwsServiceException s3Error(int status, String code) {
return S3Exception.builder().statusCode(status).message(code).build();
}
}
@@ -28,6 +28,7 @@ import stirling.software.proprietary.policy.model.Policy;
import stirling.software.proprietary.policy.store.InProcessPolicyStore;
import stirling.software.proprietary.policy.store.PolicyStore;
import stirling.software.proprietary.policy.trigger.PolicyTriggerManager;
import stirling.software.proprietary.util.SecretMasker;
/**
* Tests for {@link SourceController}'s delete guard: a source still referenced by a policy is
@@ -109,6 +110,107 @@ class SourceControllerTest {
assertEquals(404, controller.delete("nope").getStatusCode().value());
}
@Test
void documentCountsForTheEditorReturnsTheTeamSeries() {
ResponseEntity<List<Long>> response = controller.documentCounts(EditorSource.ID);
assertEquals(200, response.getStatusCode().value());
assertEquals(30, response.getBody().size());
}
@Test
void theEditorIsBuiltInAndCannotBeDeleted() {
ResponseStatusException ex =
assertThrows(
ResponseStatusException.class, () -> controller.delete(EditorSource.ID));
assertEquals(400, ex.getStatusCode().value());
}
@Test
void theEditorIsBuiltInAndCannotBeSaved() {
Source editor = new Source(null, "Editor", "editor", Map.of(), true, null, null);
ResponseStatusException ex =
assertThrows(ResponseStatusException.class, () -> controller.save(editor));
assertEquals(400, ex.getStatusCode().value());
}
@Test
void readsReturnSecretsAsTheRedactionSentinel() {
Source saved = sourceStore.save(s3Source("shh"));
Source read = controller.get(saved.id()).getBody();
assertEquals(SecretMasker.REDACTED, read.options().get("secretAccessKey"));
assertEquals("AKIAEXAMPLE", read.options().get("accessKeyId"));
// The store itself keeps the real value.
assertEquals(
"shh", sourceStore.get(saved.id()).orElseThrow().options().get("secretAccessKey"));
}
@Test
void savingTheSentinelBackKeepsTheStoredSecret() {
Source saved = sourceStore.save(s3Source("shh"));
Source edited =
new Source(
saved.id(),
"Renamed",
saved.type(),
Map.of(
"bucket", "inbox",
"accessKeyId", "AKIAEXAMPLE",
"secretAccessKey", SecretMasker.REDACTED),
true,
saved.owner(),
saved.teamId());
Source response = controller.save(edited).getBody();
assertEquals(
"shh", sourceStore.get(saved.id()).orElseThrow().options().get("secretAccessKey"));
// The save response is masked too; only the store sees the real value.
assertEquals(SecretMasker.REDACTED, response.options().get("secretAccessKey"));
}
@Test
void savingANewSecretReplacesTheStoredOne() {
Source saved = sourceStore.save(s3Source("old-secret"));
Source edited =
new Source(
saved.id(),
saved.name(),
saved.type(),
Map.of(
"bucket", "inbox",
"accessKeyId", "AKIAEXAMPLE",
"secretAccessKey", "new-secret"),
true,
saved.owner(),
saved.teamId());
controller.save(edited);
assertEquals(
"new-secret",
sourceStore.get(saved.id()).orElseThrow().options().get("secretAccessKey"));
}
private static Source s3Source(String secret) {
return new Source(
null,
"Bucket intake",
"s3",
Map.of(
"bucket", "inbox",
"accessKeyId", "AKIAEXAMPLE",
"secretAccessKey", secret),
true,
"owner",
null);
}
private static Source folderSource() {
return new Source(
null, "Claims intake", "folder", Map.of("directory", "/in"), true, "owner", null);
@@ -56,9 +56,10 @@ class SourceOverviewServiceTest {
SourcesResponse response = service.overview();
assertEquals(3, response.sources().size());
// Sorted most-referenced first, so the shared source A leads.
assertEquals(a.id(), response.sources().get(0).id());
assertEquals(4, response.sources().size());
// The built-in editor is pinned first; persisted sources follow, most-referenced leading.
assertEquals(EditorSource.ID, response.sources().get(0).id());
assertEquals(a.id(), response.sources().get(1).id());
SourceView av = find(response, a.id());
assertEquals(2, av.referenceCount());
@@ -123,13 +124,57 @@ class SourceOverviewServiceTest {
SourcesResponse response = scoped.overview();
assertEquals(1, response.sources().size());
SourceView view = response.sources().get(0);
assertEquals(ours.id(), view.id());
assertEquals(2, response.sources().size());
assertEquals(EditorSource.ID, response.sources().get(0).id());
SourceView view = find(response, ours.id());
assertEquals(1, view.referenceCount());
assertEquals(List.of(1L, 1L, 0L), response.kpis().stream().map(SourceKpi::value).toList());
}
@Test
void theEditorSourceIsAlwaysPresentEvenWithNoConnections() {
SourcesResponse response = service.overview();
assertEquals(1, response.sources().size());
SourceView editor = response.sources().get(0);
assertEquals(EditorSource.ID, editor.id());
assertEquals("editor", editor.type());
assertEquals("active", editor.status());
assertEquals(0, editor.referenceCount());
// KPIs describe configured connections, so the built-in editor is left out of them.
assertEquals(List.of(0L, 0L, 0L), response.kpis().stream().map(SourceKpi::value).toList());
}
@Test
void theEditorSourceIsUsedByEveryPolicyThatRunsFromIt() {
editorPolicy("Redact on upload");
editorPolicy("Classify on upload");
// A folder-sourced policy does not target the editor, so it must not inflate the count.
policyReferencing("Folder sweep", source("Folder", "/f").id());
SourceView editor = find(service.overview(), EditorSource.ID);
assertEquals(2, editor.referenceCount());
assertTrue(
editor.referencingPolicies().stream()
.map(SourceView.PolicyRef::name)
.toList()
.containsAll(List.of("Redact on upload", "Classify on upload")));
}
@Test
void theEditorSourceReportsTheTeamsRecordedDocumentThroughput() {
// Login disabled, so the team is null and the editor shares the global counter bucket.
docCounter.record(EditorSource.counterKey(null), 4);
docCounter.record(EditorSource.counterKey(null), 6);
SourceView editor = find(service.overview(), EditorSource.ID);
assertEquals(10, editor.docsTotal());
assertEquals(10, editor.docs24h());
assertEquals(10, editor.docs30d());
}
@Test
void documentCountsReflectRecordedDocs() {
Source a = source("A", "/a");
@@ -177,6 +222,22 @@ class SourceOverviewServiceTest {
OutputSpec.inline()));
}
/**
* A policy that targets the editor: membership rides in its output metadata, not a sourceId.
*/
private void editorPolicy(String name) {
policyStore.save(
new Policy(
null,
name,
"owner",
true,
null,
List.of(),
List.of(new PipelineStep("/api/v1/misc/compress-pdf", Map.of())),
new OutputSpec("inline", Map.of("sources", List.of("editor")))));
}
private void teamPolicy(String name, Long teamId, String... sourceIds) {
policyStore.save(
new Policy(
@@ -70,6 +70,41 @@ class ScheduleTriggerTest {
verify(policyRunner, times(1)).run(eq(policy));
}
@Test
void anIntervalMatchingTheSweepPeriodFiresEverySweepDespiteJitter() {
Policy policy = scheduled("p1", new Schedule.Every(1, Schedule.Unit.MINUTES));
when(policyStore.findByTriggerType("schedule")).thenReturn(List.of(policy));
Instant t0 = Instant.parse("2026-06-05T10:00:00Z");
trigger.sweep(t0); // baseline
// The sweep that fires runs a few ms late (scheduler jitter)...
trigger.sweep(t0.plusSeconds(60).plusMillis(5));
verify(policyRunner, times(1)).run(eq(policy));
// ...and the next sweep lands exactly on the 60s grid. Anchoring lastFired to the due
// time (not the jittered observation) means this must still fire, not alias to skip.
trigger.sweep(t0.plusSeconds(120));
verify(policyRunner, times(2)).run(eq(policy));
}
@Test
void aGapFiresOnceNotOncePerMissedInterval() {
Policy policy = scheduled("p1", new Schedule.Every(1, Schedule.Unit.MINUTES));
when(policyStore.findByTriggerType("schedule")).thenReturn(List.of(policy));
Instant t0 = Instant.parse("2026-06-05T10:00:00Z");
trigger.sweep(t0); // baseline
// Ten minutes of downtime: nine missed due points collapse into one firing.
trigger.sweep(t0.plusSeconds(600));
verify(policyRunner, times(1)).run(eq(policy));
// Not due again until a full interval after the latest due point.
trigger.sweep(t0.plusSeconds(630));
verify(policyRunner, times(1)).run(eq(policy));
trigger.sweep(t0.plusSeconds(660));
verify(policyRunner, times(2)).run(eq(policy));
}
@Test
void doesNotFireBeforeTheNextScheduledTime() {
Policy policy = scheduled("p1", new Schedule.Daily(LocalTime.of(3, 0))); // 03:00 UTC daily
@@ -14,8 +14,8 @@ import org.junit.jupiter.api.Test;
* Unit tests for {@link SecretMasker}.
*
* <p>Assumptions: - Key matching is case-insensitive via the pattern in SENSITIVE. - If the key
* matches a sensitive pattern, the value is replaced with "***REDACTED***". - Nested maps and lists
* are searched recursively. - Null maps and null values are ignored or returned as null. -
* matches a sensitive pattern, the value is replaced with SecretMasker.REDACTED. - Nested maps and
* lists are searched recursively. - Null maps and null values are ignored or returned as null. -
* Non-sensitive keys/values remain unchanged.
*/
class SecretMaskerTest {
@@ -40,7 +40,7 @@ class SecretMaskerTest {
Map<String, Object> result = SecretMasker.mask(input);
assertEquals("***REDACTED***", result.get("password"));
assertEquals(SecretMasker.REDACTED, result.get("password"));
assertEquals("john", result.get("username"));
}
@@ -55,11 +55,54 @@ class SecretMaskerTest {
Map<String, Object> result = SecretMasker.mask(input);
assertEquals("***REDACTED***", result.get("Api-Key"));
assertEquals("***REDACTED***", result.get("TOKEN"));
assertEquals(SecretMasker.REDACTED, result.get("Api-Key"));
assertEquals(SecretMasker.REDACTED, result.get("TOKEN"));
assertEquals("keepme", result.get("normal"));
}
@Test
@DisplayName("restoreRedacted swaps sentinels for stored values, leaves the rest")
void restoreRedactedRoundTripsAnEdit() {
Map<String, Object> stored =
Map.of("secretAccessKey", "shh", "accessKeyId", "AKIAEXAMPLE");
Map<String, Object> incoming =
Map.of(
"secretAccessKey", SecretMasker.REDACTED,
"accessKeyId", "AKIA-NEW",
"bucket", "inbox");
Map<String, Object> merged = SecretMasker.restoreRedacted(incoming, stored);
assertEquals("shh", merged.get("secretAccessKey"));
assertEquals("AKIA-NEW", merged.get("accessKeyId"));
assertEquals("inbox", merged.get("bucket"));
}
@Test
@DisplayName("restoreRedacted leaves a sentinel with no stored counterpart in place")
void restoreRedactedWithoutStoredValueStaysSentinel() {
Map<String, Object> merged =
SecretMasker.restoreRedacted(
Map.of("secretAccessKey", SecretMasker.REDACTED), Map.of());
assertEquals(SecretMasker.REDACTED, merged.get("secretAccessKey"));
}
@Test
@DisplayName("should mask camelCase secretAccessKey despite no word boundary")
void shouldMaskCamelCaseSecretAccessKey() {
Map<String, Object> input =
Map.of(
"secretAccessKey", "shh",
"accessKeyId", "AKIAEXAMPLE");
Map<String, Object> result = SecretMasker.mask(input);
assertEquals(SecretMasker.REDACTED, result.get("secretAccessKey"));
// Access key ids are username-like, not secrets.
assertEquals("AKIAEXAMPLE", result.get("accessKeyId"));
}
@Test
@DisplayName("should mask nested map sensitive keys")
void shouldMaskNestedMapSensitiveKeys() {
@@ -77,9 +120,9 @@ class SecretMaskerTest {
Map<String, Object> result = SecretMasker.mask(input);
Map<String, Object> outer = (Map<String, Object>) result.get("outer");
assertEquals("***REDACTED***", outer.get("jwt"));
assertEquals(SecretMasker.REDACTED, outer.get("jwt"));
Map<String, Object> inner = (Map<String, Object>) outer.get("inner");
assertEquals("***REDACTED***", inner.get("secret"));
assertEquals(SecretMasker.REDACTED, inner.get("secret"));
assertEquals("ok", inner.get("other"));
}
@@ -98,7 +141,7 @@ class SecretMaskerTest {
List<?> list = (List<?>) result.get("list");
Map<String, Object> first = (Map<String, Object>) list.get(0);
assertEquals("***REDACTED***", first.get("token"));
assertEquals(SecretMasker.REDACTED, first.get("token"));
Map<String, Object> second = (Map<String, Object>) list.get(1);
assertEquals("john", second.get("username"));
assertEquals("stringValue", list.get(2));
@@ -170,7 +213,8 @@ class SecretMaskerTest {
Map<String, Object> outer = (Map<String, Object>) result.get("outer");
assertTrue(outer.containsKey(null), "Null key should be preserved");
assertEquals("plainText", outer.get(null), "Value for null key must not be masked");
assertEquals("***REDACTED***", outer.get("password"), "Sensitive keys must be masked");
assertEquals(
SecretMasker.REDACTED, outer.get("password"), "Sensitive keys must be masked");
}
}
}
@@ -117,7 +117,8 @@ public class AiCreateController {
user.getTeam().getId(),
source,
ProcessType.SINGLE_TOOL,
BillingCategory.AI);
BillingCategory.AI,
null);
jobChargeService.chargeStandalone(ctx, 1);
} catch (RuntimeException e) {
log.warn(
@@ -173,7 +173,9 @@ public class PaygWalletController {
int spend = clampToInt(snap.periodSpendUnits());
Integer limit = snap.periodCapUnits() != null ? clampToInt(snap.periodCapUnits()) : null;
CategoryBreakdown breakdown = buildBreakdown(teamId, snap.periodStart(), snap.periodEnd());
BreakdownPair breakdowns = buildBreakdowns(teamId, snap.periodStart(), snap.periodEnd());
UsageAnalytics analytics =
buildUsageAnalytics(teamId, snap.periodStart(), snap.periodEnd());
// Estimated bill = paid (Stripe-metered) docs this period × rate the free portion was
// already netted out at charge time, so this is the metered total, not spend grant.
@@ -203,30 +205,63 @@ public class PaygWalletController {
noCap,
billing.subscriptionId(),
spend,
breakdown,
breakdowns.units(),
members,
buildActivity(teamId));
buildActivity(teamId),
breakdowns.docs(),
analytics.docsProcessed(),
analytics.uniquePdfs(),
analytics.sizeMultiplierPdfs());
return ResponseEntity.ok(body);
}
private CategoryBreakdown buildBreakdown(
/** Per-category size-scaled units + input-file counts for the same window. */
private record BreakdownPair(CategoryBreakdown units, CategoryBreakdown docs) {}
/** Period usage analytics: total input files, unique PDFs, and size-multiplier files. */
private record UsageAnalytics(int docsProcessed, int uniquePdfs, int sizeMultiplierPdfs) {}
private BreakdownPair buildBreakdowns(
Long teamId, LocalDateTime periodStart, LocalDateTime periodEnd) {
Map<BillingCategory, Long> byCategory = new HashMap<>();
Map<BillingCategory, Long> units = new HashMap<>();
Map<BillingCategory, Long> docs = new HashMap<>();
for (Object[] row :
ledgerRepo.sumPeriodAmountByCategory(
ledgerRepo.sumPeriodByCategoryWithDocs(
teamId, LedgerEntryType.DEBIT, periodStart, periodEnd)) {
if (row.length >= 2
&& row[0] instanceof BillingCategory cat
&& row[1] instanceof Number n) {
byCategory.put(cat, n.longValue());
if (row.length >= 3 && row[0] instanceof BillingCategory cat) {
if (row[1] instanceof Number u) {
units.put(cat, u.longValue());
}
if (row[2] instanceof Number d) {
docs.put(cat, d.longValue());
}
}
}
return new BreakdownPair(categoryBreakdown(units), categoryBreakdown(docs));
}
private static CategoryBreakdown categoryBreakdown(Map<BillingCategory, Long> byCategory) {
return new CategoryBreakdown(
clampToInt(byCategory.getOrDefault(BillingCategory.API, 0L)),
clampToInt(byCategory.getOrDefault(BillingCategory.AI, 0L)),
clampToInt(byCategory.getOrDefault(BillingCategory.AUTOMATION, 0L)));
}
private UsageAnalytics buildUsageAnalytics(
Long teamId, LocalDateTime periodStart, LocalDateTime periodEnd) {
List<Object[]> rows =
ledgerRepo.periodUsageAnalytics(
teamId, LedgerEntryType.DEBIT, periodStart, periodEnd);
Object[] row = rows.isEmpty() ? null : rows.get(0);
return new UsageAnalytics(analyticsInt(row, 0), analyticsInt(row, 1), analyticsInt(row, 2));
}
private static int analyticsInt(Object[] row, int idx) {
return row != null && row.length > idx && row[idx] instanceof Number n
? clampToInt(n.longValue())
: 0;
}
/**
* Latest ledger entries shaped for the FE activity feed. DEBITs read as usage, REFUNDs as
* credits-back; system entries without a category render as {@code other}.
@@ -446,6 +481,10 @@ public class PaygWalletController {
0,
new CategoryBreakdown(0, 0, 0),
List.of(),
Collections.emptyList());
Collections.emptyList(),
new CategoryBreakdown(0, 0, 0),
0,
0,
0);
}
}
@@ -75,7 +75,21 @@ public record WalletSnapshotResponse(
int spendUnitsThisPeriod,
CategoryBreakdown categoryBreakdown,
List<MemberRow> members,
List<ActivityRow> recent) {
List<ActivityRow> recent,
CategoryBreakdown categoryDocs,
int docsProcessedThisPeriod,
int uniquePdfsThisPeriod,
int sizeMultiplierPdfsThisPeriod) {
// The count dimension, kept distinct from units (which now scale with file size):
// categoryDocs per-category INPUT-file counts (parallel to
// categoryBreakdown,
// which stays the size-scaled unit totals)
// docsProcessedThisPeriod total input files processed this period (Σ doc_count)
// uniquePdfsThisPeriod distinct input documents (a file hit by N operations counts
// once)
// sizeMultiplierPdfsThisPeriod input files on charges where the size multiplier applied
// (units billed > input files)
/** Per-category breakdown of {@code spendUnitsThisPeriod} for the in-app analytics widget. */
public record CategoryBreakdown(int api, int ai, int automation) {}
@@ -14,13 +14,19 @@ import stirling.software.saas.payg.model.ProcessType;
* the interceptor before this context is built. Manual UI tools never reach {@code openProcess}
* (they short-circuit on {@link BillingCategory#BYPASSED}); any context constructed here therefore
* carries one of {@code API}, {@code AI}, or {@code AUTOMATION}.
*
* <p>{@code runId} is the automation-run correlation id ({@code X-Stirling-Run-Id}) when this call
* is a sub-step of a pipeline / policy / AI-workflow run, else {@code null} (a standalone tool
* call). Lineage joins are scoped to a single run id: a null run id never joins (each standalone
* call is its own charge), and two separate runs never merge even on identical bytes.
*/
public record ChargeContext(
Long ownerUserId,
Long ownerTeamId,
JobSource source,
ProcessType processType,
BillingCategory billingCategory) {
BillingCategory billingCategory,
String runId) {
public ChargeContext {
if (ownerUserId == null) {
@@ -36,4 +42,17 @@ public record ChargeContext(
throw new IllegalArgumentException("billingCategory is required");
}
}
/**
* Convenience for callers with no automation-run context a standalone tool call ({@code
* runId} = {@code null}, so it never lineage-joins and is always its own charge).
*/
public ChargeContext(
Long ownerUserId,
Long ownerTeamId,
JobSource source,
ProcessType processType,
BillingCategory billingCategory) {
this(ownerUserId, ownerTeamId, source, processType, billingCategory, null);
}
}
@@ -113,7 +113,8 @@ public class JobChargeService {
ctx.source(),
ctx.processType(),
policy.getId(),
stepLimit);
stepLimit,
ctx.runId());
List<Path> paths = inputs.stream().map(JobInput::path).toList();
JoinOrOpenResult result = jobService.joinOrOpen(jobCtx, paths);
@@ -122,14 +123,23 @@ public class JobChargeService {
return new ChargeOutcome(result.job().getId(), 0, ChargeOutcome.Disposition.JOINED);
}
ProcessingJob job = result.job();
int units = computeUnits(inputs, policy);
result.job().setDocUnits(units);
job.setDocUnits(units);
int freeUsed = consumeFreeGrant(ctx, units);
recordShadowRow(ctx, result.job().getId(), policy.getId(), units, freeUsed);
recordLedgerDebit(ctx, result.job().getId(), policy.getId(), units);
recordShadowRow(ctx, job.getId(), policy.getId(), units, freeUsed);
// doc_count + fingerprint were set on the fresh job by JobService.openFresh; carry them
// onto the ledger DEBIT so usage analytics query one table.
recordLedgerDebit(
ctx,
job.getId(),
policy.getId(),
units,
job.getDocCount(),
job.getDocumentFingerprint());
return new ChargeOutcome(result.job().getId(), units, ChargeOutcome.Disposition.OPENED);
return new ChargeOutcome(job.getId(), units, ChargeOutcome.Disposition.OPENED);
}
/**
@@ -164,12 +174,19 @@ public class JobChargeService {
ctx.source(),
ctx.processType(),
policy.getId(),
stepLimit);
stepLimit,
ctx.runId());
ProcessingJob job = jobService.open(jobCtx, chargeUnits);
int freeUsed = consumeFreeGrant(ctx, chargeUnits);
recordShadowRow(ctx, job.getId(), policy.getId(), chargeUnits, freeUsed);
recordLedgerDebit(ctx, job.getId(), policy.getId(), chargeUnits);
recordLedgerDebit(
ctx,
job.getId(),
policy.getId(),
chargeUnits,
job.getDocCount(),
job.getDocumentFingerprint());
// Close immediately nothing will lineage-join a standalone job so the paid portion
// meters via the same afterCommit hook + idempotency key as a normal process completion.
@@ -216,7 +233,12 @@ public class JobChargeService {
* Skipped for {@code BYPASSED} / uncategorised calls manual UI work is never billed.
*/
private void recordLedgerDebit(
ChargeContext ctx, java.util.UUID jobId, Long policyId, int units) {
ChargeContext ctx,
java.util.UUID jobId,
Long policyId,
int units,
int docCount,
String documentFingerprint) {
BillingCategory category = ctx.billingCategory();
if (category == null || category == BillingCategory.BYPASSED) {
return;
@@ -231,6 +253,10 @@ public class JobChargeService {
entry.setReferenceId(jobId.toString());
entry.setPolicyId(policyId);
entry.setBillingCategory(category);
// Count dimension + input fingerprint, denormalised from the job for usage analytics
// (PDFs processed, unique PDFs, size-multiplier average).
entry.setDocCount(docCount);
entry.setDocumentFingerprint(documentFingerprint);
ledgerRepository.save(entry);
}
@@ -33,6 +33,7 @@ import jakarta.servlet.http.HttpServletResponse;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.annotations.AutoJobPostMapping;
import stirling.software.common.service.AutomationRunContext;
import stirling.software.common.util.TempFile;
import stirling.software.common.util.TempFileManager;
import stirling.software.proprietary.security.database.repository.UserRepository;
@@ -284,13 +285,25 @@ public class PaygChargeInterceptor implements AsyncHandlerInterceptor {
request.setAttribute(ATTR_INPUT_BYTES, totalInputBytes);
request.setAttribute(ATTR_TOOL_ID, resolveToolId(request));
// Automation-run correlation id, honoured ONLY from an internal automation dispatch.
// InternalApiClient stamps X-Stirling-Automation on every loopback sub-step alongside the
// run id, so a genuine pipeline / policy / AI run always carries both. A raw external
// request that sets X-Stirling-Run-Id on its own is ignored (each such call stays its own
// charge): otherwise an API caller could pin a constant run id to collapse separate
// same-content calls into one charge, defeating "charge per API call". Null standalone.
String headerRunId = request.getHeader(AutomationRunContext.RUN_ID_HEADER);
String runId =
(hasAutomationHeader(request) && headerRunId != null && !headerRunId.isBlank())
? headerRunId
: null;
ChargeContext ctx =
new ChargeContext(
currentUser.getId(),
currentUser.getTeam() == null ? null : currentUser.getTeam().getId(),
determineSource(request, auth),
ProcessType.SINGLE_TOOL,
category);
category,
runId);
ChargeOutcome outcome;
try {
@@ -498,9 +511,20 @@ public class PaygChargeInterceptor implements AsyncHandlerInterceptor {
}
}
/**
* True when the request carries the internal-dispatch marker InternalApiClient stamps on every
* loopback sub-step ({@code X-Stirling-Automation: true}). This is the trust boundary for both
* the AUTOMATION billing category and for honouring {@code X-Stirling-Run-Id}: an external
* caller can't group charges via a run id without also declaring itself automation (which
* changes its own billing category).
*/
private static boolean hasAutomationHeader(HttpServletRequest request) {
String header = request.getHeader(AUTOMATION_HEADER);
return header != null && "true".equalsIgnoreCase(header.trim());
}
private static JobSource determineSource(HttpServletRequest request, Authentication auth) {
String automationHeader = request.getHeader(AUTOMATION_HEADER);
if (automationHeader != null && "true".equalsIgnoreCase(automationHeader.trim())) {
if (hasAutomationHeader(request)) {
return JobSource.PIPELINE;
}
String desktopHeader = request.getHeader(DESKTOP_CLIENT_HEADER);
@@ -527,8 +551,7 @@ public class PaygChargeInterceptor implements AsyncHandlerInterceptor {
*/
private static BillingCategory determineCategory(
HandlerMethod handler, HttpServletRequest request, Authentication auth) {
String automationHeader = request.getHeader(AUTOMATION_HEADER);
if (automationHeader != null && "true".equalsIgnoreCase(automationHeader.trim())) {
if (hasAutomationHeader(request)) {
return BillingCategory.AUTOMATION;
}
RequiresFeature ann =
@@ -116,7 +116,8 @@ public class InstanceUsageIngestService {
teamId,
JobSource.LINKED_INSTANCE,
ProcessType.SINGLE_TOOL,
category),
category,
null),
units);
}
if (row == null) {
@@ -15,7 +15,8 @@ public record JobContext(
JobSource source,
ProcessType processType,
Long policyId,
int stepLimit) {
int stepLimit,
String runId) {
public JobContext {
if (ownerUserId == null) {
@@ -34,4 +35,19 @@ public record JobContext(
throw new IllegalArgumentException("stepLimit must be > 0");
}
}
/**
* Convenience for callers with no automation-run context a standalone tool call ({@code
* runId} = {@code null}, so {@code joinOrOpen} always opens a fresh process rather than
* lineage-joining).
*/
public JobContext(
Long ownerUserId,
Long ownerTeamId,
JobSource source,
ProcessType processType,
Long policyId,
int stepLimit) {
this(ownerUserId, ownerTeamId, source, processType, policyId, stepLimit, null);
}
}
@@ -1,12 +1,16 @@
package stirling.software.saas.payg.job;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Path;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.time.Duration;
import java.time.LocalDateTime;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.HashMap;
import java.util.HexFormat;
import java.util.List;
import java.util.Map;
import java.util.Objects;
@@ -97,8 +101,14 @@ public class JobService {
signaturesByInput.put(input, detector.extractSignatures(input));
}
// Lineage joins are scoped to one automation run: a standalone call (no run id) never
// joins each is its own charge and within a run, matching still runs by content so a
// run's separate input files each open their own charge (a merge of N inputs = N charges),
// while a single file's chain of steps + its split outputs collapse into one.
Optional<LineageMatch> bestMatch =
findBestMatch(ctx.ownerUserId(), inputs, signaturesByInput);
ctx.runId() == null
? Optional.empty()
: findBestMatch(ctx.ownerUserId(), ctx.runId(), inputs, signaturesByInput);
if (bestMatch.isPresent()) {
ProcessingJob existing =
@@ -212,10 +222,13 @@ public class JobService {
}
private Optional<LineageMatch> findBestMatch(
Long userId, List<Path> inputs, Map<Path, Set<LineageSignature>> signaturesByInput) {
Long userId,
String runId,
List<Path> inputs,
Map<Path, Set<LineageSignature>> signaturesByInput) {
List<LineageMatch> matches = new ArrayList<>(inputs.size());
for (Path input : inputs) {
detector.detect(userId, signaturesByInput.get(input)).ifPresent(matches::add);
detector.detect(userId, runId, signaturesByInput.get(input)).ifPresent(matches::add);
}
return matches.stream().max(Comparator.comparing(LineageMatch::jobLastStepAt));
}
@@ -252,6 +265,11 @@ public class JobService {
fresh.setProcessType(ctx.processType());
fresh.setSource(ctx.source());
fresh.setPolicyId(ctx.policyId());
fresh.setRunId(ctx.runId());
// doc_count = number of input files (the count dimension). A merge (N inputs in one call)
// is N; a split (1 input) is 1; a standalone bookkeeping job (no inputs) is 1.
fresh.setDocCount(Math.max(1, signaturesByInput.size()));
fresh.setDocumentFingerprint(computeFingerprint(signaturesByInput));
fresh.setStepCount(1);
LocalDateTime now = LocalDateTime.now();
fresh.setStartedAt(now);
@@ -262,6 +280,36 @@ public class JobService {
return new JoinOrOpenResult(saved, JoinOrOpenResult.Disposition.OPENED);
}
/**
* Stable fingerprint of this job's input set SHA-256 over the sorted union of the inputs'
* lineage storage keys. {@code COUNT(DISTINCT ...)} over these gives "unique PDFs processed".
* {@code null} when there are no inputs (a standalone bookkeeping job, e.g. an AI Create
* session) such jobs still count toward doc_count but aren't a distinct input PDF.
*/
private static String computeFingerprint(Map<Path, Set<LineageSignature>> signaturesByInput) {
List<String> keys =
signaturesByInput.values().stream()
.flatMap(Set::stream)
.map(LineageSignature::asStorageKey)
.distinct()
.sorted()
.toList();
if (keys.isEmpty()) {
return null;
}
try {
MessageDigest md = MessageDigest.getInstance("SHA-256");
for (String key : keys) {
md.update(key.getBytes(StandardCharsets.UTF_8));
md.update((byte) 0);
}
return HexFormat.of().formatHex(md.digest());
} catch (NoSuchAlgorithmException e) {
// SHA-256 is guaranteed on every JRE; fall back to no fingerprint if it ever isn't.
return null;
}
}
private void recordAllInputs(UUID jobId, Map<Path, Set<LineageSignature>> signaturesByInput) {
for (Set<LineageSignature> signatures : signaturesByInput.values()) {
detector.record(jobId, signatures, ArtifactKind.INPUT);
@@ -62,6 +62,27 @@ public class ProcessingJob implements Serializable {
@Column(name = "doc_units", nullable = false)
private Integer docUnits = 0;
/**
* Number of input files this charge represents the count dimension, kept distinct from {@link
* #docUnits} (which scales with file size). A split (1 input many outputs) stays 1; a merge
* (N inputs 1 output) is N. Fixed at open; joined steps never change it.
*
* <p>The {@code columnDefinition} default keeps the ddl-auto ADD COLUMN safe on an
* already-populated {@code processing_job} (a bare {@code NOT NULL} add is rejected by Postgres
* on a non-empty table).
*/
@Column(name = "doc_count", nullable = false, columnDefinition = "integer not null default 1")
private Integer docCount = 1;
/**
* Correlation id of the automation run that opened this job ({@code X-Stirling-Run-Id}), or
* {@code null} for a standalone tool call. Lineage joins are scoped to a single run id, so two
* separate runs never merge even on identical bytes; a null run id never joins (each standalone
* call is its own charge).
*/
@Column(name = "run_id", length = 64)
private String runId;
@Column(name = "step_count", nullable = false)
private Integer stepCount = 0;
@@ -73,6 +73,21 @@ public class DefaultHashLineageDetector implements HashLineageDetector {
return store.findOpenJobForSignatures(userId, signatures, workflowWindow);
}
@Override
public Optional<LineageMatch> detect(
Long userId, String runId, Set<LineageSignature> signatures) {
Objects.requireNonNull(userId, "userId");
Objects.requireNonNull(signatures, "signatures");
if (runId == null) {
// No run context standalone call; never lineage-joins (each is its own charge).
return Optional.empty();
}
if (signatures.isEmpty()) {
return Optional.empty();
}
return store.findOpenJobForSignatures(userId, signatures, workflowWindow, runId);
}
@Override
public void record(UUID jobId, Path file, ArtifactKind kind) throws IOException {
Objects.requireNonNull(file, "file");
@@ -50,6 +50,17 @@ public interface HashLineageDetector {
*/
Optional<LineageMatch> detect(Long userId, Set<LineageSignature> signatures);
/**
* Run-scoped detect: as {@link #detect(Long, Set)} but only matches open jobs belonging to the
* automation run {@code runId}, so lineage joins never cross runs (two separate runs on
* identical bytes stay distinct charges). The default ignores {@code runId} (for simple test
* doubles); the production detector overrides it to scope the store lookup.
*/
default Optional<LineageMatch> detect(
Long userId, String runId, Set<LineageSignature> signatures) {
return detect(userId, signatures);
}
/**
* Same as {@link #record(UUID, Path, ArtifactKind)} but operating on pre-computed signatures.
* Empty {@code signatures} is a no-op.
@@ -34,6 +34,19 @@ public interface JobLineageStore {
Optional<LineageMatch> findOpenJobForSignatures(
Long userId, Set<LineageSignature> candidates, Duration workflowWindow);
/**
* Run-scoped variant: as {@link #findOpenJobForSignatures(Long, Set, Duration)} but
* additionally constrained to open jobs whose {@code run_id} equals {@code runId}. Lineage
* joins are scoped to a single automation run, so a pipeline's sub-steps group into one charge
* while two separate runs on identical bytes stay distinct. The default delegates to the
* unscoped lookup (for test doubles that don't model run ids); the production store overrides
* it with a filtered query.
*/
default Optional<LineageMatch> findOpenJobForSignatures(
Long userId, Set<LineageSignature> candidates, Duration workflowWindow, String runId) {
return findOpenJobForSignatures(userId, candidates, workflowWindow);
}
/** Deletes records created before {@code cutoff}. Returns the number of rows removed. */
int pruneOlderThan(Instant cutoff);
}
@@ -78,6 +78,24 @@ public class JpaJobLineageStore implements JobLineageStore {
return matches.isEmpty() ? Optional.empty() : Optional.of(matches.get(0));
}
@Override
public Optional<LineageMatch> findOpenJobForSignatures(
Long userId, Set<LineageSignature> candidates, Duration workflowWindow, String runId) {
Objects.requireNonNull(userId, "userId");
Objects.requireNonNull(candidates, "candidates");
Objects.requireNonNull(workflowWindow, "workflowWindow");
Objects.requireNonNull(runId, "runId");
if (candidates.isEmpty()) {
return Optional.empty();
}
List<String> storageKeys = candidates.stream().map(LineageSignature::asStorageKey).toList();
LocalDateTime since = LocalDateTime.now().minus(workflowWindow);
List<LineageMatch> matches =
hashRepository.findOpenJobsForSignaturesInRun(
userId, JobStatus.OPEN, since, storageKeys, runId, Limit.of(1));
return matches.isEmpty() ? Optional.empty() : Optional.of(matches.get(0));
}
@Override
@Transactional
public int pruneOlderThan(Instant cutoff) {
@@ -43,6 +43,29 @@ public interface JobArtifactHashRepository
@Param("signatures") Collection<String> signatures,
Limit limit);
/**
* Run-scoped lineage lookup: as {@link #findOpenJobsForSignatures} but additionally requires
* {@code j.run_id = :runId}, so only jobs belonging to the same automation run can be joined.
*/
@Query(
"SELECT new stirling.software.saas.payg.lineage.LineageMatch("
+ " h.id.jobId, h.id.kind, j.lastStepAt)"
+ " FROM JobArtifactHash h"
+ " JOIN ProcessingJob j ON j.id = h.id.jobId"
+ " WHERE j.ownerUserId = :userId"
+ " AND j.status = :openStatus"
+ " AND j.lastStepAt > :since"
+ " AND j.runId = :runId"
+ " AND h.id.contentHash IN :signatures"
+ " ORDER BY j.lastStepAt DESC")
List<LineageMatch> findOpenJobsForSignaturesInRun(
@Param("userId") Long userId,
@Param("openStatus") JobStatus openStatus,
@Param("since") LocalDateTime since,
@Param("signatures") Collection<String> signatures,
@Param("runId") String runId,
Limit limit);
/** Prunes rows older than {@code cutoff}; run from a scheduled task. */
@Modifying
@Query("DELETE FROM JobArtifactHash h WHERE h.createdAt < :cutoff")
@@ -18,14 +18,13 @@ public interface WalletLedgerRepository extends JpaRepository<WalletLedgerEntry,
List<WalletLedgerEntry> findTop20ByTeamIdOrderByIdDesc(Long teamId);
/**
* Per-category debit totals over an arbitrary window, as positive units. Replaces the
* calendar-month {@code wallet_category_summary} view on the wallet endpoint subscribed
* teams' billing windows are anchored to the Stripe subscription period, not month starts. Rows
* with {@code NULL} category (system entries) are excluded; BYPASSED never reaches the ledger
* by construction.
* Per-category debit totals with BOTH the size-scaled unit sum and the input-file count ({@code
* doc_count}) over a window. Rows: {@code [category, units, docs]}. Lets the wallet show, per
* category, "X PDFs · Y meter units" rather than conflating the two.
*/
@Query(
"SELECT e.billingCategory AS category, COALESCE(SUM(-e.amountUnits), 0) AS units"
"SELECT e.billingCategory AS category, COALESCE(SUM(-e.amountUnits), 0) AS units,"
+ " COALESCE(SUM(e.docCount), 0) AS docs"
+ " FROM WalletLedgerEntry e"
+ " WHERE e.teamId = :teamId"
+ " AND e.entryType = :entryType"
@@ -33,7 +32,36 @@ public interface WalletLedgerRepository extends JpaRepository<WalletLedgerEntry,
+ " AND e.occurredAt >= :periodStart"
+ " AND e.occurredAt < :periodEnd"
+ " GROUP BY e.billingCategory")
List<Object[]> sumPeriodAmountByCategory(
List<Object[]> sumPeriodByCategoryWithDocs(
@Param("teamId") Long teamId,
@Param("entryType") LedgerEntryType entryType,
@Param("periodStart") LocalDateTime periodStart,
@Param("periodEnd") LocalDateTime periodEnd);
/**
* Period usage analytics in one row: {@code [docsProcessed, uniquePdfs, sizeMultiplierPdfs]}.
* {@code docsProcessed} sums input-file counts; {@code uniquePdfs} counts distinct input
* fingerprints (a file hit by N operations counts once); {@code sizeMultiplierPdfs} sums the
* input files on charges where the size multiplier kicked in (units billed &gt; input files).
* DEBIT + non-null category only.
*
* <p>Returns a single-element {@code List} (aggregate-only query always one row). Declared as
* {@code List<Object[]>} rather than {@code Object[]}: Spring Data treats an {@code Object[]}
* return as a <em>collection</em> and hands back {@code Object[]{ row }}, so the caller would
* read the columns one level too deep take {@code get(0)}.
*/
@Query(
"SELECT COALESCE(SUM(e.docCount), 0) AS docs,"
+ " COUNT(DISTINCT e.documentFingerprint) AS uniquePdfs,"
+ " COALESCE(SUM(CASE WHEN (-e.amountUnits) > e.docCount THEN e.docCount ELSE 0"
+ " END), 0) AS sizeMultiplierPdfs"
+ " FROM WalletLedgerEntry e"
+ " WHERE e.teamId = :teamId"
+ " AND e.entryType = :entryType"
+ " AND e.billingCategory IS NOT NULL"
+ " AND e.occurredAt >= :periodStart"
+ " AND e.occurredAt < :periodEnd")
List<Object[]> periodUsageAnalytics(
@Param("teamId") Long teamId,
@Param("entryType") LedgerEntryType entryType,
@Param("periodStart") LocalDateTime periodStart,
@@ -74,6 +74,30 @@ public class WalletLedgerEntry implements Serializable {
@Column(name = "policy_id")
private Long policyId;
/**
* Number of input files this entry billed (the count dimension, distinct from size-scaled
* {@link #amountUnits}). Denormalised from {@code processing_job.doc_count} so usage analytics
* "PDFs processed" sum one table. Defaults to 1; system/aggregate entries may leave it 1.
*
* <p>The {@code columnDefinition} default keeps the ddl-auto ADD COLUMN safe on an
* already-populated {@code wallet_ledger} (a bare {@code NOT NULL} add is rejected by Postgres
* on a non-empty table).
*/
@Column(name = "doc_count", nullable = false, columnDefinition = "integer not null default 1")
private Integer docCount = 1;
/**
* SHA-256 of this entry's input file <em>set</em> (the charge's whole input list, sorted).
* {@code COUNT(DISTINCT ...)} over a period approximates unique PDFs processed. Exact within a
* run (a file's chain/split steps share the run's charge, so one fingerprint), and for the
* single-input common case; but the <em>same</em> file reused across different groupings e.g.
* standalone, then later merged as {A,B} yields different set-fingerprints and is counted
* once per grouping. {@code null} for aggregate/system entries (grants, linked-instance sync)
* that don't map to a single document set.
*/
@Column(name = "document_fingerprint", length = 64)
private String documentFingerprint;
@Column(name = "stripe_event_id", length = 128)
private String stripeEventId;
@@ -32,6 +32,8 @@ import stirling.software.proprietary.security.repository.TeamMembershipRepositor
import stirling.software.saas.procurement.config.ProcurementConfigurationProperties;
import stirling.software.saas.procurement.model.ProcurementDeal;
import stirling.software.saas.procurement.model.ProcurementQuote;
import stirling.software.saas.procurement.model.QuoteDetails;
import stirling.software.saas.procurement.pricing.ProcurementPricingService;
import stirling.software.saas.procurement.pricing.QuoteConfig;
import stirling.software.saas.procurement.pricing.QuoteLineItem;
import stirling.software.saas.procurement.service.ProcurementService;
@@ -56,16 +58,19 @@ public class ProcurementController {
private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
private final ProcurementService procurement;
private final ProcurementPricingService pricing;
private final TeamMembershipRepository memberRepo;
private final UserRepository userRepository;
private final ProcurementConfigurationProperties config;
public ProcurementController(
ProcurementService procurement,
ProcurementPricingService pricing,
TeamMembershipRepository memberRepo,
UserRepository userRepository,
ProcurementConfigurationProperties config) {
this.procurement = Objects.requireNonNull(procurement);
this.pricing = Objects.requireNonNull(pricing);
this.memberRepo = Objects.requireNonNull(memberRepo);
this.userRepository = Objects.requireNonNull(userRepository);
this.config = Objects.requireNonNull(config);
@@ -77,29 +82,53 @@ public class ProcurementController {
long volume,
int users,
int intensity, // policy posture (runs/PDF): 2 / 4 / 7; 0 default Governed
double sizeMult, // PDF-size tier multiplier: 1.0 / 1.4 / 2.4; 0 no uplift
String deployment,
int termYears,
String serviceLevel,
boolean indemnification,
boolean training,
boolean qbr,
boolean offlineLicense,
String currency,
String businessName) {
String businessName,
// Buyer / AP details (all optional). Country + currency intentionally out of scope.
String contactName,
String contactEmail,
String addressLine1,
String addressLine2,
String city,
String region,
String postalCode,
String poNumber,
String taxId) {
QuoteConfig toConfig() {
return new QuoteConfig(
volume,
users,
intensity,
sizeMult,
deployment,
termYears,
serviceLevel,
indemnification,
training,
qbr,
offlineLicense,
currency);
}
QuoteDetails toDetails() {
return new QuoteDetails(
businessName,
contactName,
contactEmail,
addressLine1,
addressLine2,
city,
region,
postalCode,
poNumber,
taxId);
}
}
public record QuoteResponse(
@@ -109,10 +138,15 @@ public class ProcurementController {
String currency,
long annualNetMinor,
long tcvMinor,
// First post-term renewal fee after the CPI escalator, and that escalator as a whole
// percent the committed term is flat, so these describe only the auto-renewal.
long renewalAnnualNetMinor,
int cpiRatePct,
List<QuoteLineItem> lineItems,
String validUntil,
String stripeQuoteId,
String invoiceUrl,
String invoicePdf,
QuoteConfigEcho config) {}
/**
@@ -124,19 +158,33 @@ public class ProcurementController {
long volume,
int users,
int intensity,
double sizeMult,
String deployment,
int termYears,
String serviceLevel,
boolean indemnification,
boolean training,
boolean qbr,
boolean offlineLicense,
String currency,
String businessName) {}
String businessName,
String contactName,
String contactEmail,
String addressLine1,
String addressLine2,
String city,
String region,
String postalCode,
String poNumber,
String taxId) {}
/** Trial setup captured before the trial starts: deployment target + seat count. */
public record StartTrialRequest(String deployment, int users) {}
public record SnapshotResponse(
Long dealId,
String stage,
String deployment,
int seats,
String trialStartedAt,
String trialEndsAt,
int trialExtensionsUsed,
@@ -165,12 +213,12 @@ public class ProcurementController {
}
private static final SnapshotResponse EMPTY_SNAPSHOT =
new SnapshotResponse(null, null, null, null, 0, false, null, null);
new SnapshotResponse(null, null, null, 0, null, null, 0, false, null, null);
/**
* Download the offline / air-gapped licence file (.lic) for the team, when the paid offline
* add-on was purchased. 404 when there's no licence or the add-on wasn't taken we don't leak
* that a licence exists to a team without the add-on.
* Download the offline / air-gapped licence file (.lic) for the team available for an
* air-gapped deployment from the trial licence onward. 404 when there's no licence yet or the
* deployment isn't air-gapped, so we don't leak that a licence exists.
*/
@GetMapping("/license/file")
@PreAuthorize("isAuthenticated()")
@@ -193,10 +241,15 @@ public class ProcurementController {
@PostMapping("/trial/start")
@PreAuthorize("isAuthenticated()")
public ResponseEntity<SnapshotResponse> startTrial(Authentication auth) {
public ResponseEntity<SnapshotResponse> startTrial(
@RequestBody(required = false) StartTrialRequest request, Authentication auth) {
Long teamId = requireLeader(auth);
if (teamId == null) return ResponseEntity.status(HttpStatus.FORBIDDEN).build();
return ResponseEntity.ok(toSnapshot(procurement.startTrial(teamId), true));
// Body is optional so an older client (no setup step) still starts a cloud trial.
String deployment = request != null ? request.deployment() : null;
int seats = request != null ? request.users() : 0;
return ResponseEntity.ok(
toSnapshot(procurement.startTrial(teamId, deployment, seats), true));
}
@PostMapping("/trial/extend")
@@ -218,9 +271,7 @@ public class ProcurementController {
Long teamId = requireLeader(auth);
if (teamId == null) return ResponseEntity.status(HttpStatus.FORBIDDEN).build();
return ResponseEntity.ok(
toQuote(
procurement.buildQuote(
teamId, request.toConfig(), request.businessName())));
toQuote(procurement.buildQuote(teamId, request.toConfig(), request.toDetails())));
}
// Issue + accept are Supabase edge functions (they own Stripe): issue-procurement-quote turns a
@@ -323,6 +374,8 @@ public class ProcurementController {
return new SnapshotResponse(
deal.getDealId(),
deal.getStage(),
deal.getDeployment(),
deal.getSeats(),
str(deal.getTrialStartedAt()),
str(deal.getTrialEndsAt()),
deal.getTrialExtensionsUsed(),
@@ -339,23 +392,40 @@ public class ProcurementController {
q.getCurrency(),
q.getAnnualNetMinor(),
q.getTcvMinor(),
// Prefer the renewal locked at quote time; fall back to a live projection for
// quotes
// priced before the column existed.
q.getRenewalAnnualMinor() > 0
? q.getRenewalAnnualMinor()
: pricing.renewalAnnualMinor(q.getAnnualNetMinor()),
pricing.cpiRatePct(),
parseLineItems(q.getLineItemsJson()),
q.getValidUntil() == null ? null : q.getValidUntil().toString(),
q.getStripeQuoteId(),
q.getStripeInvoiceUrl(),
q.getStripeInvoicePdf(),
new QuoteConfigEcho(
q.getVolume(),
0,
q.getIntensity(),
q.getSizeMult(),
q.getDeployment(),
q.getTermYears(),
q.getServiceLevel(),
q.isIndemnification(),
q.isTraining(),
q.isQbr(),
q.isOfflineLicense(),
q.getCurrency(),
q.getBusinessName()));
q.getBusinessName(),
q.getContactName(),
q.getContactEmail(),
q.getAddressLine1(),
q.getAddressLine2(),
q.getCity(),
q.getRegion(),
q.getPostalCode(),
q.getPoNumber(),
q.getTaxId()));
}
private List<QuoteLineItem> parseLineItems(String json) {
@@ -51,6 +51,15 @@ public class ProcurementDeal implements Serializable {
@Column(name = "stage", nullable = false, length = 32)
private String stage = STAGE_TRIAL;
// Deployment target + seat count captured at trial start (the setup step); they seed the quote
// builder so it opens on the buyer's real environment. The quote remains the commercial source
// of truth these are just the starting point, editable when the quote is built.
@Column(name = "deployment", nullable = false, length = 16)
private String deployment = "cloud";
@Column(name = "seats", nullable = false)
private int seats;
@Column(name = "trial_started_at")
private LocalDateTime trialStartedAt;
@@ -66,6 +66,10 @@ public class ProcurementQuote implements Serializable {
@Column(name = "intensity", nullable = false)
private int intensity = 4;
/** File-size tier multiplier on the rate (D93): Compact 1.0, Standard 1.4, Heavy 2.4. */
@Column(name = "size_mult", nullable = false)
private double sizeMult = 1.0;
@Column(name = "deployment", length = 24)
private String deployment;
@@ -84,15 +88,17 @@ public class ProcurementQuote implements Serializable {
@Column(name = "qbr", nullable = false)
private boolean qbr;
@Column(name = "offline_license", nullable = false)
private boolean offlineLicense;
@Column(name = "annual_net_minor", nullable = false)
private long annualNetMinor;
@Column(name = "tcv_minor", nullable = false)
private long tcvMinor;
// First post-term renewal fee (annual net + one CPI step), locked at quote time so the buyer's
// quoted renewal doesn't drift if the rate card changes later.
@Column(name = "renewal_annual_minor", nullable = false)
private long renewalAnnualMinor;
@Column(name = "line_items", columnDefinition = "text")
private String lineItemsJson;
@@ -105,10 +111,47 @@ public class ProcurementQuote implements Serializable {
@Column(name = "stripe_invoice_url", columnDefinition = "text")
private String stripeInvoiceUrl;
// Direct PDF link for that first invoice (Stripe invoice_pdf), set at accept alongside the URL;
// persisted so the portal's download button works after a reload, not just in the accept
// response.
@Column(name = "stripe_invoice_pdf", columnDefinition = "text")
private String stripeInvoicePdf;
// Buyer's company name (shown on the quote/agreement); echoed back so an edit remembers it.
@Column(name = "business_name", length = 255)
private String businessName;
// Buyer/AP details captured on the quote's "Your details" step. All optional they never gate
// quote generation; they flow onto the Stripe customer (name + bill-to address) and the invoice
// (PO number + tax id as invoice custom fields), and seed the builder on a re-edit. Country and
// currency are intentionally out of scope for now.
@Column(name = "contact_name", length = 255)
private String contactName;
@Column(name = "contact_email", length = 255)
private String contactEmail;
@Column(name = "address_line1", length = 255)
private String addressLine1;
@Column(name = "address_line2", length = 255)
private String addressLine2;
@Column(name = "city", length = 128)
private String city;
@Column(name = "region", length = 128)
private String region;
@Column(name = "postal_code", length = 32)
private String postalCode;
@Column(name = "po_number", length = 128)
private String poNumber;
@Column(name = "tax_id", length = 64)
private String taxId;
@Column(name = "valid_until")
private LocalDate validUntil;
@@ -0,0 +1,21 @@
package stirling.software.saas.procurement.model;
/**
* Buyer / AP details captured on the quote's "Your details" step: the company and signatory
* contact, a billing address, and a PO number / tax id for the invoice. These are not pricing
* inputs (they never touch {@link stirling.software.saas.procurement.pricing.QuoteConfig}); they
* ride alongside the priced config so the quote can be re-seeded on an edit and the fields can flow
* onto the Stripe customer and invoice. All fields are optional. Country and currency are out of
* scope for now.
*/
public record QuoteDetails(
String businessName,
String contactName,
String contactEmail,
String addressLine1,
String addressLine2,
String city,
String region,
String postalCode,
String poNumber,
String taxId) {}
@@ -21,7 +21,8 @@ public record PricingRates(
long selfHostDeployMinor, // flat: self-hosted deployment
long airgapDeployMinor, // flat: air-gapped deployment
long qbrAnnualMinor, // flat: quarterly business reviews
long trainingOneTimeMinor) { // one-time: onboarding & training
long trainingOneTimeMinor, // one-time: onboarding & training
double cpiEscalator) { // fixed CPI uplift on the annual fee at each post-term renewal
public static PricingRates defaults() {
return new PricingRates(
@@ -34,7 +35,8 @@ public record PricingRates(
1_200_000, // self-hosted $12,000 / yr
3_600_000, // air-gapped $36,000 / yr
800_000, // QBRs $8,000 / yr
750_000); // onboarding & training $7,500 one-time
750_000, // onboarding & training $7,500 one-time
0.03); // 3% CPI escalator per renewal (committed term stays flat)
}
public double termDiscount(int termYears) {
@@ -61,6 +61,11 @@ public class ProcurementPricingService {
* (Math.log(runVol / (double) RUN_CURVE_KNEE) / LOG2))
: 0.0;
double rate = Math.max(rates.floorRatePerRun(), rates.listRatePerRun() * (1.0 - volDisc));
// File-size multiplier (D93): larger, image-heavy PDFs cost more OCR/compute/storage. Folds
// into the per-run rate after the floor, so it flows through the meter, TCV and renewal.
// QuoteConfig has already snapped it to a known tier, so a tampered request can't sneak a
// cheaper factor in.
rate *= cfg.sizeMult();
double termDisc = rates.termDiscount(cfg.termYears());
// The meter is a whole-dollar figure (the quote reads in dollars), then minor units.
@@ -82,6 +87,7 @@ public class ProcurementPricingService {
long annualNet = meterNetMinor + support + deploy + indemnity + qbr;
long tcv = annualNet * cfg.termYears() + training;
long renewalAnnual = renewalAnnualMinor(annualNet, rates);
double effectivePerPdf = rate * intensity; // quotes speak per-PDF-at-posture, never per-run
@@ -151,7 +157,29 @@ public class ProcurementPricingService {
QuoteLineItem.Kind.ONE_TIME,
training));
}
return new QuoteBreakdown(lines, annualNet, tcv, cfg.currency());
return new QuoteBreakdown(lines, annualNet, tcv, renewalAnnual, cfg.currency());
}
/** The default CPI escalator (fraction) applied to the annual fee on each post-term renewal. */
public double cpiEscalator() {
return PricingRates.defaults().cpiEscalator();
}
/** The CPI escalator as a whole-percent figure for buyer-facing copy (3% → 3). */
public int cpiRatePct() {
return (int) Math.round(cpiEscalator() * 100.0);
}
/**
* The escalated annual fee at the first renewal: the committed annual plus one CPI step. Used
* both when pricing a fresh quote and when echoing a stored one, so the two always agree.
*/
public long renewalAnnualMinor(long annualNetMinor) {
return renewalAnnualMinor(annualNetMinor, PricingRates.defaults());
}
private static long renewalAnnualMinor(long annualNetMinor, PricingRates rates) {
return Math.round(annualNetMinor * (1.0 + rates.cpiEscalator()));
}
private static long deployFeeMinor(String deployment, PricingRates rates) {
@@ -3,10 +3,15 @@ package stirling.software.saas.procurement.pricing;
import java.util.List;
/**
* The priced result of a {@link QuoteConfig}: the itemised lines plus the two headline figures the
* The priced result of a {@link QuoteConfig}: the itemised lines plus the headline figures the
* order form and Stripe checkout are built from. {@code annualNetMinor} is the recurring annual fee
* after the multi-year discount; {@code tcvMinor} is total contract value across the term including
* one-time fees. Minor units (cents).
* after the multi-year discount; {@code tcvMinor} is total contract value across the committed term
* including one-time fees; {@code renewalAnnualNetMinor} is the annual fee at the first post-term
* renewal after the fixed CPI escalator (the committed term itself is flat). Minor units (cents).
*/
public record QuoteBreakdown(
List<QuoteLineItem> lineItems, long annualNetMinor, long tcvMinor, String currency) {}
List<QuoteLineItem> lineItems,
long annualNetMinor,
long tcvMinor,
long renewalAnnualNetMinor,
String currency) {}
@@ -10,23 +10,35 @@ public record QuoteConfig(
long volume, // committed PDFs per year
int users, // seats (drives the volume auto-estimate when the buyer hasn't overridden)
int intensity, // policy posture: runs per PDF Essentials 2, Governed 4, Regulated 7
double sizeMult, // file-size tier multiplier on the rate Compact 1.0 / Standard 1.4 /
// Heavy 2.4
String deployment, // cloud | selfhost | airgap (priced flat; inherited from the trial)
int termYears, // 1..5
String serviceLevel, // standard | priority (both included) | dedicated (flat SE/CSM fee)
boolean indemnification,
boolean training,
boolean qbr,
boolean offlineLicense, // offline .lic availability (gates download; no longer priced here)
String currency) { // USD only for now
/** Default posture when none is chosen — Governed (x4), per the pricing alignment decision. */
public static final int DEFAULT_INTENSITY = 4;
/** Known file-size tier multipliers (D93): Compact 1.0, Standard 1.4, Heavy 2.4. */
private static final double[] SIZE_MULTS = {1.0, 1.4, 2.4};
public QuoteConfig {
if (termYears < 1) termYears = 1;
if (termYears > 5) termYears = 5;
if (intensity < 1) intensity = DEFAULT_INTENSITY;
if (serviceLevel == null || serviceLevel.isBlank()) serviceLevel = "standard";
if (currency == null || currency.isBlank()) currency = "USD";
// Snap the file-size multiplier to a known tier so a tampered request can't invent a
// cheaper
// one; absent (0.0) or unknown falls back to 1.0 (no uplift).
double snapped = 1.0;
for (double s : SIZE_MULTS) {
if (Math.abs(s - sizeMult) < 1e-9) snapped = s;
}
sizeMult = snapped;
}
}
@@ -24,6 +24,7 @@ import stirling.software.saas.procurement.license.EnterpriseLicenseService;
import stirling.software.saas.procurement.license.LicenseEntitlements;
import stirling.software.saas.procurement.model.ProcurementDeal;
import stirling.software.saas.procurement.model.ProcurementQuote;
import stirling.software.saas.procurement.model.QuoteDetails;
import stirling.software.saas.procurement.pricing.ProcurementPricingService;
import stirling.software.saas.procurement.pricing.QuoteBreakdown;
import stirling.software.saas.procurement.pricing.QuoteConfig;
@@ -96,28 +97,46 @@ public class ProcurementService {
/**
* Start (or restart) the free trial for a team: issue a mock trial licence and stamp the trial
* window on the deal. No Stripe: a no-card trial has no subscription; the entitlement is the
* Keygen licence, and the deal row is the journey state.
* Keygen licence, and the deal row is the journey state. The buyer's chosen deployment target
* ({@code cloud}/{@code selfhost}/{@code airgap}) and seat count are captured here so the quote
* builder opens seeded to their environment; both are still editable when the quote is built.
*/
@Transactional
public ProcurementDeal startTrial(Long teamId) {
public ProcurementDeal startTrial(Long teamId, String deployment, int seats) {
ProcurementDeal deal =
dealRepo.findByTeamId(teamId).orElseGet(() -> new ProcurementDeal(teamId));
LocalDateTime now = LocalDateTime.now();
LocalDateTime ends = now.plusDays(config.getTrialDurationDays());
deal.setStage(ProcurementDeal.STAGE_TRIAL);
deal.setDeployment(normalizeDeployment(deployment));
deal.setSeats(Math.max(0, seats));
deal.setTrialStartedAt(now);
deal.setTrialEndsAt(ends);
deal.setTrialExtensionsUsed(0);
deal.setLicenseRef(licenses.issueTrialLicense(teamId, leaderEmail(teamId), ends));
deal = dealRepo.save(deal);
log.info(
"[procurement] trial started team={} deal={} ends={}",
"[procurement] trial started team={} deal={} deployment={} seats={} ends={}",
teamId,
deal.getDealId(),
deal.getDeployment(),
deal.getSeats(),
ends);
return deal;
}
/**
* Constrain a caller-supplied deployment to the known set; anything else falls back to cloud.
*/
private static String normalizeDeployment(String deployment) {
if (deployment == null) return "cloud";
String d = deployment.trim().toLowerCase(Locale.ROOT);
return switch (d) {
case "selfhost", "airgap", "cloud" -> d;
default -> "cloud";
};
}
/** Extend the current trial by the configured increment, up to the cap. */
@Transactional
public ProcurementDeal extendTrial(Long teamId) {
@@ -145,7 +164,7 @@ public class ProcurementService {
/** Price a quote config server-side and persist it as a draft against the team's deal. */
@Transactional
public ProcurementQuote buildQuote(Long teamId, QuoteConfig cfg, String businessName) {
public ProcurementQuote buildQuote(Long teamId, QuoteConfig cfg, QuoteDetails details) {
ProcurementDeal deal =
dealRepo.findByTeamId(teamId).orElseGet(() -> new ProcurementDeal(teamId));
if (ProcurementDeal.STAGE_LIVE.equals(deal.getStage())) {
@@ -168,16 +187,26 @@ public class ProcurementService {
quote.setVolume(cfg.volume());
quote.setSeats(cfg.users() > 0 ? cfg.users() : null);
quote.setIntensity(cfg.intensity());
quote.setSizeMult(cfg.sizeMult());
quote.setDeployment(cfg.deployment());
quote.setTermYears(cfg.termYears());
quote.setServiceLevel(cfg.serviceLevel());
quote.setIndemnification(cfg.indemnification());
quote.setTraining(cfg.training());
quote.setQbr(cfg.qbr());
quote.setOfflineLicense(cfg.offlineLicense());
quote.setBusinessName(businessName);
quote.setBusinessName(details.businessName());
quote.setContactName(details.contactName());
quote.setContactEmail(details.contactEmail());
quote.setAddressLine1(details.addressLine1());
quote.setAddressLine2(details.addressLine2());
quote.setCity(details.city());
quote.setRegion(details.region());
quote.setPostalCode(details.postalCode());
quote.setPoNumber(details.poNumber());
quote.setTaxId(details.taxId());
quote.setAnnualNetMinor(breakdown.annualNetMinor());
quote.setTcvMinor(breakdown.tcvMinor());
quote.setRenewalAnnualMinor(breakdown.renewalAnnualNetMinor());
quote.setLineItemsJson(writeLineItems(breakdown));
quote.setValidUntil(LocalDate.now().plusDays(30));
quote = quoteRepo.save(quote);
@@ -275,7 +304,7 @@ public class ProcurementService {
q != null && q.isIndemnification(),
q != null && q.isTraining(),
q != null && q.isQbr(),
q != null && q.isOfflineLicense(),
"airgap".equalsIgnoreCase(deployment), // offline .lic = air-gapped deploy
deal.getDealId(),
deal.getSubscriptionId());
return licenses.issueAnnualLicense(
@@ -287,30 +316,26 @@ public class ProcurementService {
}
/**
* Check out the offline/air-gapped licence file for a team, when the offline add-on was
* purchased. Requires an issued licence on the deal and the accepted quote to carry the offline
* add-on; returns empty otherwise (so the controller can 404 rather than leak that a licence
* exists). The certificate is generated on demand by Keygen and never stored.
* Check out the offline/air-gapped licence file (.lic) for a team. Available for an air-gapped
* deployment (chosen at trial setup) from the trial licence onward cloud/self-hosted verify
* online against Keygen and don't get a file. Returns empty when there's no licence yet or the
* deployment isn't air-gapped, so the controller can 404 rather than leak that a licence
* exists. The certificate is generated on demand by Keygen (from whatever licence the deal
* currently holds trial or committed annual) and never stored.
*
* <p>By design a team can self-select air-gapped at trial and download a real signed .lic
* before paying that's bounded: the trial licence carries {@code expiry = trialEndsAt}, so
* the file the verifier accepts self-expires at trial end. The buyer must re-download after
* provisioning to get the committed-term file (the portal warns about this).
*/
@Transactional(readOnly = true)
public Optional<String> offlineLicenseFile(Long teamId) {
ProcurementDeal deal = dealRepo.findByTeamId(teamId).orElse(null);
if (deal == null || deal.getLicenseRef() == null) return Optional.empty();
if (!hasOfflineAddOn(deal)) return Optional.empty();
if (!"airgap".equalsIgnoreCase(deal.getDeployment())) return Optional.empty();
return Optional.of(licenses.checkOutLicenseFile(deal.getLicenseRef()));
}
/**
* Whether the deal's <b>accepted</b> quote carries the paid offline-licence add-on. Gated on
* the accepted quote (not the latest) so merely toggling the add-on on an unaccepted draft
* can't unlock the offline file it's only available once the add-on has actually been bought.
*/
private boolean hasOfflineAddOn(ProcurementDeal deal) {
if (deal.getAcceptedQuoteId() == null) return false;
ProcurementQuote quote = quoteRepo.findById(deal.getAcceptedQuoteId()).orElse(null);
return quote != null && quote.isOfflineLicense();
}
/**
* Reset a team's procurement: delete the deal (quotes + activity cascade). For
* re-demos/testing.
@@ -0,0 +1,103 @@
package stirling.software.saas.usage;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.List;
import org.springframework.context.annotation.Profile;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.security.access.prepost.PreAuthorize;
import org.springframework.security.core.Authentication;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.proprietary.audit.AuditLevel;
import stirling.software.proprietary.config.AuditConfigurationProperties;
import stirling.software.proprietary.model.TeamMembership;
import stirling.software.proprietary.model.api.usage.FleetUsageStats;
import stirling.software.proprietary.repository.PersistentAuditEventRepository;
import stirling.software.proprietary.security.database.repository.UserRepository;
import stirling.software.proprietary.security.model.User;
import stirling.software.proprietary.security.repository.TeamMembershipRepository;
import stirling.software.saas.util.AuthenticationUtils;
/**
* SaaS counterpart to {@code FleetUsageController}: the same "Free PDF Editors" figures, scoped to
* the caller's team (one SaaS backend serves many tenants, so the self-hosted server-wide variant
* is disabled here via {@code @Profile("!saas")}).
*
* <ul>
* <li>editorsDeployed number of team members ({@code team_memberships}), not seat limits;
* <li>activeThisMonth distinct members with a free-UI ("WEB", non-{@code UI_DATA}) audit event
* in the last 30 days, clamped to a subset of deployed;
* <li>pdfsProcessed the team's cumulative free-UI PDF/file operations.
* </ul>
*
* <p>Team resolution + membership mirror {@code PaygWalletController}. Audit-derived figures are
* null (rendered "N/A") when EE auditing is below STANDARD. Cost is always $0 (client literal).
*/
@Slf4j
@RestController
@RequestMapping("/api/v1/usage")
@Profile("saas")
@RequiredArgsConstructor
public class SaasFleetUsageController {
private static final List<String> PDF_TYPES = List.of("PDF_PROCESS", "FILE_OPERATION");
private final UserRepository userRepository;
private final TeamMembershipRepository memberRepo;
private final PersistentAuditEventRepository auditRepository;
private final AuditConfigurationProperties auditConfig;
@GetMapping("/fleet-stats")
@PreAuthorize("isAuthenticated()")
@Transactional(readOnly = true)
public ResponseEntity<FleetUsageStats> fleetStats(Authentication auth) {
User user;
try {
user = AuthenticationUtils.getCurrentUser(auth, userRepository);
} catch (SecurityException e) {
return ResponseEntity.status(HttpStatus.UNAUTHORIZED).build();
}
List<TeamMembership> primary = memberRepo.findPrimaryMembership(user.getId());
if (primary.isEmpty()) {
// Authenticated caller without a team shouldn't happen post-migration; report an
// empty fleet rather than 500.
return ResponseEntity.ok(new FleetUsageStats(0L, null, null));
}
Long teamId = primary.get(0).getTeam().getId();
List<String> members =
memberRepo.findByTeamId(teamId).stream()
.map(m -> m.getUser().getUsername())
.toList();
Long deployed = (long) members.size();
// Guard the empty IN-list (invalid JPQL) as well as the audit-level gate.
boolean auditOn = !members.isEmpty() && auditConfig.isLevelEnabled(AuditLevel.STANDARD);
Instant since = Instant.now().minus(30, ChronoUnit.DAYS);
Long active =
auditOn
? auditRepository
.countDistinctPrincipalsBySourceExcludingTypeAndPrincipalInAfter(
"WEB", "UI_DATA", members, since)
: null;
Long pdfs =
auditOn
? auditRepository.countByTypeInAndSourceAndPrincipalInAndTimestampAfter(
PDF_TYPES, "WEB", members, Instant.EPOCH)
: null;
if (active != null && active > deployed) {
active = deployed; // active editors are a subset of those deployed
}
return ResponseEntity.ok(new FleetUsageStats(deployed, active, pdfs));
}
}
@@ -0,0 +1,7 @@
-- Direct PDF link for a procurement quote's first invoice (Stripe invoice_pdf), stored at accept
-- alongside stripe_invoice_url so the portal's "Download invoice" button survives a reload instead
-- of relying on the transient accept response. Written by the accept edge function via the
-- procurement_set_quote_accepted RPC; read by the Java backend via JPA. A Supabase twin mirrors it.
ALTER TABLE stirling_pdf.procurement_quote
ADD COLUMN IF NOT EXISTS stripe_invoice_pdf TEXT;
@@ -0,0 +1,10 @@
-- Deployment target + seat count captured at the trial-start step (the setup dialog the demo shows
-- before a trial begins), stored on the deal so the quote builder seeds from the buyer's real
-- environment instead of a hardcoded default. deployment: cloud | selfhost | airgap. seats: 0 =
-- unspecified. Written and read by the Java backend via JPA. A Supabase twin migration mirrors it.
ALTER TABLE stirling_pdf.procurement_deal
ADD COLUMN IF NOT EXISTS deployment VARCHAR(16) NOT NULL DEFAULT 'cloud';
ALTER TABLE stirling_pdf.procurement_deal
ADD COLUMN IF NOT EXISTS seats INTEGER NOT NULL DEFAULT 0;
@@ -0,0 +1,7 @@
-- Persist the first post-term renewal fee (annual net + one CPI step) computed at quote time, so the
-- figure shown to the buyer is locked to what they were quoted rather than recomputed from the
-- current rate card on every read. Minor units. Written and read by the Java backend via JPA; a
-- Supabase twin migration mirrors it.
ALTER TABLE stirling_pdf.procurement_quote
ADD COLUMN IF NOT EXISTS renewal_annual_minor BIGINT NOT NULL DEFAULT 0;
@@ -0,0 +1,7 @@
-- File-size tier multiplier on the quote (D93): larger, image-heavy PDFs cost more, so the buyer
-- picks a size tier (Compact 1.0 / Standard 1.4 / Heavy 2.4) that scales the per-run rate. Persisted
-- so the quote re-prices and re-seeds the builder consistently. Defaults to 1.0 (no uplift) for rows
-- that predate the column. Written and read by the Java backend via JPA. A Supabase twin mirrors it.
ALTER TABLE stirling_pdf.procurement_quote
ADD COLUMN IF NOT EXISTS size_mult DOUBLE PRECISION NOT NULL DEFAULT 1.0;
@@ -0,0 +1,17 @@
-- Buyer / AP details captured on the quote's "Your details" step: the signatory contact and a
-- billing address, plus a PO number and tax id for the invoice. All optional (never gate quote
-- generation). Persisted so the quote re-seeds the builder on a re-edit and so the issue edge
-- function can put them on the Stripe customer (name + bill-to address) and invoice (PO / tax id
-- as custom fields). Country and currency are intentionally out of scope for now. Written and read
-- by the Java backend via JPA. A Supabase twin mirrors these columns.
ALTER TABLE stirling_pdf.procurement_quote
ADD COLUMN IF NOT EXISTS contact_name VARCHAR(255),
ADD COLUMN IF NOT EXISTS contact_email VARCHAR(255),
ADD COLUMN IF NOT EXISTS address_line1 VARCHAR(255),
ADD COLUMN IF NOT EXISTS address_line2 VARCHAR(255),
ADD COLUMN IF NOT EXISTS city VARCHAR(128),
ADD COLUMN IF NOT EXISTS region VARCHAR(128),
ADD COLUMN IF NOT EXISTS postal_code VARCHAR(32),
ADD COLUMN IF NOT EXISTS po_number VARCHAR(128),
ADD COLUMN IF NOT EXISTS tax_id VARCHAR(64);
@@ -0,0 +1,39 @@
-- PAYG size-scaled billing: run-scoped grouping + per-input-file counting.
--
-- Two independent axes now live on a charge:
-- * doc_units — billing quantity, scales with file size (existing column)
-- * doc_count — number of INPUT files (the "unique PDFs" dimension); a split (1→many) stays 1,
-- a merge (N→1) is N. Fixed at open; joined steps never change it.
-- Plus run_id, the automation-run correlation id used to group a run's tool sub-steps into one
-- charge (replacing the old content+time-window grouping) and to keep two separate runs distinct.
--
-- Everything is additive; no existing rows are modified, no columns dropped.
-- ── processing_job ───────────────────────────────────────────────────────────
ALTER TABLE processing_job ADD COLUMN IF NOT EXISTS run_id VARCHAR(64);
ALTER TABLE processing_job ADD COLUMN IF NOT EXISTS doc_count INTEGER NOT NULL DEFAULT 1;
COMMENT ON COLUMN processing_job.run_id IS
'Automation-run correlation id (X-Stirling-Run-Id); NULL for a standalone tool call. Lineage '
'joins are scoped to one run_id, so separate runs never merge even on identical bytes.';
COMMENT ON COLUMN processing_job.doc_count IS
'Number of input files this charge represents (the count dimension, distinct from size-scaled '
'doc_units). Split=1, merge=N. Fixed at open.';
-- ── wallet_ledger ────────────────────────────────────────────────────────────
-- Denormalise the count dimension + input fingerprint onto the DEBIT row so usage analytics
-- (unique PDFs, per-category counts, size-multiplier average) query one table and survive
-- processing_job pruning.
ALTER TABLE wallet_ledger ADD COLUMN IF NOT EXISTS doc_count INTEGER NOT NULL DEFAULT 1;
ALTER TABLE wallet_ledger ADD COLUMN IF NOT EXISTS document_fingerprint VARCHAR(64);
COMMENT ON COLUMN wallet_ledger.doc_count IS
'Input-file count for this entry (mirrors processing_job.doc_count); summed for "PDFs processed".';
COMMENT ON COLUMN wallet_ledger.document_fingerprint IS
'SHA-256 of the entry''s input file set; COUNT(DISTINCT ...) gives unique PDFs. NULL for '
'aggregate/system entries (e.g. linked-instance sync).';
-- Distinct-PDF + size-multiplier queries scan by team + period.
CREATE INDEX IF NOT EXISTS idx_wallet_ledger_team_period_fp
ON wallet_ledger (team_id, occurred_at, document_fingerprint)
WHERE document_fingerprint IS NOT NULL;
@@ -127,9 +127,11 @@ class PaygWalletControllerTest {
}
private void stubEmptyLedgerReads(long teamId) {
when(ledgerRepo.sumPeriodAmountByCategory(
when(ledgerRepo.sumPeriodByCategoryWithDocs(
eq(teamId), eq(LedgerEntryType.DEBIT), any(), any()))
.thenReturn(List.of());
when(ledgerRepo.periodUsageAnalytics(eq(teamId), eq(LedgerEntryType.DEBIT), any(), any()))
.thenReturn(List.<Object[]>of(new Object[] {0L, 0L, 0L}));
when(ledgerRepo.findTop20ByTeamIdOrderByIdDesc(teamId)).thenReturn(List.of());
}
@@ -187,12 +189,17 @@ class PaygWalletControllerTest {
when(shadowRepo.sumPaidUnits(eq(99L), any(), any())).thenReturn(312L);
when(billingService.estimateBillMinor(any(), eq(312L))).thenReturn(Optional.of(624L));
when(entitlementService.getSnapshot(99L)).thenReturn(snapshot(312L, 1250L));
when(ledgerRepo.sumPeriodAmountByCategory(eq(99L), eq(LedgerEntryType.DEBIT), any(), any()))
// [category, units, docs]: units scale with size, docs = input-file count.
when(ledgerRepo.sumPeriodByCategoryWithDocs(
eq(99L), eq(LedgerEntryType.DEBIT), any(), any()))
.thenReturn(
List.of(
new Object[] {BillingCategory.API, 110L},
new Object[] {BillingCategory.AI, 200L},
new Object[] {BillingCategory.AUTOMATION, 2L}));
new Object[] {BillingCategory.API, 110L, 90L},
new Object[] {BillingCategory.AI, 200L, 50L},
new Object[] {BillingCategory.AUTOMATION, 2L, 2L}));
// [docsProcessed, uniquePdfs, sizeMultiplierPdfs] single-row aggregate as List<Object[]>
when(ledgerRepo.periodUsageAnalytics(eq(99L), eq(LedgerEntryType.DEBIT), any(), any()))
.thenReturn(List.<Object[]>of(new Object[] {142L, 120L, 30L}));
when(ledgerRepo.findTop20ByTeamIdOrderByIdDesc(99L)).thenReturn(List.of());
ResponseEntity<WalletSnapshotResponse> resp =
@@ -214,6 +221,13 @@ class PaygWalletControllerTest {
assertThat(body.categoryBreakdown().api()).isEqualTo(110);
assertThat(body.categoryBreakdown().ai()).isEqualTo(200);
assertThat(body.categoryBreakdown().automation()).isEqualTo(2);
// Count dimension is surfaced separately from the size-scaled units.
assertThat(body.categoryDocs().api()).isEqualTo(90);
assertThat(body.categoryDocs().ai()).isEqualTo(50);
assertThat(body.categoryDocs().automation()).isEqualTo(2);
assertThat(body.docsProcessedThisPeriod()).isEqualTo(142);
assertThat(body.uniquePdfsThisPeriod()).isEqualTo(120);
assertThat(body.sizeMultiplierPdfsThisPeriod()).isEqualTo(30);
assertThat(body.stripeSubscriptionId()).isEqualTo("sub_test_99");
// Member role ledger never queried per-user.
verify(ledgerRepo, never()).sumPeriodAmountForMember(any(), any(), any(), any(), any());
@@ -35,7 +35,11 @@ class WalletSnapshotResponseTest {
/* spendUnitsThisPeriod= */ 12,
new CategoryBreakdown(5, 4, 3),
List.of(new MemberRow("u1", "Ann", "ann@example.com", 8)),
List.of(new ActivityRow(1L, "api", "API usage", "2026-06-02T10:00", 4)));
List.of(new ActivityRow(1L, "api", "API usage", "2026-06-02T10:00", 4)),
/* categoryDocs= */ new CategoryBreakdown(3, 2, 1),
/* docsProcessedThisPeriod= */ 6,
/* uniquePdfsThisPeriod= */ 5,
/* sizeMultiplierPdfsThisPeriod= */ 2);
}
@Test
@@ -107,7 +111,11 @@ class WalletSnapshotResponseTest {
0,
new CategoryBreakdown(0, 0, 0),
List.of(),
List.of());
List.of(),
new CategoryBreakdown(0, 0, 0),
0,
0,
0);
assertThat(free.billableLimit()).isNull();
assertThat(free.pricePerDocMinor()).isNull();
@@ -434,6 +434,52 @@ class PaygChargeInterceptorTest {
.isEqualTo(stirling.software.saas.payg.model.JobSource.PIPELINE);
}
@Test
void preHandle_runId_honouredOnlyWithAutomationHeader() throws Exception {
// An internal dispatch carries BOTH X-Stirling-Automation and X-Stirling-Run-Id, so the
// run id flows onto the ChargeContext (sub-steps of one run group into a single charge).
authenticateWithApiKey(makeUser(7L, 42L));
UUID jobId = UUID.randomUUID();
when(chargeService.openProcess(any(), anyList()))
.thenReturn(new ChargeOutcome(jobId, 1, ChargeOutcome.Disposition.OPENED));
org.mockito.ArgumentCaptor<stirling.software.saas.payg.charge.ChargeContext> ctxCaptor =
org.mockito.ArgumentCaptor.forClass(
stirling.software.saas.payg.charge.ChargeContext.class);
MockMultipartHttpServletRequest req = newMultipart();
req.addFile(new MockMultipartFile("file", "x.pdf", "application/pdf", "abc".getBytes()));
req.addHeader("X-Stirling-Automation", "true");
req.addHeader("X-Stirling-Run-Id", "run-abc");
interceptor.preHandle(req, new MockHttpServletResponse(), handlerMethodForFakeController());
verify(chargeService).openProcess(ctxCaptor.capture(), anyList());
assertThat(ctxCaptor.getValue().runId()).isEqualTo("run-abc");
}
@Test
void preHandle_runIdWithoutAutomationHeader_isIgnored() throws Exception {
// A raw external API call that sets X-Stirling-Run-Id on its own must NOT be able to group
// charges the run id is dropped so each call stays its own charge ("charge per API
// call").
authenticateWithApiKey(makeUser(7L, 42L));
UUID jobId = UUID.randomUUID();
when(chargeService.openProcess(any(), anyList()))
.thenReturn(new ChargeOutcome(jobId, 1, ChargeOutcome.Disposition.OPENED));
org.mockito.ArgumentCaptor<stirling.software.saas.payg.charge.ChargeContext> ctxCaptor =
org.mockito.ArgumentCaptor.forClass(
stirling.software.saas.payg.charge.ChargeContext.class);
MockMultipartHttpServletRequest req = newMultipart();
req.addFile(new MockMultipartFile("file", "x.pdf", "application/pdf", "abc".getBytes()));
req.addHeader("X-Stirling-Run-Id", "run-spoofed");
interceptor.preHandle(req, new MockHttpServletResponse(), handlerMethodForFakeController());
verify(chargeService).openProcess(ctxCaptor.capture(), anyList());
assertThat(ctxCaptor.getValue().runId()).isNull();
}
// --- BillingCategory categorisation + bypass fast-path -------------------------------------
@Test
@@ -183,6 +183,40 @@ class JobServiceTest {
assertThat(detector.recorded(result.job().getId(), input, ArtifactKind.INPUT)).isTrue();
}
@Test
void joinOrOpen_nullRunId_neverJoins(@TempDir Path tmp) throws IOException {
// A standalone call (no run id) opens its own charge even when content matches an open
// job run-scoped grouping only joins within the same automation run.
UUID existingId = UUID.randomUUID();
Path input = givenFile(tmp, "in.bin");
detector.willMatch(
input, new LineageMatch(existingId, ArtifactKind.INPUT, LocalDateTime.now()));
JobContext standalone =
new JobContext(
42L, 100L, JobSource.API, ProcessType.SINGLE_TOOL, 1L, 10); // null runId
JoinOrOpenResult result = service.joinOrOpen(standalone, List.of(input));
assertThat(result.disposition()).isEqualTo(JoinOrOpenResult.Disposition.OPENED);
verify(jobRepo, never()).findById(existingId);
}
@Test
void joinOrOpen_opened_setsRunIdAndDocCountAndFingerprint(@TempDir Path tmp)
throws IOException {
// Two input files, no match one fresh charge with doc_count = input-file count (2),
// the run id stamped, and a non-null fingerprint (used for unique-PDF counting).
Path a = givenFile(tmp, "a.bin");
Path b = givenFile(tmp, "b.bin");
JoinOrOpenResult result = service.joinOrOpen(ctx(42L, 100L, 10), List.of(a, b));
assertThat(result.disposition()).isEqualTo(JoinOrOpenResult.Disposition.OPENED);
assertThat(result.job().getDocCount()).isEqualTo(2);
assertThat(result.job().getRunId()).isEqualTo("test-run");
assertThat(result.job().getDocumentFingerprint()).isNotNull();
}
@Test
void joinOrOpen_emptyInputs_throws() {
assertThatThrownBy(() -> service.joinOrOpen(ctx(42L, 100L, 10), List.of()))
@@ -297,8 +331,10 @@ class JobServiceTest {
// --- helpers --------------------------------------------------------------------------------
private static JobContext ctx(long userId, long teamId, int stepLimit) {
// Lineage joins are scoped to an automation run, so the join scenarios below run inside a
// run id. The standalone (null run id) path is covered by joinOrOpen_nullRunId_neverJoins.
return new JobContext(
userId, teamId, JobSource.WEB, ProcessType.SINGLE_TOOL, 1L, stepLimit);
userId, teamId, JobSource.WEB, ProcessType.SINGLE_TOOL, 1L, stepLimit, "test-run");
}
private static ProcessingJob openJob(
@@ -18,8 +18,13 @@ class ProcurementPricingServiceTest {
private static QuoteConfig cfg(
long volume, int intensity, String deployment, int term, String sla) {
return cfgSize(volume, intensity, deployment, term, sla, 1.0);
}
private static QuoteConfig cfgSize(
long volume, int intensity, String deployment, int term, String sla, double sizeMult) {
return new QuoteConfig(
volume, 0, intensity, deployment, term, sla, false, false, false, false, "USD");
volume, 0, intensity, sizeMult, deployment, term, sla, false, false, false, "USD");
}
@Test
@@ -30,6 +35,7 @@ class ProcurementPricingServiceTest {
assertThat(q.annualNetMinor()).isEqualTo(175_200_000L); // $1,752,000
assertThat(q.tcvMinor()).isEqualTo(525_600_000L); // $5,256,000
assertThat(q.renewalAnnualNetMinor()).isEqualTo(180_456_000L); // $1,752,000 + 3% CPI
assertThat(lineAmount(q, "support")).isEqualTo(3_000_000L); // dedicated SE/CSM $30K
assertThat(lineAmount(q, "deployment")).isEqualTo(1_200_000L); // self-hosted $12K
}
@@ -46,6 +52,41 @@ class ProcurementPricingServiceTest {
assertThat(q.lineItems()).noneMatch(l -> l.key().equals("support"));
}
@Test
void renewalAppliesCpiEscalatorAfterAFlatTerm() {
// The committed term is flat (TCV = annual × years, asserted above). The 3% CPI escalator
// describes only the first post-term renewal: annual + one 3% step. It never touches TCV.
QuoteBreakdown q = pricing.price(cfg(6_000_000, 4, "cloud", 3, "standard"));
assertThat(q.renewalAnnualNetMinor())
.isEqualTo(Math.round(q.annualNetMinor() * 1.03)); // 16,527,800 17,023,634
assertThat(q.tcvMinor()).isEqualTo(q.annualNetMinor() * 3); // renewal is outside the TCV
assertThat(pricing.cpiRatePct()).isEqualTo(3);
assertThat(pricing.renewalAnnualMinor(q.annualNetMinor()))
.isEqualTo(q.renewalAnnualNetMinor()); // stored-quote echo agrees with pricing
}
@Test
void fileSizeTierMultipliesTheMeter() {
// D93: the size tier scales the per-run rate, so the meter (hence annual/TCV/renewal) grows
// while flat fees stay put. Compact (1.0) is the anchor; Standard is ×1.4, Heavy ×2.4.
long compact =
pricing.price(cfgSize(6_000_000, 4, "cloud", 3, "standard", 1.0)).annualNetMinor();
long standard =
pricing.price(cfgSize(6_000_000, 4, "cloud", 3, "standard", 1.4)).annualNetMinor();
long heavy =
pricing.price(cfgSize(6_000_000, 4, "cloud", 3, "standard", 2.4)).annualNetMinor();
assertThat(compact).isEqualTo(16_527_800L); // == the Northwind anchor (size 1.0)
assertThat(standard).isEqualTo(23_138_900L); // rate ×1.4
assertThat(compact).isLessThan(standard);
assertThat(standard).isLessThan(heavy);
// An unknown/tampered multiplier snaps back to 1.0 (no cheaper factor sneaks through).
assertThat(
pricing.price(cfgSize(6_000_000, 4, "cloud", 3, "standard", 0.3))
.annualNetMinor())
.isEqualTo(compact);
}
@Test
void rateFloorsAtHalfACent() {
// 100M × Regulated(×7) = 700M runs deep past the knee, so the per-run rate is pinned to
@@ -104,7 +145,7 @@ class ProcurementPricingServiceTest {
long base = pricing.price(cfg(6_000_000, 4, "cloud", 3, "standard")).annualNetMinor();
QuoteConfig c =
new QuoteConfig(
6_000_000, 0, 4, "cloud", 3, "standard", true, false, false, false, "USD");
6_000_000, 0, 4, 1.0, "cloud", 3, "standard", true, false, false, "USD");
QuoteBreakdown q = pricing.price(c);
assertThat(lineAmount(q, "indemnification")).isEqualTo(Math.round(base * 0.05));
}
@@ -113,7 +154,7 @@ class ProcurementPricingServiceTest {
void trainingIsOneTimeOutsideTheAnnual() {
QuoteConfig withTraining =
new QuoteConfig(
6_000_000, 0, 4, "cloud", 3, "standard", false, true, false, false, "USD");
6_000_000, 0, 4, 1.0, "cloud", 3, "standard", false, true, false, "USD");
QuoteBreakdown q = pricing.price(withTraining);
long baseAnnual = pricing.price(cfg(6_000_000, 4, "cloud", 3, "standard")).annualNetMinor();
@@ -0,0 +1,168 @@
package stirling.software.saas.usage;
import static org.assertj.core.api.Assertions.assertThat;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.ArgumentMatchers.anyList;
import static org.mockito.ArgumentMatchers.eq;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import java.time.Instant;
import java.util.List;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.mockito.Mock;
import org.mockito.junit.jupiter.MockitoExtension;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.security.core.Authentication;
import stirling.software.proprietary.audit.AuditLevel;
import stirling.software.proprietary.config.AuditConfigurationProperties;
import stirling.software.proprietary.model.Team;
import stirling.software.proprietary.model.TeamMembership;
import stirling.software.proprietary.model.api.usage.FleetUsageStats;
import stirling.software.proprietary.repository.PersistentAuditEventRepository;
import stirling.software.proprietary.security.database.repository.UserRepository;
import stirling.software.proprietary.security.model.User;
import stirling.software.proprietary.security.repository.TeamMembershipRepository;
@ExtendWith(MockitoExtension.class)
class SaasFleetUsageControllerTest {
@Mock private UserRepository userRepository;
@Mock private TeamMembershipRepository memberRepo;
@Mock private PersistentAuditEventRepository auditRepository;
@Mock private AuditConfigurationProperties auditConfig;
private SaasFleetUsageController controller;
@BeforeEach
void setUp() {
controller =
new SaasFleetUsageController(
userRepository, memberRepo, auditRepository, auditConfig);
}
/** An Authentication whose principal is a User (AuthenticationUtils returns it directly). */
private Authentication authFor(long userId) {
User user = mock(User.class);
when(user.getId()).thenReturn(userId);
Authentication auth = mock(Authentication.class);
when(auth.getPrincipal()).thenReturn(user);
return auth;
}
private TeamMembership memberOf(long teamId, String username) {
// lenient: a member used only in the roster has its team.getId() stub unused, which strict
// stubbing would otherwise flag.
Team team = mock(Team.class);
lenient().when(team.getId()).thenReturn(teamId);
User u = mock(User.class);
lenient().when(u.getUsername()).thenReturn(username);
TeamMembership m = mock(TeamMembership.class);
lenient().when(m.getTeam()).thenReturn(team);
lenient().when(m.getUser()).thenReturn(u);
return m;
}
@Test
@DisplayName("figures are scoped to the caller's team members")
void teamScopedFigures() {
Authentication auth = authFor(1L);
TeamMembership leader = memberOf(42L, "leader@acme.test");
TeamMembership bob = memberOf(42L, "bob@acme.test");
when(memberRepo.findPrimaryMembership(1L)).thenReturn(List.of(leader));
when(memberRepo.findByTeamId(42L)).thenReturn(List.of(leader, bob));
when(auditConfig.isLevelEnabled(AuditLevel.STANDARD)).thenReturn(true);
when(auditRepository.countDistinctPrincipalsBySourceExcludingTypeAndPrincipalInAfter(
eq("WEB"), eq("UI_DATA"), anyList(), any(Instant.class)))
.thenReturn(1L);
when(auditRepository.countByTypeInAndSourceAndPrincipalInAndTimestampAfter(
anyList(), eq("WEB"), anyList(), any(Instant.class)))
.thenReturn(88L);
ResponseEntity<FleetUsageStats> res = controller.fleetStats(auth);
FleetUsageStats stats = res.getBody();
assertThat(stats).isNotNull();
assertThat(stats.editorsDeployed()).isEqualTo(2L);
assertThat(stats.activeThisMonth()).isEqualTo(1L);
assertThat(stats.pdfsProcessed()).isEqualTo(88L);
verify(auditRepository)
.countByTypeInAndSourceAndPrincipalInAndTimestampAfter(
eq(List.of("PDF_PROCESS", "FILE_OPERATION")),
eq("WEB"),
eq(List.of("leader@acme.test", "bob@acme.test")),
any(Instant.class));
}
@Test
@DisplayName("audit-derived figures are null when auditing is below STANDARD")
void auditOffYieldsNulls() {
Authentication auth = authFor(1L);
TeamMembership leader = memberOf(42L, "leader@acme.test");
when(memberRepo.findPrimaryMembership(1L)).thenReturn(List.of(leader));
when(memberRepo.findByTeamId(42L)).thenReturn(List.of(leader));
when(auditConfig.isLevelEnabled(AuditLevel.STANDARD)).thenReturn(false);
FleetUsageStats stats = controller.fleetStats(auth).getBody();
assertThat(stats).isNotNull();
assertThat(stats.editorsDeployed()).isEqualTo(1L);
assertThat(stats.activeThisMonth()).isNull();
assertThat(stats.pdfsProcessed()).isNull();
verify(auditRepository, never())
.countByTypeInAndSourceAndPrincipalInAndTimestampAfter(
anyList(), any(), anyList(), any(Instant.class));
}
@Test
@DisplayName("active is clamped to deployed (a subset)")
void activeClampedToDeployed() {
Authentication auth = authFor(1L);
TeamMembership leader = memberOf(42L, "leader@acme.test");
when(memberRepo.findPrimaryMembership(1L)).thenReturn(List.of(leader));
when(memberRepo.findByTeamId(42L)).thenReturn(List.of(leader));
when(auditConfig.isLevelEnabled(AuditLevel.STANDARD)).thenReturn(true);
when(auditRepository.countDistinctPrincipalsBySourceExcludingTypeAndPrincipalInAfter(
eq("WEB"), eq("UI_DATA"), anyList(), any(Instant.class)))
.thenReturn(5L);
when(auditRepository.countByTypeInAndSourceAndPrincipalInAndTimestampAfter(
anyList(), eq("WEB"), anyList(), any(Instant.class)))
.thenReturn(10L);
FleetUsageStats stats = controller.fleetStats(auth).getBody();
assertThat(stats).isNotNull();
assertThat(stats.activeThisMonth()).isEqualTo(1L);
}
@Test
@DisplayName("a caller with no team gets an empty fleet, not a 500")
void noTeamReturnsEmpty() {
Authentication auth = authFor(1L);
when(memberRepo.findPrimaryMembership(1L)).thenReturn(List.of());
FleetUsageStats stats = controller.fleetStats(auth).getBody();
assertThat(stats).isNotNull();
assertThat(stats.editorsDeployed()).isEqualTo(0L);
assertThat(stats.activeThisMonth()).isNull();
assertThat(stats.pdfsProcessed()).isNull();
}
@Test
@DisplayName("an unauthenticated request is 401")
void unauthenticatedIs401() {
ResponseEntity<FleetUsageStats> res = controller.fleetStats(null);
assertThat(res.getStatusCode()).isEqualTo(HttpStatus.UNAUTHORIZED);
}
}
+7
View File
@@ -1 +1,8 @@
declare module "*.css" {}
// Vite `?raw` suffix imports a file's contents as a string (used in preview.tsx
// to load the English translation TOML synchronously).
declare module "*?raw" {
const content: string;
export default content;
}
+18 -4
View File
@@ -21,19 +21,33 @@ import { handlers } from "@portal/mocks/handlers";
import { configureSupabase } from "@proprietary/auth/supabase/supabaseClient";
import i18next from "i18next";
import { initReactI18next } from "react-i18next";
import { parse as parseToml } from "smol-toml";
// Load the real English copy so stories render human text, not raw keys. Bundled
// synchronously via ?raw so it's present on the very first render (no async flash).
// eslint-disable-next-line no-restricted-imports -- Storybook-only: read the public i18n asset; no @-alias covers editor/public/.
import enTranslationToml from "../editor/public/locales/en-US/translation.toml?raw";
import "@mantine/core/styles.css";
import "@core/tokens/tokens.css";
import "@core/tokens/base.css";
// Storybook-only: init react-i18next so t(key, fallback, vars) interpolates its
// English fallback (there's no backend here to load locale files). Without this,
// the default t() returns raw templates like "{{count}} people · led by {{owner}}".
// Storybook-only: init react-i18next with the real English resources parsed from
// the app's TOML, so t(key) renders the shipped copy (e.g. "No sources connected
// yet") rather than the raw key. Falls back to an empty bundle if parsing ever
// fails, so a malformed TOML can't take the whole Storybook down.
function parseEnTranslation(): Record<string, unknown> {
try {
return parseToml(enTranslationToml) as Record<string, unknown>;
} catch {
return {};
}
}
if (!i18next.isInitialized) {
void i18next.use(initReactI18next).init({
lng: "en",
fallbackLng: "en",
resources: { en: { translation: {} } },
resources: { en: { translation: parseEnTranslation() } },
interpolation: { escapeValue: false },
react: { useSuspense: false },
});
+9
View File
@@ -10,3 +10,12 @@ VITE_USERBACK_TOKEN=
# Dev-only auth bypass for localhost. Subpath comes from RUN_SUBPATH (build-time).
VITE_DEV_BYPASS_AUTH=false
# Login landing mode - soft-release flag for the processor (portal):
# dynamic = role-based landing: team leads -> processor, members -> editor,
# with a per-user override in Settings > General. Default.
# editor = every user lands on the editor after login. The processor stays
# reachable via the app switcher, but no one is auto-routed to it and
# the per-user landing setting is hidden. Soft-release escape hatch.
# Set to `editor` (here or via the build env) to hold the processor back.
VITE_LOGIN_LANDING_MODE=dynamic
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -5937,59 +5937,12 @@ keyDescription = "Paste the license key from your email"
success = "License Activated!"
successMessage = "Your license has been successfully activated. You can now close this window."
[policies]
deleteConfirmBody = "This removes the policy and its workflow. Documents already processed are not affected."
deleteConfirmTitle = "Delete {{label}} policy?"
[policies.activity]
enforced = "enforced"
enforcing = "Enforcing..."
failed = "Enforcement failed"
outputsUnavailable = "Policy outputs are no longer available to download."
partialOutputsUnavailable = "Some policy outputs are no longer available to download."
retrying = "Busy, retrying..."
runNotFound = "The enforcement run could no longer be found."
step = "step {{current}}/{{total}}"
timedOut = "Enforcement timed out before the run could finish."
[policies.catalog]
compliance = "Compliance"
ingestion = "Ingestion"
retention = "Retention"
routing = "Routing"
security = "Security"
[policies.detail]
close = "Close"
editSettings = "Edit Settings"
enforces = "Enforces"
managedByOrg = "Managed by your organization. Contact a team leader to change this policy."
noActivityDescription = "Documents will appear here once this policy runs."
noActivityTitle = "No activity yet"
onEveryUpload = "On every upload"
originalsNote = "Originals stay untouched • Enforced version saved alongside"
pause = "Pause"
recentActivity = "Recent Activity"
resume = "Resume"
retry = "Retry"
showLess = "Show less"
showMore = "Show more"
statActive = "Active"
statDataProcessed = "Data processed"
statDocsEnforced = "Docs enforced"
statusActive = "Active"
statusPaused = "Paused"
[policies.docType]
Contracts = "Contracts"
"Financial reports" = "Financial reports"
"HR records" = "HR records"
Insurance = "Insurance"
Invoices = "Invoices"
"Legal filings" = "Legal filings"
"Medical / PHI" = "Medical / PHI"
"Tax documents" = "Tax documents"
[policies.enforcement]
applying = "Applying {{names}}"
applyingProgress = "Applying {{names}} ({{done}} of {{total}})"
@@ -5999,32 +5952,10 @@ failureBody = "{{failures}} of {{total}} file(s) couldn't be processed and were
failureTitle = "Exported without full enforcement"
printPolicyAppliedBody = "This PDF was updated to meet a policy. Review the changes, then print again."
printPolicyAppliedTitle = "Policy applied before printing"
queued = "+{{count}} queued"
successTitle = "{{names}} applied"
summaryMore = "{{first}}, {{second}} and {{more}} more"
summaryTwo = "{{first}} and {{second}}"
[policies.enforcement.triggerVerb]
convert = "Enforcing before convert"
default = "Enforcing"
export = "Enforcing before export"
input = "Enforcing on import"
print = "Enforcing before print"
[policies.field]
accessLog = "Access log"
archiveAfter = "Archive after"
auditTrail = "Audit trail"
belowThreshold = "Below threshold"
destination = "Destination"
frameworks = "Frameworks"
immutableHold = "Immutable hold"
keepFor = "Keep for"
minConfidence = "Min confidence"
notify = "Notify on route"
onViolation = "When non-compliant"
webhookUrl = "Webhook URL"
[policies.fieldOption.archiveAfter]
"1 year" = "1 year"
"30 days" = "30 days"
@@ -6070,9 +6001,6 @@ Indefinite = "Indefinite"
"Flag for review" = "Flag for review"
"Quarantine document" = "Quarantine document"
[policies.fields]
selectedCount = "{{count}} selected"
[policies.labels]
add = "Add"
addPlaceholder = "Add a label…"
@@ -6111,90 +6039,6 @@ placeholder = "Select PII types"
routing = "US routing numbers (ABA)"
ssn = "Social Security numbers"
[policies.settings]
noneExport = "No policies currently run on export."
noneUpload = "No policies currently run on upload."
onExport = "On export"
onUpload = "On upload"
reorderHandle = "Drag to reorder"
runOrderDesc = "When more than one policy runs on the same trigger, they run in this order — each on the previous policy's output. Drag to reorder."
title = "Policy settings"
[policies.sidebar]
activeCount = "{{count}} active"
infoTooltip = "A policy is a fixed set of tools that runs automatically whenever it's triggered — for example when a new document arrives — enforcing rules like redacting PII with no manual steps."
loading = "Loading…"
optionsAriaLabel = "Policy options"
policySettings = "Policy settings"
railAriaLabel = "{{label}} policy — {{status}}"
railSuffixActive = " (Active)"
railSuffixPaused = " (Paused)"
retryFailed = "Retry failed policies ({{count}})"
rowProgress = "{{completed}} of {{total}} files processed"
setUp = "Set up"
title = "Policies"
upgradeToEnterprise = "Upgrade to enterprise"
whatIsPolicy = "What is a policy?"
[policies.status]
active = "Active"
paused = "Paused"
setup = "Set up"
[policies.toolConfig]
enableAriaLabel = "Enable {{tool}}"
infoAriaLabel = "What does {{tool}} do?"
[policies.toolConfig.info]
redact = "Automatically finds and blacks out sensitive details — like Social Security and card numbers — so they can't be read in the document."
sanitize = "Removes hidden JavaScript from the file, so nothing can run automatically when someone opens it."
watermark = "Stamps a visible mark (e.g. \"Confidential\") across every page."
[policies.wizard]
allDocTypesDescription = "Enable the Classification policy to filter by document type."
allDocTypesTitle = "All document types"
back = "Back"
builderDesc = "Build the sequence of tools this policy runs on each document."
clear = "Clear"
close = "Close"
continue = "Continue"
docTypesLabel = "Document types"
edit = "Edit"
editTitle = "Edit {{label}} Policy"
enablePolicy = "Enable Policy"
filenameAutoNumber = "Auto-number"
filenamePositionAria = "Filename position"
filenamePrefix = "Prefix"
filenameSuffix = "Suffix"
filenameTextAria = "Filename text"
filenameTextPlaceholder = "Text to add (optional)"
lockedDescription = "Contact a team leader to change this policy."
lockedTitle = "Managed by your organization"
maxRetriesLabel = "Max retries"
noToolsError = "Add at least one configured tool to the workflow first."
outputAsLabel = "Output as"
outputFilenameSubhead = "Output filename"
outputModeAria = "Output mode"
outputNewFile = "New file"
outputNewVersion = "New version"
outputRetriesLabel = "Output & retries"
outputSubhead = "Output"
retryDelayAria = "Retry delay minutes"
retryDelayLabel = "Retry delay (min)"
runOnExport = "Export"
runOnLabel = "Run on"
runOnSubhead = "Run on"
runOnUpload = "Upload"
saveChanges = "Save Changes"
saveError = "Couldn't save the policy. Please try again."
setupClassification = "Set up Classification"
setupTitle = "Set up {{label}} Policy"
sourcesDesc = "Choose where this policy runs and which document types it applies to."
sourcesLabel = "Sources"
stepOf = "Step {{step}} of {{total}}"
toolChainDesc = "Configure the tools this policy runs on each document."
typesSelected = "{{count}} types selected"
[policy]
badgeEnforcing = "{{name}} enforcing..."
badgeRan = "{{name}} policy ran on this file"
@@ -6406,7 +6250,7 @@ bodyBefore = "Set"
title = "Stripe not configured"
[portal.billing.enterpriseUpsell]
cta = "Build your Enterprise quote"
cta = "Explore Enterprise"
description = "Committed volume discounts, air-gapped deployment, custom MSA and security reviews, and 3rd-party distributor partnerships."
eyebrow = "Volume discount · 1M+ PDFs"
title = "Stirling Enterprise"
@@ -6471,18 +6315,23 @@ managedTitle = "Managed in Stripe"
update = "Update"
[portal.billing.pdfsProcessed]
emptyPeriod = "No metered processing yet this period."
emptyPeriod = "No processing yet this period."
eyebrow = "PDFs processed this period"
legendValue_one = "{{formatted}} PDFs"
legendValue_other = "{{formatted}} PDFs"
segbarAriaLabel = "Metered PDFs split by category"
segbarAriaLabel = "PDFs split by category"
segmentAgentsDesc = "AI agent actions"
segmentAgentsLabel = "Agents"
segmentApiDesc = "Direct API requests"
segmentApiLabel = "API"
segmentAutomationDesc = "Automations & pipelines"
segmentAutomationLabel = "Automation"
unit = "metered PDFs"
sizeMultiplier = "{{formatted}} PDFs used a size multiplier"
summary = "{{unique}} unique · {{units}} meter units · {{avg}} avg per PDF"
summaryNoRate = "{{unique}} unique · {{units}} meter units"
unit = "PDFs"
unitsPending_one = "{{units}} meter unit pending sync from linked instances"
unitsPending_other = "{{units}} meter units pending sync from linked instances"
[portal.billing.spendLimit]
adjustLimit = "Adjust limit"
@@ -6511,6 +6360,8 @@ label = "Projected to exceed."
[portal.billing.spendThisMonth]
eyebrow = "Spend this month"
freeRemaining_one = "{{formatted}} free PDF remaining"
freeRemaining_other = "{{formatted}} free PDFs remaining"
processed_one = "{{formattedCount}} PDF processed."
processed_other = "{{formattedCount}} PDFs processed."
processedWithRate_one = "{{formattedCount}} PDF processed, at {{rate}} each."
@@ -6783,6 +6634,8 @@ type = "Type"
user = "User"
[portal.documents.queue.empty]
connectSource = "Connect a source"
createPipeline = "Create a pipeline"
description = "As sources feed documents into your pipelines they'll appear here for review."
title = "No documents in the queue"
@@ -6892,8 +6745,63 @@ description = "This view hit an unexpected error. Try again, or pick another sec
retry = "Try again"
title = "Something went wrong on this page"
[portal.home.download]
back = "Back"
body = "Download the desktop app or self-host the server."
done = "Done"
guide = "Open the {{name}} guide"
openInBrowser = "Open in browser instead"
sectionDesktop = "Desktop app"
sectionSelfHosted = "Self-hosted"
title = "Install the Stirling PDF Editor"
[portal.home.download.docker]
detailBody = "Run the official image on your server."
note = "Opens at :8080. Default login: admin / stirling."
tagline = "Run the server in a container"
title = "Docker"
variantFat = "Fat"
variantLite = "Ultra-Lite"
variantStandard = "Standard"
[portal.home.download.kubernetes]
detailBody = "Install the official Helm chart."
note = "Add a PVC for /configs to persist settings."
tagline = "Deploy to a cluster with Helm"
title = "Kubernetes"
[portal.home.download.linux]
detailBody = "Native builds for your distribution."
guideBtn = "View Linux install options"
note = "AppImage, .deb and .rpm builds are available for common distros."
tagline = "Native desktop app for your distro"
title = "Linux"
[portal.home.download.mac]
altLabel = "Or via Homebrew"
detailBody = "Run the installer, or use Homebrew."
downloadBtn = "Download for macOS"
tagline = "Native app for Apple silicon and Intel"
title = "macOS"
[portal.home.download.manual]
detailBody = "Run the .jar on a bare metal server."
downloadBtn = "Download the .jar"
note = "Requires Java 21+. Opens at :8080; default login admin / stirling."
runLabel = "Then run it"
tagline = "Run the .jar on a bare metal server"
title = "Manual server setup"
[portal.home.download.windows]
altLabel = "Or via winget"
detailBody = "Run the installer, or use a package manager."
downloadBtn = "Download installer"
tagline = "Native app for Microsoft Windows"
title = "Windows"
[portal.home.editor]
activeUsers = "{{n}} active"
install = "Install the editor"
invite = "Invite teammates"
name = "Stirling PDF Editor"
open = "Open in browser"
@@ -6909,12 +6817,6 @@ afternoon = "Good afternoon"
evening = "Good evening"
morning = "Good morning"
[portal.home.onboarding]
dismiss = "Dismiss setup"
notStarted = "Not started"
progress = "{{done}} of {{total}} done"
title = "Finish setting up"
[portal.home.onboarding.enterprise]
body = "Org-wide SSO + SCIM + RBAC, committed volume pricing, and air-gapped deployment."
cta = "Start Trial"
@@ -6923,18 +6825,16 @@ lead = "For 250+ employees."
tag = "Enterprise"
[portal.home.onboarding.steps.editor]
blurb = "Deploy the desktop app, free forever."
title = "Download the PDF Editor"
blurb = "Install the desktop app or self-host"
title = "Download the editor"
[portal.home.onboarding.steps.invite]
blurb = "Bring your team into the secure workspace"
title = "Invite teammates"
[portal.home.onboarding.steps.policies]
blurb = "{{active}} active · {{recommended}} recommended"
chip = "{{active}} active"
title = "Turn on policies"
[portal.home.onboarding.steps.sources]
blurb = "{{connected}} connected · every PDF governed where it lands"
chip = "{{connected}} connected"
title = "Connect your sources"
title = "Confirm your policies"
[portal.home.quickActions]
subtitle = "Top tasks for today"
@@ -6958,6 +6858,12 @@ sectionsAriaLabel = "Infrastructure sections"
subtitle = "Deployments, credentials, security posture, storage, and the audit trail for your Stirling workspace."
title = "Infrastructure"
# Fixed-enum label maps rendered via t(MAP[value]) in the infrastructure tabs.
[portal.infrastructure.apiKeyPermission]
admin = "Admin"
read = "Read"
write = "Write"
[portal.infrastructure.apiKeys]
createKey = "Create key"
heading = "API keys"
@@ -6978,6 +6884,11 @@ usageToday = "Usage today"
description = "Create a scoped key to start calling the Stirling API."
title = "No API keys yet"
[portal.infrastructure.attestationLabel]
attested = "Attested"
inScope = "In scope"
notApplicable = "N/A"
[portal.infrastructure.audit]
filterAriaLabel = "Filter audit events by category"
heading = "Audit logs"
@@ -7034,6 +6945,24 @@ elevation = "Elevation"
processing = "Processing"
totalEvents = "Total events · 24h"
[portal.infrastructure.auditCatLabel]
auth = "Auth"
config = "Config"
elevation = "Elevation"
processing = "Processing"
security = "Security"
[portal.infrastructure.auditStatusLabel]
danger = "Error"
info = "Info"
success = "Success"
warning = "Warning"
[portal.infrastructure.certLabel]
certified = "Certified"
inProgress = "In progress"
notStarted = "Not started"
[portal.infrastructure.createKey]
cancel = "Cancel"
createKey = "Create key"
@@ -7050,6 +6979,12 @@ subtitleCreated = "Copy this secret now — it won't be shown again."
title = "Create API key"
titleCreated = "Key created"
[portal.infrastructure.deployLabel]
live = "Live"
queued = "Queued"
rolledBack = "Rolled back"
rolling = "Rolling out"
[portal.infrastructure.deployments]
msValue = "{{value}} ms"
throughputValue = "{{value}}/min"
@@ -7085,6 +7020,16 @@ subheading = "Live health for every deployed Stirling region — latency, load,
description = "Deployed regions appear here once your workspace is provisioned."
title = "No regions deployed"
[portal.infrastructure.keyLabel]
active = "Active"
revoked = "Revoked"
rotateSoon = "Rotate soon"
[portal.infrastructure.modelLabel]
active = "Active"
degraded = "Degraded"
disabled = "Disabled"
[portal.infrastructure.models]
heading = "Models"
msValue = "{{value}} ms"
@@ -7112,6 +7057,10 @@ status = "Status"
type = "Type"
version = "Version"
[portal.infrastructure.models.cost]
perCall = "{{price}}/call"
perThousand = "{{price}}/1k"
[portal.infrastructure.models.metrics]
activeModels = "Active models"
avgLatency = "Avg latency"
@@ -7135,6 +7084,17 @@ modelForAria = "Model for {{operation}}"
operation = "Operation"
routedTo = "Routed to"
[portal.infrastructure.modelTypeLabel]
classification = "Classification"
extraction = "Extraction"
llm = "LLM"
ocr = "OCR"
[portal.infrastructure.regionLabel]
degraded = "Degraded"
down = "Down"
healthy = "Healthy"
[portal.infrastructure.security.access.byok]
description = "Supply a key from your own KMS. Stirling encrypts with it but can still read."
label = "Bring your own key (BYOK)"
@@ -7276,26 +7236,6 @@ sources = "Sources"
usage = "Usage & Billing"
users = "Users"
[portal.notifications]
markAllRead = "Mark all read"
title = "Notifications"
viewAll = "View all"
[portal.notifications.ariaLabel]
none = "Notifications, no unread"
unread_one = "Notifications, {{count}} unread"
unread_other = "Notifications, {{count}} unread"
[portal.notifications.count]
allRead = "all read"
loading = "loading"
new_one = "{{count}} new"
new_other = "{{count}} new"
[portal.notifications.empty]
description = "No new notifications."
title = "You're all caught up"
[portal.pipelines]
subtitle = "Every automated document pipeline on the backend: an ordered chain of operations over a set of sources, run on a trigger. Click a row for its steps and sources."
title = "Pipelines"
@@ -7340,6 +7280,11 @@ operations_one = "Operation ({{count}})"
operations_other = "Operations ({{count}})"
output = "Output"
removeStep = "Remove operation"
s3Configure = "Configure"
s3Done = "Done"
s3ModalTitle = "Amazon S3 output"
s3NotConfigured = "Not configured"
s3PrefixHelp = "Outputs are uploaded under this key prefix."
save = "Save changes"
scheduleEvery = "Run every"
sources = "Sources"
@@ -7359,11 +7304,13 @@ confirm = "Delete"
title = "Delete pipeline?"
[portal.pipelines.detail]
clearHistory = "Clear history"
delete = "Delete pipeline"
run = "Run now"
[portal.pipelines.empty]
action = "Create a pipeline"
connectSource = "Connect a source"
description = "Create your first pipeline: pick the sources it runs over, chain the operations, and choose where output goes."
title = "No pipelines yet"
@@ -7375,12 +7322,19 @@ total = "Pipelines"
[portal.pipelines.output]
folder = "Write to folder"
inline = "Return files"
s3 = "Write to Amazon S3"
[portal.pipelines.run]
allProcessed_one = "Nothing to run: the source's {{count}} document has already been processed."
allProcessed_other = "Nothing to run: all {{count}} documents in the sources have already been processed."
completed_one = "Run completed."
completed_other = "All {{count}} runs completed."
empty = "Nothing to run: the sources had no documents to process."
failed = "Run failed: {{error}}"
historyCleared = "History cleared. The next run reprocesses everything currently in the sources."
inFlight = "Nothing new to run: documents are still being processed from an earlier run."
parked_one = "Nothing to run: {{count}} document failed previously and is parked. Fix the cause, then clear history to retry it."
parked_other = "Nothing to run: {{count}} documents failed previously and are parked. Fix the cause, then clear history to retry them."
running = "Run started; still in progress."
timeout = "Run is taking longer than expected; it may still finish in the background."
@@ -7664,7 +7618,6 @@ managePlan = "Manage plan"
volumeSuffix = "PDFs processed · last 30 days"
[portal.procurement]
reset = "Reset procurement (demo)"
subtitle = "Get your team evaluated, contracted, and onboarded. Every document in one place."
title = "Procurement"
@@ -7684,23 +7637,42 @@ title = "Review your enterprise agreement"
[portal.procurement.builder]
addons = "Add-ons"
addressLine1 = "Address line 1"
addressLine1Placeholder = "500 Howard St"
addressLine2 = "Address line 2"
addressLine2Placeholder = "Suite, floor, building"
back = "Back"
businessName = "Business name"
businessNamePlaceholder = "Your company"
city = "City"
cityPlaceholder = "San Francisco"
contactEmail = "Contact email"
contactEmailPlaceholder = "jane@acme.com"
contactName = "Contact name"
contactNamePlaceholder = "Jane Doe"
continue = "Continue"
country = "Country"
countryEuro = "Eurozone (EUR €)"
countryUK = "United Kingdom (GBP £)"
countryUS = "United States (USD $)"
eula = "I have read and agree to the Stirling Enterprise EULA. It governs the agreement generated from this quote."
generate = "Generate quote"
included = "Included"
indemnification = "IP indemnification"
indemnificationSub = "We defend qualifying IP claims, per the EULA"
offlineLicense = "Offline / air-gapped licence"
offlineLicenseSub = "A downloadable licence file for an air-gapped self-hosted instance"
pdfSize = "PDF size"
poNumber = "PO number"
poNumberPlaceholder = "Optional"
postalCode = "Postal code"
postalCodePlaceholder = "94105"
posture = "Governance"
posture_count = "~{{count}} policies"
posture_essentials = "Essentials"
posture_essentialsSub = "Classification + Sharing — the defaults"
posture_governed = "Governed"
posture_governedSub = "Adds Security and Routing"
posture_regulated = "Regulated"
posture_regulatedSub = "Every category — Compliance, Retention, Ingestion"
qbr = "Quarterly business reviews"
qbrSub = "Your SE reviews usage and roadmap each quarter"
region = "State / region"
regionPlaceholder = "California"
running = "{{annual}} / yr · {{years}}-yr {{tcv}}"
s1Sub = "Your team, and the PDFs you expect to run each year."
# Step 1 — volume
@@ -7712,13 +7684,21 @@ s3Sub = "For the quote and the agreement it generates."
# Step 3 — details
s3Title = "Your details"
serviceLevel = "Service level"
size_compact = "Compact"
size_compactSub = "Mostly text, under 1 MB"
size_heavy = "Heavy"
size_heavySub = "Scanned or image-heavy, 5 MB+"
size_standard = "Standard"
size_standardSub = "Mixed text and images, 1 to 5 MB"
slDedicated = "Dedicated"
slDedicatedSub = "4 business hours · dedicated account manager · +30%"
slDedicatedSub = "4 business hours · dedicated SE / CSM · +$30,000/yr"
slPriority = "Priority"
slPrioritySub = "Same business day · named CSM · +15%"
slPrioritySub = "Same business day · named CSM · included"
slStandard = "Standard"
slStandardSub = "Next business day · shared CSM · included"
stepOf = "Step {{n}} of {{total}}"
taxId = "VAT / Tax ID"
taxIdPlaceholder = "Optional"
term = "Term"
termDiscount = "{{pct}}% multi-year commitment discount applied"
title = "Build your quote"
@@ -7754,14 +7734,13 @@ title = "Something went wrong"
[portal.procurement.hero]
company = "Your enterprise deal"
ctaAgreement = "Review & sign agreement"
ctaLive = "You're live"
ctaPayment = "Add payment"
ctaQuote = "Review your quote"
ctaTrial = "Build your quote"
eyebrow = "Enterprise procurement"
inviteTeammates = "Invite teammates"
keyDocs = "Key documents"
licenseKey = "Licence key"
nextStep = "Next step: {{action}}"
notStarted = "Not started"
open = "Open procurement"
@@ -7816,6 +7795,9 @@ downloadError = "Could not generate the offline licence file just yet — please
downloadOffline = "Download offline licence (.lic)"
hint = "Paste this key into a self-hosted instance to activate it, or keep it for your records. Keep it safe."
label = "Your licence key"
subtitle = "Activate a self-hosted instance with this key, or keep it for your records."
title = "Your licence key"
trialFileHint = "This is your trial licence file. Once your agreement is in place, come back and download it again — unlike the online licence key, the .lic file won't update on its own."
[portal.procurement.link]
cta = "Link account"
@@ -7835,16 +7817,9 @@ talkToSales = "Talk to sales"
title = "The procurement track opens with Enterprise"
[portal.procurement.milestone]
accept = "Accept & continue"
description = "Download the PDF to share it with your team, come back to accept when you're ready, or make changes."
download = "Download PDF"
downloadError = "Could not download the quote PDF just yet — please try again in a moment."
edit = "Edit quote"
eyebrow = "Quote {{number}}"
perYear = " / yr"
preparedFor = "Prepared for {{company}}"
tcv = "{{value}} total contract value"
title = "Your quote is ready"
[portal.procurement.modal]
cancel = "Cancel"
@@ -7871,7 +7846,6 @@ uploadTitle = "Upload your purchase order"
[portal.procurement.payment]
description = "Your quote is accepted and your licence is already active — your team can start right away. Pay the first invoice when you're ready; you can pay or download it here, no email needed."
downloadInvoice = "Download invoice"
simulate = "Simulate payment received (demo)"
title = "Subscription created"
viewInvoice = "View & pay invoice"
@@ -7881,6 +7855,21 @@ fallbackLink = "Open scheduling in a new tab"
subtitle = "Your solutions engineer will walk your team through the rollout. Pick a time that suits you."
title = "Schedule a call"
[portal.procurement.setup]
airgap = "Air-gapped"
airgapSub = "Fully offline, isolated network. Includes a downloadable licence file."
cloud = "Cloud"
cloudSub = "Fully managed by Stirling. Nothing for you to run."
deployment = "Where will you run Stirling?"
seats = "Team size"
seatsHint = "Roughly how many people will use it. You can refine this when you build your quote."
seatsPlaceholder = "e.g. 250"
selfhost = "Self-hosted"
selfhostSub = "Run it in your own cloud or data centre."
start = "Start trial"
subtitle = "Tell us how you plan to run Stirling so we can tailor your trial and quote. No card required."
title = "Set up your trial"
[portal.procurement.status]
action = "Action needed"
available = "Available"
@@ -7927,26 +7916,11 @@ admin = "Admin"
[portal.settings.sections]
account-link = "Account link"
[portal.shell.header]
accountFallback = "Account"
accountMenu = "Account menu"
darkMode = "Dark mode"
lightMode = "Light mode"
search = "Search"
searchPlaceholder = "Search…"
signOut = "Sign out"
switchToDark = "Switch to dark theme"
switchToLight = "Switch to light theme"
[portal.shell.sidebar]
appEditor = "Editor"
appProcessor = "Processor"
brandSuffix = "Stirling Processor"
docsCount = "{{docs}} docs"
linkAccount = "Link Stirling account"
planEditor = "Editor plan"
planEnterprise = "Enterprise plan"
planProcessor = "Processor plan"
primaryNav = "Primary navigation"
switchApp = "Switch app"
@@ -7999,7 +7973,9 @@ status = "Status"
usedBy = "Policies"
[portal.sources.types.editor]
description = "Documents your team has processed in the editor, across policy and AI runs."
label = "Editor"
noPolicies = "No policies run from the editor yet."
[portal.sources.types.folder]
description = "Watch a directory on the server for new documents."
@@ -8032,6 +8008,42 @@ label = "Folder depth"
all = "Include subfolders"
top = "Top level only"
[portal.sources.types.s3]
description = "Pull documents from an Amazon S3 or S3-compatible bucket."
label = "Amazon S3"
[portal.sources.types.s3.fields.accessKeyId]
label = "Access key ID"
[portal.sources.types.s3.fields.bucket]
label = "Bucket"
placeholder = "my-company-inbox"
[portal.sources.types.s3.fields.endpoint]
helperText = "Leave blank for Amazon S3. Set to use an S3-compatible service such as MinIO."
label = "Custom endpoint"
placeholder = "https://s3.example.com"
[portal.sources.types.s3.fields.mode]
helperText = "Consume removes each object from the bucket once every policy has processed it."
label = "Read mode"
[portal.sources.types.s3.fields.mode.options]
consume = "Consume: process each object once"
snapshot = "Snapshot: re-read the bucket every run"
[portal.sources.types.s3.fields.prefix]
helperText = "Only objects whose keys start with this prefix are processed."
label = "Key prefix"
placeholder = "incoming/"
[portal.sources.types.s3.fields.region]
label = "Region"
placeholder = "us-east-1"
[portal.sources.types.s3.fields.secretAccessKey]
label = "Secret access key"
[portal.sources.types.unknown]
label = "Source"
@@ -8114,16 +8126,17 @@ summary = "Owns a team — manages its members' resources and shared configs."
2 = "Portal access via the default policy"
3 = "Everything Member can do"
[portal.users.seats]
limited = "{{used}} / {{limit}}"
unlimited = "{{used}} · Unlimited"
[portal.welcome]
ariaLabel = "Welcome to Stirling PDF"
badge = "Open-source"
installEditor = "Install the Editor"
inviteTeammates = "Invite teammates"
install = "Install the editor"
invite = "Invite teammates"
openInBrowser = "Open in browser"
perks = "Free forever · Self-hostable"
subtitle = "The world's most secure PDF Editor is free for teams of all sizes. Includes 60+ PDF operations and SSO."
title = "Welcome to"
titleAccent = "Stirling PDF"
productName = "PDF Editor"
stats = "30M downloads · 60+ PDF operations · Free forever"
[printFile]
title = "Print File"
@@ -8926,6 +8939,12 @@ intro = "Enable user authentication, team management, and workspace features for
learnMore = "Learn more in documentation"
title = "For System Administrators"
[settings.general.loginLanding]
description = "Choose which app opens when you sign in to Stirling Cloud."
editor = "Editor"
processor = "Processor"
title = "After signing in"
[settings.general.mode]
fullscreen = "Fullscreen"
sidebar = "Sidebar"
@@ -10043,11 +10062,13 @@ title = "Users"
you = "(you)"
[users.action]
cancelInvite = "Cancel invitation"
deleteTeam = "Delete team"
disableMfa = "Reset MFA"
move = "Move to team"
reinstate = "Reinstate"
remove = "Remove from org"
removeTeam = "Remove from team"
rename = "Rename team"
resetPw = "Reset password"
suspend = "Suspend"
@@ -10060,11 +10081,14 @@ editor = "Editor"
processor = "Processor"
[users.confirm]
cancelInviteBody = "Cancel the invitation to {{email}}? They won't be able to join with the current link."
cancelInviteTitle = "Cancel invitation"
deleteTeamBody = "Delete the {{name}} team? The team must be empty first - move its members to another team, and it can't still own any integration configs."
deleteTeamTitle = "Delete team"
disableMfaBody = "Remove {{name}}'s MFA enrolment? They'll set it up again on next login if required."
disableMfaTitle = "Reset MFA"
removeBody = "Permanently remove {{name}} from the organization? This cannot be undone."
removeTeamBody = "Remove {{name}} from the team? They keep their account but lose access to this team's resources."
removeTitle = "Remove member"
[users.empty]
@@ -10115,6 +10139,14 @@ username = "Username"
usernameError = "Username must be at least 3 characters"
usernamePlaceholder = "jsmith"
[users.invites]
by = "Invited by {{who}}"
cancel = "Cancel"
count = "{{count}} pending"
desc = "Invited people who haven't joined yet. They hold a seat until they accept."
expires = "Expires"
title = "Pending invitations"
[users.loadError]
description = "Something went wrong reaching the backend, or you don't have access. Try again."
title = "Couldn't load members"
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More