Compare commits

..
Author SHA1 Message Date
James Brunton 9cd461cc08 Fail Playwright tests on warnings/errors being thrown to the console 2026-05-29 17:03:25 +01:00
James Brunton 9c8be25ff0 Fail Playwright tests on warnings/errors being thrown to the console 2026-05-29 16:28:00 +01:00
ConnorYohandGitHub 83ea07ed6a saas: DocumentClassifier + PAYG data model (#6460)
# Description of Changes

Two layers — the `DocumentClassifier` utility plus the full data model
for the new billing engine. Nothing wires the entities into application
behaviour yet; services and controllers land in follow-up PRs.

**Companion PR:**
[Stirling-PDF-SaaS#296](https://github.com/Stirling-Tools/Stirling-PDF-SaaS/pull/296)
— Supabase migration for the v3 dev branch, schema-equivalent to the
Flyway migration in this PR.

## 1. DocumentClassifier (under `payg.docs`)

`DocumentClassifier` computes the doc-unit cost of an uploaded file (or
multi-file input) under a `PricingPolicy`. PDFs read page count via
`stirling.software.jpdfium.PdfDocument`; non-PDFs are bytes-only.
Formula: `max(ceil(pages / docPagesPerUnit), ceil(bytes /
docBytesPerUnit))` clamped to `[1, fileUnitCap]`. Multi-file is the sum
of raw per-file units capped at `fileUnitCap × file_count`.

Two floors, by design: the classifier returns `docUnits` with an
absolute `1` floor for non-empty input; the policy-level
`minChargeUnits` is intentionally applied later, at process-open time in
`JobChargeService`, per design § 3.4 (`unitsForProcess =
max(policy.min_charge_units, docUnits)`). Documented in the interface +
impl javadoc.

Upload bytes are materialised through
`TempFileManager.createManagedTempFile` so jpdfium gets a `Path`; the
temp file auto-deletes on close.

Twelve tests, all in-memory fixtures generated with PDFBox at test time
— no committed binary blobs.

## 2. PAYG data model (under `payg.*`)

JPA entities, repositories, and a Flyway migration covering the full
schema in §6 of the design.

**Enums** (`payg.model`):

`JobSource`, `ProcessType`, `JobStatus`, `JobStepStatus`,
`ArtifactKind`, `LedgerEntryType`, `LedgerBucket`, `ReferenceType`,
`EntitlementState`, `FeatureSet`, `FeatureGate`, `WalletEngine`,
`CapPeriod`, `AutoGroupStrategy`.

**Entities + repositories:**

| Entity | Table | Notes |
|---|---|---|
| `PricingPolicy` | `pricing_policy` | Promoted from a record.
`stepLimits` is `Map<JobSource, Integer>` persisted via normalised child
table `pricing_policy_step_limit`. `stripePriceIds` is `Set<String>`
persisted via `pricing_policy_stripe_price` — currency comes from
`stripe.prices` via Sync Engine, not stored locally. |
| `ProcessingJob` | `processing_job` | UUID PK. Tracks lineage window
via `step_count` and `last_step_at`. |
| `ProcessingJobStep` | `processing_job_step` | Per-tool-call audit. |
| `JobArtifactHash` | `job_artifact_hash` | Composite key `(job_id,
content_hash, kind)`. `content_hash VARCHAR(128)` so multiple signature
schemes coexist as `"type:value"` storage keys. Lineage detector queries
this. |
| `WalletLedgerEntry` | `wallet_ledger` | Append-only, signed
`amount_units`. Two unique indexes kill double-posting. |
| `WalletPolicy` | `wallet_policy` | Per-team engine + cap + degradation
rules + lineage strategy. No `@Version` — admin-only writes (documented
in javadoc). |
| `WalletEntitlementSnapshot` | `wallet_entitlement_snapshot` |
Composite key `(team_id, user_id)`; `user_id = 0` is the team-wide
sentinel. No `@Version` — full-row recompute via
`EntitlementService.recompute` (documented in javadoc). |
| `PaygShadowCharge` | `payg_shadow_charge` | Per-job diff while in
`PAYG_SHADOW` engine mode. |
| `PaygTeamExtensions` | `payg_team_extensions` | Sidecar 1:1 with
`teams` carrying `pricing_policy_id` (per-team override) +
`stripe_customer_id`. Sidecar pattern (mirrors `saas_team_extensions`)
so OSS Hibernate ddl-auto never sees PAYG columns on `teams`. |

**Column adds:**

- `team_memberships.cap_units` (optional per-member sub-cap)

**Width split (intentional, documented in V11):** per-row deltas
(`wallet_ledger.amount_units`, `processing_job.charged_units`) are
`INTEGER` because no single charge realistically approaches 2B units.
Cap and period-rollup columns (`team_memberships.cap_units`,
`wallet_policy.cap_units`,
`wallet_entitlement_snapshot.period_spend_units / period_cap_units`) are
`BIGINT` because they accumulate across a billing period and admins may
legitimately set headroom-cap values into the millions.

**JPA wiring:** `SaasJpaConfig` was updated to include
`stirling.software.saas.payg.repository` in
`@EnableJpaRepositories.basePackages` and `stirling.software.saas.payg`
in `@EntityScan` (covers `payg.policy` / `payg.job` / `payg.wallet` /
`payg.entitlement` / `payg.shadow` recursively). New
`SaasJpaConfigScanTest` reads the annotations reflectively and asserts
every expected package is wired — catches the next time someone adds a
new sub-package without updating the scan paths.

**Migration:** `V11__saas_payg_model.sql` (purely additive).
Schema-equivalent to the Supabase migration in the companion PR —
including the `VARCHAR(128) content_hash` width that's needed for the
multi-signature-scheme storage encoding the lineage layer uses.

## 3. Smoke tests

`PaygEntitiesSmokeTest` exercises each entity via the no-arg ctor JPA
requires, plus getter/setter round-trips and composite-key equality —
catches Lombok/annotation regressions without needing a database.
Real-DB integration coverage lands alongside the services that consume
each entity.

## Why this is safe to land now

- All schema changes are additive — no existing rows modified, no
columns dropped.
- The entities are not yet referenced from any production code path;
they exist for the next PRs to build on.
- The v3 Supabase dev branch picks up the schema via the companion PR;
the main repo's Flyway migration applies the same shape when an instance
boots against a freshly-migrated v3 database.

## Open decisions made

- **Step-limits keyed by `JobSource`** rather than by `ProcessType`.
Captures the "self-hosted gets a different knob" framing in earlier
feedback. Trivially overridable per pricing policy version.
- **Step limits + Stripe price IDs normalised into child tables** rather
than JSONB on `pricing_policy` (per Connor's review on #296). Typed
columns, queryable directly, no JSON parsing.
- **Currency dropped from `pricing_policy_stripe_price`** — it lives on
`stripe.prices.currency` and is resolved via Sync Engine. App is
currency-blind.

## Rollback

Straight `git revert` on this PR. The Supabase migration in #296 is
additive and can be left in place safely — the running app ignores
tables it doesn't reference.

---

## Checklist

- [x] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [x] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
- [x] I have performed a self-review of my own code
- [x] My changes generate no new warnings
- [x] I have run `task check` (via `./gradlew :saas:test` with
`ENABLE_SAAS=true`) — passes
2026-05-29 12:03:01 +00:00
James BruntonandGitHub 61ebe977d3 Auto-delete CI linting comments on success (#6465)
# Description of Changes
Set CI backend & engine comments to auto-delete once the CI has passed. 

Also redesign the engine CI to call `task engine:check` like it should
have been, and make it post a comment when the tool models need to be
updated.

Also makes the comment wording more consistent between the three
languages.
2026-05-29 10:13:12 +00:00
EthanHealy01andGitHub 763595a5a3 feat: add Agents UI to proprietary right sidebar (#6454)
Update UI to include agents

Run `task dev:all` to test
2026-05-28 17:26:23 +00:00
Anthony StirlingandGitHub 398617391b Fix SSO auto-login and custom metadata settings not persisting on restart (#6468) 2026-05-28 17:39:05 +01:00
ConnorYohandGitHub a0e0e88f07 saas: harden CreditService Stripe ordering + lint @AutoJobPostMapping weights (#6458)
# Description of Changes

Two narrowly-scoped hardening changes to the credits engine.

## 1. CreditService — move Stripe meter call to `afterCommit`

The Stripe metered-usage call sits inside the surrounding
`@Transactional`, holding the `user_credits` row lock for the duration
of an HTTP round-trip to Supabase. Under load this starves concurrent
debits; a transient Stripe blip rolls back a (correct) free-credit
consumption and forces the caller to retry.

The Stripe call now runs in a `TransactionSynchronization.afterCommit`
hook — DB commits first, Stripe fires immediately after. If Stripe fails
after commit, we log + increment a new `credits.stripe_report.failures`
counter; the idempotency key is stable, so a manual replay recovers
without double-charging.

Applied to both `consumeCreditBySupabaseId` and
`consumeCreditWithWaterfall`.

**Dead-code removed:**
- Unreachable UUID fallback for MDC `requestId` — `CorrelationIdFilter`
already guarantees the key on every request.
- The `"Unable to report usage to Stripe"` `RuntimeException` and its
catch block — the afterCommit refactor eliminates the throw path.
- `StripeRollbackOnFailureTest` — pinned the rollback-on-Stripe-fail
behaviour this refactor replaces.

## 2. `@AutoJobPostMapping` — build-time lint for `resourceWeight`

`UnifiedCreditInterceptor` multiplies `resourceWeight` into the per-call
charge. An endpoint that falls through to the annotation default
produces a charge derived from a value nobody chose.

- Annotation default flipped from `1` to `Integer.MIN_VALUE` (sentinel).
Both runtime readers (`UnifiedCreditInterceptor`, `AutoJobAspect`)
already clamp into `[1, 100]` so behaviour is unchanged.
- New `AutoJobPostMappingWeightTest` scans the classpath and fails the
build if any method leaves the sentinel.
- Initial run caught 11 endpoints relying on the default. Explicit
weights now declared, chosen by comparing to peer endpoints:
  - `EditTextController` — LARGE
  - `EmailController#sendEmailWithAttachment` — SMALL
  - `ConvertPDFToMarkdown` — MEDIUM
  - `AttachmentController` (extract/list/rename/delete) — SMALL × 4
  - `ConvertImgPDFController` (cbr/cbz ↔ pdf) — MEDIUM × 2, LARGE × 2

## Tests

- `StripeUsageIdempotencyKeyTest` — pins the `(supabaseId, overage,
requestId)` idempotency key shape so Stripe always dedupes a retry.
- `StripeAfterCommitOrderingTest` — pins that `afterCommit` fires after
commit and NOT on rollback.
- `AutoJobPostMappingWeightTest` — the lint itself, plus a self-check
that the classpath scan finds at least 10 `@AutoJobPostMapping` methods
(guards against the lint passing vacuously).

Build verified: `ENABLE_SAAS=true ./gradlew :stirling-pdf:test
:saas:test`.

---

## Checklist

### General

- [x] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [x] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable) — no translation changes
- [x] I have performed a self-review of my own code
- [x] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
— internal-billing change, no public docs impact
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
— N/A

### Translations (if applicable)

- [ ] Not applicable

### UI Changes (if applicable)

- [ ] Not applicable

### Testing (if applicable)

- [x] I have run `task check` (via `./gradlew :stirling-pdf:test
:saas:test` with `ENABLE_SAAS=true`) — passes
- [x] I have tested my changes locally
2026-05-28 14:57:59 +00:00
Anthony StirlingandGitHub c80a5db5f5 folder and file fixes (#6461) 2026-05-28 15:57:35 +01:00
Anthony StirlingandGitHub 4fa67afc3d Fix Tauri artifact copy path so installers upload (smoke + release) (#6466)
## Summary
Regression from #6404 (Restructure/frontend editor). Two CI workflows
copy the built installers to the wrong directory, so installer artifacts
(MSI / DMG / DEB / RPM / AppImage) silently vanish:

- **`tauri-build.yml`** (PR/desktop smoke builds) - uploads zero
installer artifacts.
- **`multiOSReleases.yml`** (production releases) - the empty artifacts
are downloaded by `create-release` and fed to `action-gh-release`, so a
release would publish **only the JARs, no desktop installers**.

## Root cause
#6404 moved the Tauri project from `frontend/` to `frontend/editor/` and
updated every **absolute** path (`projectPath`, `cd`, `Get-ChildItem`)
to add the `editor/` segment - but left the **relative** copy targets
`../../../dist`. Those resolve against the (now one level deeper)
working dir after `cd ./frontend/editor/src-tauri/target`:

| | resolves to |
|---|---|
| before #6404 (`frontend/src-tauri/target`) | repo-root `dist/` ✅ |
| after #6404 (`frontend/editor/src-tauri/target`) | `frontend/dist/` ❌
(missing) |

The `cp` fails, repo-root `dist/` (from `mkdir -p ./dist`) stays empty,
and the upload finds nothing. `find -exec cp` failing is non-fatal, so
jobs still report success - that's why it went unnoticed. No release has
shipped broken yet: the last release (v2.11.0, 2026-05-19) predates
#6404 (2026-05-22).

## Fix
Copy to an absolute `$GITHUB_WORKSPACE/dist` in both workflows so the
`cd` can't drift the destination again. This matches where the upload /
signature-verify steps already read from.

## Evidence (run 26574078559, all 3 OS legs)
```
cp: cannot create regular file '../../../dist/Stirling-PDF-windows-x86_64.msi': No such file or directory
##[warning]No files were found with the provided path: ./dist/*. No artifacts will be uploaded.
```
The Tauri builds themselves succeeded - only the copy/upload was broken.

## Test plan
- [ ] `tauri-build` on this PR uploads non-empty `Stirling-PDF-<name>`
artifacts on Windows/macOS/Linux.
- [ ] Next release (or a `workflow_dispatch` of multiOSReleases)
attaches MSI/DMG/DEB/RPM/AppImage to the release.
2026-05-28 15:57:01 +01:00
8bd78d2624 Add landscape page size options (#6248)
# Description of Changes

Adds orientation (portrait/landscape) to the Adjust Page Scale tool.

- Orientation as a separate parameter (per review), sent through to the
backend
- ScalePagesController simplified; PDFWithPageSize gains the orientation
field
- Regenerated tool_models.py; frontend + backend tests added

---

## Checklist

### General

- [ ] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [ ] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable)
- [ ] I have performed a self-review of my own code
- [ ] My changes generate no new warnings

### Documentation

- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
(if functionality has heavily changed)
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
(for new translation tags only)

### Translations (if applicable)

- [ ] I ran
[`scripts/counter_translation.py`](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/docs/counter_translation.md)

### UI Changes (if applicable)

- [ ] Screenshots or videos demonstrating the UI changes are attached
(e.g., as comments or direct attachments in the PR)

### Testing (if applicable)

- [ ] I have run `task check` to verify linters, typechecks, and tests
pass
- [ ] I have tested my changes locally. Refer to the [Testing
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md#7-testing)
for more details.

---------

Co-authored-by: ConnorYoh <40631091+ConnorYoh@users.noreply.github.com>
2026-05-28 14:16:15 +00:00
Anthony StirlingandGitHub b3c4b8b463 Add S3 storage and cluster artifact backend (#6457) 2026-05-28 13:06:27 +01:00
James BruntonandGitHub 57af5b9dc2 Fix Tauri testing (#6462)
# Description of Changes
#6402 introduced a Rust test `refresh_token_fallback.rs`, but it wasn't
moved properly after the restructure of the `frontend/` folder in #6404.
This PR moves the file to the right place, and also hooks up Task and CI
rules for `cargo test` since nothing was actually running the test in
the first place.
2026-05-28 11:05:56 +00:00
James BruntonandGitHub 44fbf8c587 Various bug fixes found while testing SaaS build (#6459)
# Description of Changes
Various fixes and improvements I made while testing the SaaS code:
- Changes the new `.env.saas` file to live in `app/` and match the
semantics of the other `.env` files
- Adds top-level `task dev:saas` command to spawn SaaS frontend &
backend
- Deletes dead SaaS code and improves some overriding logic
- Fixes refreshing issue when coming back to the tab
- Fix the Compare tool's selection logic
- Make Compare handle error cases properly
- Fixes the location of the "Dismiss All Errors" button (was rendering
on top of the top-bar with a transparent background previously so it
looked rubbish)
- Fixes file selection in PDF Editor
2026-05-28 11:05:30 +00:00
Anthony StirlingandGitHub 76840d8a57 Add CI DB migration smoke test against v2.0/v2.5/v2.10 updates (#6453) 2026-05-28 11:36:07 +01:00
James BruntonandGitHub d459ded168 Add cancel button to kill long-running AI tasks (#6351)
# Description of Changes
Adds a cancel button to the AI chat to allow the user to abort
long-running AI tasks. Just disconnects the SSE stream (all the backend
code already interrupts when it notices the stream is dead).
2026-05-28 09:25:23 +00:00
ConnorYohandGitHub 43b67d213d feat(oauth2): opt-in claim-dump diagnostics for OIDC login failures (#6456)
# Description of Changes

## What & why

Customers using ADFS (or any generic OIDC provider that doesn't emit
`email`) hit `Attribute value for 'email' cannot be null` during OAuth2
login with no visibility into what claims the provider actually sent.
The only available remedy was guessing at
`security.oauth2.useAsUsername` until something worked.

This PR adds a new opt-in `security.oauth2.debugLogging` flag (default
`false`). When enabled, `CustomOAuth2UserService` logs:

- All ID token claims (sorted, with values)
- All UserInfo endpoint claims (if any)
- The merged attribute key set Spring exposes to `getAttribute()`
- The value the configured `useAsUsername` actually resolved to
- A **`Hint:`** line listing the claim keys present in the token that
map to a valid `UsernameAttribute` enum value — i.e. exactly what the
operator could put in `useAsUsername` to make login work

Logged at `INFO` on the success path and `ERROR` on failure (inside the
existing `catch (IllegalArgumentException)` block that throws
`OAuth2AuthenticationException`). The block is wrapped with a `[OAUTH2
DEBUG] ... [/OAUTH2 DEBUG]` banner and ends with a PII warning so
operators don't leave it on in production.

Default off → zero observable change for anyone not actively
troubleshooting.

## Files changed

| File | Why |
|---|---|
| `app/common/.../ApplicationProperties.java` | New `debugLogging` field
on the `OAUTH2` config class with javadoc warning about PII |
| `app/core/src/main/resources/settings.yml.template` | Documents
`oauth2.debugLogging` so it appears on next startup |
| `app/proprietary/.../security/service/CustomOAuth2UserService.java` |
Emits the claim dump + suggestion hint when the flag is on |
|
`app/proprietary/.../security/service/CustomOAuth2UserServiceDebugLoggingTest.java`
(new) | Unit test: mocks the OIDC delegate, asserts off-path is silent
and on-path emits the dump with the right Hint contents |

## End-to-end verification

Ran the bundled `testing/compose/docker-compose-keycloak-oauth.yml`
Keycloak realm, configured `security.oauth2.useAsUsername: mail`
(Keycloak emits `email`, not `mail`) and `provider: demarest` (matches
the original customer bug report). Triggered the OAuth flow at
`http://localhost:8080/oauth2/authorization/demarest` and confirmed:

- The ERROR-level dump fires with the full 19-claim ID token decoded
- `-- Value at 'mail' : <NULL — this is why login fails>` correctly
identifies the missing claim
- `-- Hint:` correctly suggests `[email, family_name, given_name,
preferred_username]` (the four keys present that map to valid
`UsernameAttribute` values)
- Auth still fails with the original `OAuth2AuthenticationException` —
no change to control flow, just added diagnostic logging

Unit test (`CustomOAuth2UserServiceDebugLoggingTest`) covers both
branches.

## Reviewer notes

- **No new public APIs.** The flag is config-only; no servlet endpoints
exposed.
- **PII is logged when the flag is on.** This is the whole point —
operators need to see the claims to fix their config — but it's gated,
defaults off, and the dump self-documents with a `WARNING: ... Set
security.oauth2.debugLogging=false once troubleshooting is complete.`
footer.
- **Why log everything, not just sub/email?** Because the operator
doesn't know in advance which claim they actually want. ADFS uses `upn`
in some configs and `preferred_username` in others; Azure AD uses `oid`;
the customer here had neither. Dumping the full set is the only way to
make the diagnostic self-service.
- **Out of scope for this PR (follow-ups):**
- The `UsernameAttribute` enum doesn't include `upn` / `unique_name`
(common ADFS claims). If the customer's token only has `upn`, the Hint
will be empty even though the operator can see `upn` in the dump. Worth
a separate PR to extend the enum.
- The known-provider validator in `Provider.java` (rejects e.g.
`useAsUsername: mail` for `provider: keycloak` at startup) bypasses our
diagnostic for those provider names. ADFS customers using `provider:
<name>` fall into the `default` branch so are not affected — but it's a
sharp edge worth documenting.

---

## Checklist

### General

- [x] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [x] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable) — N/A, backend-only change
- [x] I have performed a self-review of my own code
- [x] My changes generate no new warnings

### Documentation

- [ ] Doc-repo update (if functionality has heavily changed) —
diagnostic flag is self-documenting via the `settings.yml.template`
comment and the in-log warning; happy to add a doc-repo entry if
reviewers want one
- [ ] Translation tags — N/A

### UI Changes (if applicable)

- [ ] N/A — backend-only

### Testing (if applicable)

- [x] Unit test added (`CustomOAuth2UserServiceDebugLoggingTest`)
covering on/off paths and Hint correctness
- [x] End-to-end verified locally against bundled Keycloak compose with
intentionally misconfigured `useAsUsername`
- [x] Full `:proprietary:test` suite passes
2026-05-27 13:01:51 +00:00
Anthony StirlingandGitHub d42b779644 Add server-side folders and files page UI (#6383) 2026-05-27 12:52:46 +01:00
320 changed files with 26391 additions and 7334 deletions
+2
View File
@@ -38,6 +38,8 @@ project: &project
- frontend/**
- docker/**
- scripts/RestartHelper.java
- scripts/db-migration/**
- .github/workflows/db-migration-test.yml
frontend: &frontend
- frontend/**
+2 -2
View File
@@ -13,7 +13,7 @@ Usage:
"""
# Sample for Windows:
# python .github/scripts/check_language_toml.py --reference-file frontend/public/locales/en-GB/translation.toml --branch "" --files frontend/public/locales/de-DE/translation.toml frontend/public/locales/fr-FR/translation.toml
# python .github/scripts/check_language_toml.py --reference-file frontend/editor/public/locales/en-GB/translation.toml --branch "" --files frontend/editor/public/locales/de-DE/translation.toml frontend/editor/public/locales/fr-FR/translation.toml
import argparse
import glob
@@ -308,7 +308,7 @@ def check_for_differences(reference_file, file_list, branch, actor):
report.append("## ❌ Overall Check Status: **_Failed_**")
report.append("")
report.append(
f"@{actor} please check your translation if it conforms to the standard. Follow the format of [en-GB/translation.toml](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/frontend/public/locales/en-GB/translation.toml)"
f"@{actor} please check your translation if it conforms to the standard. Follow the format of [en-GB/translation.toml](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/frontend/editor/public/locales/en-GB/translation.toml)"
)
else:
report.append("## ✅ Overall Check Status: **_Success_**")
+2 -1
View File
@@ -287,6 +287,7 @@ jobs:
- /stirling/V2-PR-${{ needs.check-pr.outputs.pr_number }}/data:/usr/share/tessdata:rw
- /stirling/V2-PR-${{ needs.check-pr.outputs.pr_number }}/config:/configs:rw
- /stirling/V2-PR-${{ needs.check-pr.outputs.pr_number }}/logs:/logs:rw
- /stirling/V2-PR-${{ needs.check-pr.outputs.pr_number }}/storage:/storage:rw
environment:
DISABLE_ADDITIONAL_FEATURES: "false"
SECURITY_ENABLELOGIN: "true"
@@ -309,7 +310,7 @@ jobs:
ssh -i ../private.key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -T ${{ secrets.NEW_VPS_USERNAME }}@${{ secrets.NEW_VPS_HOST }} << ENDSSH
# Create V2 PR-specific directories
mkdir -p /stirling/V2-PR-${{ needs.check-pr.outputs.pr_number }}/{data,config,logs}
mkdir -p /stirling/V2-PR-${{ needs.check-pr.outputs.pr_number }}/{data,config,logs,storage}
# Move docker-compose file to correct location
mv /tmp/docker-compose-v2.yml /stirling/V2-PR-${{ needs.check-pr.outputs.pr_number }}/docker-compose.yml
+124 -27
View File
@@ -1,8 +1,9 @@
name: AI Engine CI
# Validates the Python AI engine: regenerates tool models, runs fixers,
# lint, type-check, and tests. Called from build.yml on PRs and merge_group;
# also runs directly on push to main as a post-merge safety net.
# Validates the Python AI engine: regenerates tool models and runs the
# engine quality gate (lint, type-check, format-check, tests). Called from
# build.yml on PRs and merge_group; also runs directly on push to main as
# a post-merge safety net.
on:
workflow_call:
push:
@@ -51,27 +52,95 @@ jobs:
run: task engine:tool-models
- name: Verify tool models are up to date
id: tool-models-check
continue-on-error: true
run: git diff --exit-code engine/src/stirling/models/tool_models.py
- name: Comment on tool models check failure
# Only post a comment on PRs. github-script's PR helpers need an
# issue/PR number, which doesn't exist on merge_group runs.
if: steps.tool-models-check.outcome == 'failure' && github.event_name == 'pull_request'
continue-on-error: true
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const marker = '<!-- tool-models-check -->';
const body = [
marker,
'### Tool Models Check Failed',
'',
'The generated `engine/src/stirling/models/tool_models.py` is out of date with the Java OpenAPI spec and will need to be regenerated before it can be merged in.',
'',
'Run `task engine:tool-models` to regenerate, then commit the updated file.',
].join('\n');
const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
});
const existing = comments.find(c => c.body.includes(marker));
if (existing) {
await github.rest.issues.updateComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: existing.id,
body,
});
} else {
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body,
});
}
- name: Fail if tool models check failed
if: steps.tool-models-check.outcome == 'failure'
run: |
if ! git diff --exit-code engine/src/stirling/models/tool_models.py; then
echo "tool_models.py is out of date."
echo "Run 'task engine:tool-models' locally and commit the updated file."
exit 1
fi
echo "============================================"
echo " Tool Models Check Failed"
echo "============================================"
echo ""
echo "The generated engine/src/stirling/models/tool_models.py"
echo "is out of date with the Java OpenAPI spec and will"
echo "need to be regenerated before it can be merged in."
echo ""
echo "Run 'task engine:tool-models' to regenerate, then"
echo "commit the updated file."
echo "============================================"
exit 1
- name: Run fixers
run: task engine:fix
- name: Remove tool models check comment on success
if: steps.tool-models-check.outcome == 'success' && github.event_name == 'pull_request'
continue-on-error: true
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const marker = '<!-- tool-models-check -->';
const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
});
const existing = comments.find(c => c.body.includes(marker));
if (existing) {
await github.rest.issues.deleteComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: existing.id,
});
}
- name: Verify fixes are committed
id: fixer_changes
run: |
if ! git diff --quiet; then
git --no-pager diff --stat
echo "::error::There are issues with your Python code that will need to be fixed before they can be merged in. Run 'task engine:fix' to auto-fix what can be fixed automatically, then run 'task engine:check' to see what still needs fixing manually."
exit 1
fi
- name: Quality-check engine
id: engine-check
run: task engine:check
continue-on-error: true
- name: Comment on fixer failures
if: steps.fixer_changes.outcome == 'failure' && github.event_name == 'pull_request'
- name: Comment on engine check failure
# Only post a comment on PRs. github-script's PR helpers need an
# issue/PR number, which doesn't exist on merge_group runs.
if: steps.engine-check.outcome == 'failure' && github.event_name == 'pull_request'
continue-on-error: true
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
@@ -107,11 +176,39 @@ jobs:
});
}
- name: Run linting
run: task engine:lint
- name: Fail if engine check failed
if: steps.engine-check.outcome == 'failure'
run: |
echo "============================================"
echo " Engine Check Failed"
echo "============================================"
echo ""
echo "There are issues with your Python code that"
echo "will need to be fixed before they can be merged in."
echo ""
echo "Run 'task engine:fix' to auto-fix what can be"
echo "fixed automatically, then run 'task engine:check'"
echo "to see what still needs fixing manually."
echo "============================================"
exit 1
- name: Run type checking
run: task engine:typecheck
- name: Run tests
run: task engine:test
- name: Remove engine check comment on success
if: steps.engine-check.outcome == 'success' && github.event_name == 'pull_request'
continue-on-error: true
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const marker = '<!-- engine-check -->';
const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
});
const existing = comments.find(c => c.body.includes(marker));
if (existing) {
await github.rest.issues.deleteComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: existing.id,
});
}
+32 -15
View File
@@ -67,7 +67,7 @@ jobs:
MAVEN_PASSWORD: ${{ secrets.MAVEN_PASSWORD }}
MAVEN_PUBLIC_URL: ${{ secrets.MAVEN_PUBLIC_URL }}
- name: Comment on Java formatting failure
- name: Comment on backend format check failure
# Only post a comment on PRs. github-script's PR helpers need an
# issue/PR number, which doesn't exist on merge_group runs.
if: steps.spotless-check.outcome == 'failure' && github.event_name == 'pull_request'
@@ -78,15 +78,11 @@ jobs:
const marker = '<!-- java-formatting-check -->';
const body = [
marker,
'### Java Formatting Check Failed',
'### Backend Format Check Failed',
'',
'Your code has formatting issues. Run the following command to fix them:',
'There are formatting issues in your Java code that will need to be fixed before they can be merged in.',
'',
'```bash',
'task backend:format',
'```',
'',
'Then commit and push the changes.',
'Run `task backend:format` to auto-fix, then commit and push the changes.',
].join('\n');
const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
@@ -110,22 +106,43 @@ jobs:
});
}
- name: Fail if Java formatting issues found
- name: Fail if backend format check failed
if: steps.spotless-check.outcome == 'failure'
run: |
echo "============================================"
echo " Java Formatting Check Failed"
echo " Backend Format Check Failed"
echo "============================================"
echo ""
echo "Your code has formatting issues."
echo "Run the following command to fix them:"
echo "There are formatting issues in your Java code"
echo "that will need to be fixed before they can be"
echo "merged in."
echo ""
echo " task backend:format"
echo ""
echo "Then commit and push the changes."
echo "Run 'task backend:format' to auto-fix, then"
echo "commit and push the changes."
echo "============================================"
exit 1
- name: Remove backend format check comment on success
if: steps.spotless-check.outcome == 'success' && github.event_name == 'pull_request'
continue-on-error: true
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const marker = '<!-- java-formatting-check -->';
const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
});
const existing = comments.find(c => c.body.includes(marker));
if (existing) {
await github.rest.issues.deleteComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: existing.id,
});
}
- name: Build with Gradle and spring security ${{ matrix.spring-security }}
run: task backend:build:ci
env:
+14
View File
@@ -68,6 +68,18 @@ jobs:
uses: ./.github/workflows/backend-build.yml
secrets: inherit
db-migration-test:
# Boots the current bootJar against H2 fixtures captured from past
# releases (v2.0.0 / v2.5.0 / v2.10.0) and verifies admin login still
# works after Hibernate's ddl-auto=update migrates the schema. Gated on
# the `project` filter so doc-only PRs skip this ~5-minute job.
if: needs.files-changed.outputs.project == 'true'
needs: [files-changed]
permissions:
contents: read
uses: ./.github/workflows/db-migration-test.yml
secrets: inherit
check-generateOpenApiDocs:
if: needs.files-changed.outputs.openapi == 'true'
needs: [files-changed]
@@ -184,6 +196,7 @@ jobs:
needs:
- files-changed
- build
- db-migration-test
- check-generateOpenApiDocs
- frontend-validation
- playwright-e2e
@@ -208,6 +221,7 @@ jobs:
RESULTS: |
files-changed=${{ needs.files-changed.result }}
build=${{ needs.build.result }}
db-migration-test=${{ needs.db-migration-test.result }}
check-generateOpenApiDocs=${{ needs.check-generateOpenApiDocs.result }}
frontend-validation=${{ needs.frontend-validation.result }}
playwright-e2e=${{ needs.playwright-e2e.result }}
+93
View File
@@ -0,0 +1,93 @@
name: DB migration smoke test
# Boots the current Stirling-PDF JAR against H2 fixtures captured from past
# releases (v2.0.0 / v2.5.0 / v2.10.0) and verifies admin login still works.
# Catches schema changes that would break existing user databases under
# Hibernate's `ddl-auto=update` upgrade path.
on:
workflow_call:
permissions:
contents: read
jobs:
pick:
uses: ./.github/workflows/_runner-pick.yml
migration-test:
needs: pick
runs-on: ${{ needs.pick.outputs.is_fork == 'true' && 'ubuntu-latest' || 'depot-ubuntu-24.04-8' }}
timeout-minutes: 30
env:
DEPOT_TOKEN: ${{ secrets.DEPOT_TOKEN }}
steps:
- name: Harden Runner
uses: step-security/harden-runner@ab7a9404c0f3da075243ca237b5fac12c98deaa5 # v2.19.3
with:
egress-policy: audit
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Set up JDK 25
uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5.2.0
with:
java-version: 25
distribution: temurin
- name: Cache Gradle dependency artifacts
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: |
~/.gradle/wrapper
~/.gradle/caches/modules-2/files-2.1
~/.gradle/caches/modules-2/metadata-2.*
key: gradle-deps-${{ runner.os }}-jdk-25-${{ hashFiles('**/gradle/wrapper/gradle-wrapper.properties', '**/*.gradle', '**/*.gradle.kts', 'settings.gradle', 'settings.gradle.kts', 'gradle/libs.versions.toml') }}
- name: Setup Gradle
uses: gradle/actions/setup-gradle@50e97c2cd7a37755bbfafc9c5b7cafaece252f6e # v6.1.0
with:
gradle-version: 9.3.1
cache-disabled: true
# No `-PnoSpotless` here yet because the upstream cache layer matches the
# backend build's; reuse keeps cold-cache cost identical.
- name: Build Stirling-PDF JAR
env:
MAVEN_USER: ${{ secrets.MAVEN_USER }}
MAVEN_PASSWORD: ${{ secrets.MAVEN_PASSWORD }}
MAVEN_PUBLIC_URL: ${{ secrets.MAVEN_PUBLIC_URL }}
run: ./gradlew :stirling-pdf:bootJar -PnoSpotless --no-daemon
- name: Locate built JAR
id: jar
run: |
jar=$(find app/core/build/libs -maxdepth 1 -name 'Stirling-PDF*.jar' -o -name 'stirling-pdf*.jar' 2>/dev/null \
| grep -vE '(-plain|-sources)\.jar$' | head -n 1)
if [[ -z "$jar" ]]; then
echo "::error::No JAR under app/core/build/libs"
ls -lah app/core/build/libs || true
exit 1
fi
# Absolute path - the migration script pushd's into a temp workdir
# before invoking java, which would dangle a relative path.
jar=$(realpath "$jar")
echo "path=$jar" >> "$GITHUB_OUTPUT"
echo "Built JAR: $jar"
- name: Run migration smoke test
env:
STIRLING_JAR: ${{ steps.jar.outputs.path }}
run: bash scripts/db-migration/run-migration-test.sh
- name: Upload app logs on failure
if: failure()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: db-migration-app-logs
# Path matches the preserved workdir in run-migration-test.sh -
# only failing fixtures leave a directory behind.
path: /tmp/stirling-migration-failed-*/app.log
retention-days: 7
if-no-files-found: warn
+9 -7
View File
@@ -586,21 +586,23 @@ jobs:
if: always() && steps.digicert-setup.conclusion != 'failure'
shell: bash
run: |
mkdir -p ./dist
# Absolute dist path so the cd below can't break the copy targets.
DIST="$GITHUB_WORKSPACE/dist"
mkdir -p "$DIST"
cd ./frontend/editor/src-tauri/target
# Find and rename artifacts based on platform
if [ "${{ matrix.platform }}" = "windows-latest" ]; then
# Only ship the MSI installer on Windows. The loose exe and WiX toolset exes
# are not the user-facing installer - the MSI contains the signed inner exe.
find . -name "*.msi" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.msi" \;
find . -name "*.msi" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.msi" \;
elif [ "${{ matrix.platform }}" = "macos-15" ]; then
find . -name "*.dmg" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.dmg" \;
find . -name "*.app" -exec cp -r {} "../../../dist/Stirling-PDF-${{ matrix.name }}.app" \;
find . -name "*.dmg" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.dmg" \;
find . -name "*.app" -exec cp -r {} "$DIST/Stirling-PDF-${{ matrix.name }}.app" \;
else
find . -name "*.deb" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.deb" \;
find . -name "*.rpm" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.rpm" \;
find . -name "*.AppImage" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.AppImage" \;
find . -name "*.deb" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.deb" \;
find . -name "*.rpm" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.rpm" \;
find . -name "*.AppImage" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.AppImage" \;
fi
- name: Upload build artifacts
+11 -6
View File
@@ -157,6 +157,9 @@ jobs:
JPDFIUM_PLATFORMS: ${{ matrix.jpdfium_platforms }}
run: task desktop:prepare
- name: Run Tauri/Cargo tests
run: task desktop:test
# DigiCert KeyLocker Setup (Cloud HSM)
- name: Setup DigiCert KeyLocker
id: digicert-setup
@@ -417,20 +420,22 @@ jobs:
- name: Rename artifacts
shell: bash
run: |
mkdir -p ./dist
# Absolute dist path so the cd below can't break the copy targets.
DIST="$GITHUB_WORKSPACE/dist"
mkdir -p "$DIST"
cd ./frontend/editor/src-tauri/target
# Find and rename artifacts based on platform
if [ "${{ matrix.platform }}" = "windows-latest" ]; then
# Only ship the MSI installer. The loose exe and WiX toolset exes
# are not the user-facing installer - the MSI contains the signed inner exe.
find . -name "*.msi" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.msi" \;
find . -name "*.msi" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.msi" \;
elif [ "${{ matrix.platform }}" = "macos-15" ]; then
find . -name "*.dmg" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.dmg" \;
find . -name "*.dmg" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.dmg" \;
else
find . -name "*.deb" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.deb" \;
find . -name "*.rpm" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.rpm" \;
find . -name "*.AppImage" -exec cp {} "../../../dist/Stirling-PDF-${{ matrix.name }}.AppImage" \;
find . -name "*.deb" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.deb" \;
find . -name "*.rpm" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.rpm" \;
find . -name "*.AppImage" -exec cp {} "$DIST/Stirling-PDF-${{ matrix.name }}.AppImage" \;
fi
# Verify the MSI AND the inner exe extracted from it are signed.
+7 -1
View File
@@ -23,6 +23,10 @@ customFiles/
configs/
watchedFolders/
clientWebUI/
# Scratch dir used by local fixture-regeneration runs (see
# app/proprietary/src/test/resources/db-migration-fixtures/README.md).
# Holds downloaded JARs and disposable workdirs. Never committed.
.alpha-local/
!cucumber/
!cucumber/exampleFiles/
!cucumber/exampleFiles/example_html.zip
@@ -174,7 +178,6 @@ venv.bak/
# Env files (secrets / local overrides). Subproject .gitignore files whitelist any committed defaults.
.env*
!.env.saas.example
# VS Code
/.vscode/**/*
@@ -274,3 +277,6 @@ docs/type3/signatures/
# Playwright MCP screenshots / traces
.playwright-mcp/
*.playwright-mcp.png
# Local screenshot artifacts from *-screenshots.spec.ts
frontend/screenshots/
+6 -5
View File
@@ -22,12 +22,13 @@ tasks:
vars:
PORT: '{{.PORT | default "8080"}}'
AIENGINE_URL: '{{.AIENGINE_URL | default ""}}'
AIENGINE_TIMEOUTSECONDS: '{{.AIENGINE_TIMEOUTSECONDS | default "120"}}'
env:
SERVER_PORT: '{{.PORT}}'
cmds:
- cmd: '{{if .AIENGINE_URL}}AIENGINE_URL={{.AIENGINE_URL}} AIENGINE_ENABLED=true {{end}}cmd /c ".\gradlew.bat :stirling-pdf:bootRun"'
- cmd: '{{if .AIENGINE_URL}}AIENGINE_URL={{.AIENGINE_URL}} AIENGINE_ENABLED=true AIENGINE_TIMEOUTSECONDS={{.AIENGINE_TIMEOUTSECONDS}} {{end}}cmd /c ".\gradlew.bat :stirling-pdf:bootRun"'
platforms: [windows]
- cmd: '{{if .AIENGINE_URL}}AIENGINE_URL={{.AIENGINE_URL}} AIENGINE_ENABLED=true {{end}}./gradlew :stirling-pdf:bootRun'
- cmd: '{{if .AIENGINE_URL}}AIENGINE_URL={{.AIENGINE_URL}} AIENGINE_ENABLED=true AIENGINE_TIMEOUTSECONDS={{.AIENGINE_TIMEOUTSECONDS}} {{end}}./gradlew :stirling-pdf:bootRun'
platforms: [linux, darwin]
dev:bundled:
@@ -40,10 +41,10 @@ tasks:
platforms: [linux, darwin]
dev:saas:
desc: "Start backend in SaaS flavor against Supabase (loads .env.saas.local)"
desc: "Start backend in SaaS flavor against Supabase"
# `dotenv:` reads from the root Taskfile's directory (".") because this
# subtaskfile is included with `dir: .`. Drop the file at the repo root.
dotenv: ['.env.saas.local']
# subtaskfile is included with `dir: .`.
dotenv: ['app/.env.saas.local', 'app/.env.saas']
ignore_error: true
vars:
PORT: '{{.PORT | default "8080"}}'
+7
View File
@@ -78,6 +78,13 @@ tasks:
cmds:
- npx tauri build --bundles appimage
test:
desc: "Run Tauri/Cargo tests"
deps: [prepare]
dir: editor/src-tauri
cmds:
- cargo test
clean:
desc: "Clean Tauri/Cargo build artifacts"
dir: editor
+18 -1
View File
@@ -58,6 +58,23 @@ tasks:
BACKEND_URL: 'http://localhost:{{.BACKEND_PORT}}'
OPEN: "true"
dev:saas:
desc: "Start SaaS backend + frontend concurrently on free ports"
vars:
PORTS:
sh: '{{if eq OS "windows"}}{{.FIND_FREE_PORT_PS}} 8080 5173{{else}}{{.FIND_FREE_PORT_SH}} 8080 5173{{end}}'
BACKEND_PORT: '{{index (splitList "\n" .PORTS) 0}}'
FRONTEND_PORT: '{{index (splitList "\n" .PORTS) 1}}'
deps:
- task: backend:dev:saas
vars:
PORT: '{{.BACKEND_PORT}}'
- task: frontend:dev:saas
vars:
PORT: '{{.FRONTEND_PORT}}'
BACKEND_URL: 'http://localhost:{{.BACKEND_PORT}}'
OPEN: "true"
dev:all:
desc: "Start backend + frontend + engine concurrently on free ports"
vars:
@@ -74,7 +91,7 @@ tasks:
vars:
PORT: '{{.BACKEND_PORT}}'
AIENGINE_URL: 'http://localhost:{{.ENGINE_PORT}}'
- task: frontend:dev:prototypes
- task: frontend:dev
vars:
PORT: '{{.FRONTEND_PORT}}'
BACKEND_URL: 'http://localhost:{{.BACKEND_PORT}}'
+9 -9
View File
@@ -1,20 +1,20 @@
###############################################################################
# Stirling-PDF SaaS local environment template.
# Stirling-PDF SaaS environment defaults.
#
# Copy this file to `.env.saas.local` (gitignored) and fill in real values.
# Loaded by `task backend:dev:saas` via Taskfile's `dotenv:` directive, then
# read by Spring Boot's `${...}` placeholders in application-saas.properties
# and application-dev.properties.
# This file is committed and provides non-secret defaults loaded by
# `task backend:dev:saas`. Put real values for secrets (passwords, project
# refs, edge function secrets) in `.env.saas.local` - any variable set there
# takes precedence over what's defined here.
#
# DO NOT commit `.env.saas.local`. Only `.env.saas.example` is checked in.
# DO NOT commit `.env.saas.local`. Only `.env.saas` is checked in.
###############################################################################
# ---------- Supabase project ----------
# Project reference (the subdomain part of <ref>.supabase.co). Required.
# Example dev project:
# Set in .env.saas.local.
SAAS_DB_PROJECT_REF=
# Edge function secret used by billing/license rollup calls.
# Edge function secret used by billing/license rollup calls. Set in .env.saas.local.
SUPABASE_EDGE_FUNCTION_SECRET=
# ---------- Database (saas profile) ----------
@@ -28,7 +28,7 @@ SAAS_DB_PASSWORD=
# ---------- Database (dev profile overrides) ----------
# Used when `--spring.profiles.include=dev` is active. The dev profile
# defaults the URL/username to the shared dev Supabase project, but the
# password must still be provided here.
# password must still be provided in .env.saas.local.
SAAS_DEV_DB_URL=
SAAS_DEV_DB_USERNAME=postgres
SAAS_DEV_DB_PASSWORD=
+3
View File
@@ -0,0 +1,3 @@
# Whitelist committed env defaults. `.env.saas.local` (and any other .env*)
# stays ignored via the root .gitignore.
!.env.saas
+4
View File
@@ -44,6 +44,10 @@
"moduleName": ".*",
"moduleLicense": "The MIT License"
},
{
"moduleName": ".*",
"moduleLicense": "MIT-0"
},
{
"moduleName": "com.github.jai-imageio:jai-imageio-core",
"moduleLicense": "LICENSE.txt"
@@ -77,6 +77,10 @@ public @interface AutoJobPostMapping {
/**
* Relative resource weight (1-100). See {@link
* stirling.software.common.enumeration.ResourceWeight} for the standard tiers.
*
* <p>The default is a sentinel ({@link Integer#MIN_VALUE}); {@code
* AutoJobPostMappingWeightTest} fails the build if any endpoint leaves it unset. Runtime
* readers clamp the value into {@code [1, 100]}.
*/
int resourceWeight() default 1;
int resourceWeight() default Integer.MIN_VALUE;
}
@@ -2,15 +2,7 @@ package stirling.software.common.cluster;
import java.time.Duration;
/**
* Token-bucket rate limiting backed by the cluster backplane.
*
* <p>In-process implementations enforce a per-JVM limit (identical to today's behaviour).
* Distributed implementations enforce a single global limit across every node.
*
* <p>Both implementations use a Bucket4j greedy-refill token bucket so semantics match across
* single-node and cluster deployments (no fixed-window boundary doubling).
*/
/** Token-bucket rate limiting backed by the cluster backplane. */
public interface RateLimitStore {
/**
@@ -80,6 +80,7 @@ public class ConfigInitializer {
YamlHelper settingsFile = new YamlHelper(settingTempPath);
migrateEnterpriseEditionToPremium(settingsFile, settingsTemplateFile);
migrateProFeaturesKeyCasing(settingsFile, settingsTemplateFile);
boolean changesMade =
settingsTemplateFile.updateValuesFromYaml(settingsFile, settingsTemplateFile);
@@ -116,31 +117,52 @@ public class ConfigInitializer {
}
if (yaml.getValueByExactKeyPath("enterpriseEdition", "SSOAutoLogin") != null) {
template.updateValue(
List.of("premium", "proFeatures", "SSOAutoLogin"),
List.of("premium", "proFeatures", "ssoAutoLogin"),
yaml.getValueByExactKeyPath("enterpriseEdition", "SSOAutoLogin"));
}
if (yaml.getValueByExactKeyPath("enterpriseEdition", "CustomMetadata", "autoUpdateMetadata")
!= null) {
template.updateValue(
List.of("premium", "proFeatures", "CustomMetadata", "autoUpdateMetadata"),
List.of("premium", "proFeatures", "customMetadata", "autoUpdateMetadata"),
yaml.getValueByExactKeyPath(
"enterpriseEdition", "CustomMetadata", "autoUpdateMetadata"));
}
if (yaml.getValueByExactKeyPath("enterpriseEdition", "CustomMetadata", "author") != null) {
template.updateValue(
List.of("premium", "proFeatures", "CustomMetadata", "author"),
List.of("premium", "proFeatures", "customMetadata", "author"),
yaml.getValueByExactKeyPath("enterpriseEdition", "CustomMetadata", "author"));
}
if (yaml.getValueByExactKeyPath("enterpriseEdition", "CustomMetadata", "creator") != null) {
template.updateValue(
List.of("premium", "proFeatures", "CustomMetadata", "creator"),
List.of("premium", "proFeatures", "customMetadata", "creator"),
yaml.getValueByExactKeyPath("enterpriseEdition", "CustomMetadata", "creator"));
}
if (yaml.getValueByExactKeyPath("enterpriseEdition", "CustomMetadata", "producer")
!= null) {
template.updateValue(
List.of("premium", "proFeatures", "CustomMetadata", "producer"),
List.of("premium", "proFeatures", "customMetadata", "producer"),
yaml.getValueByExactKeyPath("enterpriseEdition", "CustomMetadata", "producer"));
}
}
// TODO: Remove post migration
// settings.yml.template renamed the two non-camelCase proFeatures keys
// ("SSOAutoLogin" -> "ssoAutoLogin", "CustomMetadata" -> "customMetadata") so the whole
// settings pipeline is consistent camelCase. The save path (YamlHelper.updateValue) matches
// keys case-sensitively, so without this carry-forward an existing install's values written
// under the old PascalCase keys would be dropped on upgrade and reset to template defaults.
void migrateProFeaturesKeyCasing(YamlHelper yaml, YamlHelper template) {
Object ssoAutoLogin = yaml.getValueByExactKeyPath("premium", "proFeatures", "SSOAutoLogin");
if (ssoAutoLogin != null) {
template.updateValue(List.of("premium", "proFeatures", "ssoAutoLogin"), ssoAutoLogin);
}
for (String field : List.of("autoUpdateMetadata", "author", "creator", "producer")) {
Object value =
yaml.getValueByExactKeyPath("premium", "proFeatures", "CustomMetadata", field);
if (value != null) {
template.updateValue(
List.of("premium", "proFeatures", "customMetadata", field), value);
}
}
}
}
@@ -528,6 +528,16 @@ public class ApplicationProperties {
private String provider;
private Client client = new Client();
/**
* When true, the OAuth2/OIDC login flow logs the full set of ID token and UserInfo
* claims at INFO level (and again at ERROR level if the username attribute cannot be
* resolved). Used to diagnose provider misconfiguration (for example ADFS not returning
* an {@code email} claim). WARNING: writes PII (sub, email, name) to application logs.
* Leave disabled in production; enable only while actively troubleshooting and disable
* again afterwards.
*/
private Boolean debugLogging = false;
public void setScopes(String scopes) {
List<String> scopesList =
Arrays.stream(scopes.split(",")).map(String::trim).toList();
@@ -778,6 +788,7 @@ public class ApplicationProperties {
private boolean enabled = false;
private String provider = "local";
private Local local = new Local();
private S3 s3 = new S3();
private Quotas quotas = new Quotas();
private Sharing sharing = new Sharing();
private Signing signing = new Signing();
@@ -787,6 +798,57 @@ public class ApplicationProperties {
private String basePath = InstallationPathConfig.getPath() + "storage";
}
@Data
public static class S3 {
/**
* Optional custom endpoint (e.g. {@code https://<account>.r2.cloudflarestorage.com},
* {@code https://<project>.supabase.co/storage/v1/s3}, or {@code http://localhost:9000}
* for MinIO). Blank = use AWS regional default.
*/
private String endpoint = "";
private String bucket = "";
private String region = "us-east-1";
private String accessKey = "";
private String secretKey = "";
/**
* When {@code true} use path-style URLs ({@code <endpoint>/<bucket>/<key>}) instead of
* virtual-hosted ({@code <bucket>.<endpoint>/<key>}). MinIO and most S3-compatible
* gateways require path-style; AWS S3 prefers virtual-hosted.
*/
private boolean pathStyleAccess = false;
/**
* When {@code false} (default), {@code endpoint} hostnames that resolve to private,
* loopback, or link-local addresses are rejected at startup to block SSRF attacks via
* the cloud metadata service (e.g. {@code http://169.254.169.254/}). Set to {@code
* true} to opt in for MinIO / in-cluster S3 endpoints on private networks.
*/
private boolean allowPrivateEndpoints = false;
/**
* Controls when the SDK adds an {@code x-amz-checksum-*} header on PUT/UploadPart.
* Default {@code WHEN_SUPPORTED} (the SDK default since 2.30) makes the SDK send a
* CRC32 checksum on every upload - this works on AWS S3, MinIO, current Supabase,
* Backblaze B2 (post-July-2025), and modern R2. Set to {@code WHEN_REQUIRED} to
* suppress the auto-checksum on vendors that reject unknown {@code x-amz-checksum-*}
* headers (older Backblaze B2, some R2 corner cases, GCS S3 endpoint). Invalid values
* fall back to {@code WHEN_SUPPORTED}.
*/
private String requestChecksumCalculation = "WHEN_SUPPORTED";
/**
* Controls when the SDK validates returned {@code x-amz-checksum-*} headers on GET
* responses. Default {@code WHEN_SUPPORTED}. Set to {@code WHEN_REQUIRED} if your
* vendor never returns these headers and you see false-positive checksum-mismatch
* errors. Invalid values fall back to {@code WHEN_SUPPORTED}.
*/
private String responseChecksumValidation = "WHEN_SUPPORTED";
}
@Data
public static class Sharing {
private boolean enabled = false;
@@ -60,7 +60,7 @@ public class CustomPDFDocumentFactory {
this(pdfMetadataService, null);
}
/** Documents ≤ this size are loaded entirely into heap - no temp files needed. */
/** Documents ≤ this size are loaded entirely into heap — no temp files needed. */
public static final long SMALL_FILE_THRESHOLD = 10L * 1024 * 1024; // 10 MB
/** Upper boundary of the "mixed" memory+file zone; above this always file-backed. */
@@ -130,6 +130,7 @@ public class CustomPDFDocumentFactory {
MemorySnapshot mem = MemorySnapshot.capture();
// Use the overridable method so that test spies (SpyPDFDocumentFactory) can intercept.
StreamCacheCreateFunction cache = getStreamCacheFunction(size, mem);
// Non-destructive — caller's file is never deleted
RandomAccessReadBufferedFile raf = new RandomAccessReadBufferedFile(file);
PDDocument doc;
try {
@@ -112,12 +112,25 @@ public class JobExecutorService {
log.debug("Generated jobId: {} (base: {})", scopedJobKey, baseJobId);
// Store the scoped job ID in the request for potential use by other components.
// Ownership lives in the scoped key itself (userId:jobId) plus the cluster-visible
// JobStore entry, so we no longer mirror it into the HTTP session - that did not
// survive a node hop in cluster mode.
// Store the scoped job ID in the request for potential use by other components
if (request != null) {
request.setAttribute("jobId", scopedJobKey);
// Also track this job ID in the user's session for authorization purposes
// This ensures users can only cancel their own jobs
if (request.getSession() != null) {
@SuppressWarnings("unchecked")
java.util.Set<String> userJobIds =
(java.util.Set<String>) request.getSession().getAttribute("userJobIds");
if (userJobIds == null) {
userJobIds = new java.util.concurrent.ConcurrentSkipListSet<>();
request.getSession().setAttribute("userJobIds", userJobIds);
}
userJobIds.add(scopedJobKey);
log.debug("Added scoped job ID {} to user session", scopedJobKey);
}
}
String jobId = scopedJobKey;
@@ -83,7 +83,16 @@ public class RequestUriUtils {
return false;
}
// Blocklist of backend/non-frontend paths that should still go through filters
// Blocklist of backend/non-frontend paths that should still go through filters.
//
// `/files` was historically a backend route; it is now a frontend route
// owned by HomePage / FileManagerView. Direct-nav or refresh on /files
// (or /files/<folder-uuid>) was returning the Spring auth filter's 401
// JSON instead of serving index.html, so the SPA never got a chance to
// mount and the user saw a raw error response. There are no `/files`
// backend mappings at the servlet root - the real storage endpoints
// live under `/api/v1/storage/files`, which is filtered out a few lines
// up by the `startsWith("/api/")` guard.
String[] backendOnlyPrefixes = {
"/register",
"/pipeline",
@@ -91,7 +100,6 @@ public class RequestUriUtils {
"/pdfjs-legacy",
"/fonts",
"/images",
"/files",
"/css",
"/js",
"/swagger",
@@ -181,7 +189,7 @@ public class RequestUriUtils {
|| trimmedUri.startsWith(
"/api/v1/mobile-scanner/") // Mobile scanner endpoints (no auth)
|| trimmedUri.startsWith("/v1/api-docs")
// Workflow participant endpoints — access controlled by share tokens, not login
// Workflow participant endpoints - access controlled by share tokens, not login
|| trimmedUri.startsWith("/api/v1/workflow/participant/")
// Share-link SPA bootstrap; data APIs remain protected
|| trimmedUri.matches("^/share/[^/]+/?$");
@@ -24,9 +24,7 @@ class InProcessDistributedLockTest {
}
@Test
void reentryFromSameThreadFails_parityWithValkey() {
// The Valkey impl refuses reentry (SET NX semantics); the in-process impl must match,
// otherwise code working in single-instance silently breaks in cluster mode.
void reentryFromSameThreadFails() {
DistributedLock lock = new InProcessDistributedLock();
DistributedLock.LockHandle h1 = lock.tryAcquire("k", Duration.ofSeconds(30)).orElseThrow();
Optional<DistributedLock.LockHandle> reentry = lock.tryAcquire("k", Duration.ofSeconds(30));
@@ -0,0 +1,95 @@
package stirling.software.common.configuration;
import static org.junit.jupiter.api.Assertions.*;
import org.junit.jupiter.api.Test;
import org.snakeyaml.engine.v2.api.LoadSettings;
import stirling.software.common.util.YamlHelper;
class ConfigInitializerTest {
private static final LoadSettings LOAD_SETTINGS =
LoadSettings.builder()
.setUseMarks(true)
.setMaxAliasesForCollections(Integer.MAX_VALUE)
.setAllowRecursiveKeys(true)
.setParseComments(true)
.build();
// Mirrors the proFeatures block of settings.yml.template after the camelCase rename.
private static final String CAMEL_CASE_TEMPLATE =
"""
premium:
proFeatures:
ssoAutoLogin: false
customMetadata:
autoUpdateMetadata: false
author: username
creator: Stirling-PDF
producer: Stirling-PDF
""";
@Test
void migrateProFeaturesKeyCasing_carriesForwardLegacyPascalCaseValues() {
// An existing install whose settings.yml still uses the old PascalCase keys.
String legacy =
"""
premium:
proFeatures:
SSOAutoLogin: true
CustomMetadata:
autoUpdateMetadata: true
author: alice
creator: bob
producer: carol
""";
YamlHelper template = new YamlHelper(LOAD_SETTINGS, CAMEL_CASE_TEMPLATE);
YamlHelper existing = new YamlHelper(LOAD_SETTINGS, legacy);
new ConfigInitializer().migrateProFeaturesKeyCasing(existing, template);
assertEquals(
"true", template.getValueByExactKeyPath("premium", "proFeatures", "ssoAutoLogin"));
assertEquals(
"true",
template.getValueByExactKeyPath(
"premium", "proFeatures", "customMetadata", "autoUpdateMetadata"));
assertEquals(
"alice",
template.getValueByExactKeyPath(
"premium", "proFeatures", "customMetadata", "author"));
assertEquals(
"bob",
template.getValueByExactKeyPath(
"premium", "proFeatures", "customMetadata", "creator"));
assertEquals(
"carol",
template.getValueByExactKeyPath(
"premium", "proFeatures", "customMetadata", "producer"));
}
@Test
void migrateProFeaturesKeyCasing_withoutLegacyKeys_keepsTemplateDefaults() {
// No PascalCase keys present -> this migration step must be a no-op.
String alreadyCamel =
"""
premium:
proFeatures:
ssoAutoLogin: true
customMetadata:
author: dave
""";
YamlHelper template = new YamlHelper(LOAD_SETTINGS, CAMEL_CASE_TEMPLATE);
YamlHelper existing = new YamlHelper(LOAD_SETTINGS, alreadyCamel);
new ConfigInitializer().migrateProFeaturesKeyCasing(existing, template);
assertEquals(
"false", template.getValueByExactKeyPath("premium", "proFeatures", "ssoAutoLogin"));
assertEquals(
"username",
template.getValueByExactKeyPath(
"premium", "proFeatures", "customMetadata", "author"));
}
}
@@ -86,8 +86,6 @@ class TaskManagerJobStoreDelegationTest {
@Override
public boolean shouldRunLocalCleanup() {
// Distributed backplanes own job TTL eviction themselves; this mock
// mirrors the real ValkeyClusterBackplane override of the default true.
return false;
}
};
@@ -2,6 +2,8 @@ package stirling.software.common.util;
import static org.junit.jupiter.api.Assertions.*;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
@@ -12,9 +14,47 @@ import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.mockito.MockedStatic;
import org.mockito.Mockito;
import stirling.software.common.configuration.InstallationPathConfig;
public class GeneralUtilsTest {
// Regression guard for the SSO auto-login persistence bug: the admin UI writes camelCase
// proFeatures keys, so saveKeyToSettings must match (and persist) them against the camelCase
// settings.yml.template. A case mismatch makes YamlHelper.updateValue silently no-op.
@Test
void saveKeyToSettings_persistsCamelCaseProFeatureKeys(@TempDir Path tempDir) throws Exception {
Path settings = tempDir.resolve("settings.yml");
Files.writeString(
settings,
"""
premium:
proFeatures:
ssoAutoLogin: false
customMetadata:
author: username
""");
try (MockedStatic<InstallationPathConfig> mocked =
Mockito.mockStatic(InstallationPathConfig.class)) {
mocked.when(InstallationPathConfig::getSettingsPath).thenReturn(settings.toString());
GeneralUtils.saveKeyToSettings("premium.proFeatures.ssoAutoLogin", true);
GeneralUtils.saveKeyToSettings("premium.proFeatures.customMetadata.author", "alice");
}
YamlHelper reloaded = new YamlHelper(settings);
assertEquals(
"true", reloaded.getValueByExactKeyPath("premium", "proFeatures", "ssoAutoLogin"));
assertEquals(
"alice",
reloaded.getValueByExactKeyPath(
"premium", "proFeatures", "customMetadata", "author"));
}
@Test
void testParsePageListWithAll() {
List<Integer> result = GeneralUtils.parsePageList(new String[] {"all"}, 5, false);
@@ -98,6 +98,17 @@ class RequestUriUtilsTest {
assertTrue(RequestUriUtils.isFrontendRoute("", "/split-pdf"));
}
@Test
void testIsFrontendRoute_filesRouteOwnedByFrontend() {
// /files and /files/<folder-uuid> are FileManagerView routes - they
// must fall through to the SPA index.html, not get blocked by the
// backend auth filter. Regression test for direct-nav/refresh on
// the file manager returning a 401 JSON.
assertTrue(RequestUriUtils.isFrontendRoute("", "/files"));
assertTrue(
RequestUriUtils.isFrontendRoute("", "/files/3331910a-4155-4f71-8111-e38c896bc458"));
}
@Test
void testIsFrontendRoute_pathWithExtension() {
assertFalse(RequestUriUtils.isFrontendRoute("", "/some/file.pdf"));
@@ -183,7 +194,7 @@ class RequestUriUtilsTest {
@Test
void testIsPublicAuthEndpoint_shareRootNotPublic() {
// Avoid matching bare "/share" or "/share/" — must have a token segment
// Avoid matching bare "/share" or "/share/" - must have a token segment
assertFalse(RequestUriUtils.isPublicAuthEndpoint("/share", ""));
assertFalse(RequestUriUtils.isPublicAuthEndpoint("/share/", ""));
}
@@ -197,7 +208,7 @@ class RequestUriUtilsTest {
@Test
void testIsPublicAuthEndpoint_shareApiStillProtected() {
// Share-link data APIs must NOT be public — they enforce auth + access checks
// Share-link data APIs must NOT be public - they enforce auth + access checks
assertFalse(RequestUriUtils.isPublicAuthEndpoint("/api/v1/storage/share-links/abc123", ""));
assertFalse(
RequestUriUtils.isPublicAuthEndpoint(
@@ -32,6 +32,7 @@ import stirling.software.SPDF.model.json.PdfJsonTextElement;
import stirling.software.SPDF.service.PdfJsonConversionService;
import stirling.software.common.annotations.AutoJobPostMapping;
import stirling.software.common.annotations.api.GeneralApi;
import stirling.software.common.enumeration.ResourceWeight;
import stirling.software.common.model.api.general.EditTextOperation;
import stirling.software.common.util.ExceptionUtils;
import stirling.software.common.util.GeneralUtils;
@@ -75,7 +76,10 @@ public class EditTextController {
new StringToArrayListPropertyEditor<>(EditTextOperation.class));
}
@AutoJobPostMapping(consumes = "multipart/form-data", value = "/edit-text")
@AutoJobPostMapping(
consumes = "multipart/form-data",
value = "/edit-text",
resourceWeight = ResourceWeight.LARGE_WEIGHT)
@StandardPdfResponse
@Operation(
summary = "Edit text in a PDF via find and replace",
@@ -40,7 +40,8 @@ public class ScalePagesController {
private final CustomPDFDocumentFactory pdfDocumentFactory;
private final TempFileManager tempFileManager;
private static PDRectangle getTargetSize(String targetPDRectangle, PDDocument sourceDocument) {
private static PDRectangle getTargetSize(
String targetPDRectangle, String orientation, PDDocument sourceDocument) {
if ("KEEP".equals(targetPDRectangle)) {
if (sourceDocument.getNumberOfPages() == 0) {
throw ExceptionUtils.createInvalidPageSizeException("KEEP");
@@ -57,18 +58,19 @@ public class ScalePagesController {
}
Map<String, PDRectangle> sizeMap = getSizeMap();
if (sizeMap.containsKey(targetPDRectangle)) {
return sizeMap.get(targetPDRectangle);
PDRectangle base = sizeMap.get(targetPDRectangle);
if (base == null) {
throw ExceptionUtils.createInvalidPageSizeException(targetPDRectangle);
}
throw ExceptionUtils.createInvalidPageSizeException(targetPDRectangle);
if ("LANDSCAPE".equalsIgnoreCase(orientation)) {
return new PDRectangle(base.getHeight(), base.getWidth());
}
return base;
}
private static Map<String, PDRectangle> getSizeMap() {
Map<String, PDRectangle> sizeMap = new HashMap<>();
// Portrait sizes (A0-A6)
sizeMap.put("A0", PDRectangle.A0);
sizeMap.put("A1", PDRectangle.A1);
sizeMap.put("A2", PDRectangle.A2);
@@ -76,42 +78,8 @@ public class ScalePagesController {
sizeMap.put("A4", PDRectangle.A4);
sizeMap.put("A5", PDRectangle.A5);
sizeMap.put("A6", PDRectangle.A6);
// Landscape sizes (A0-A6)
sizeMap.put(
"A0_LANDSCAPE",
new PDRectangle(PDRectangle.A0.getHeight(), PDRectangle.A0.getWidth()));
sizeMap.put(
"A1_LANDSCAPE",
new PDRectangle(PDRectangle.A1.getHeight(), PDRectangle.A1.getWidth()));
sizeMap.put(
"A2_LANDSCAPE",
new PDRectangle(PDRectangle.A2.getHeight(), PDRectangle.A2.getWidth()));
sizeMap.put(
"A3_LANDSCAPE",
new PDRectangle(PDRectangle.A3.getHeight(), PDRectangle.A3.getWidth()));
sizeMap.put(
"A4_LANDSCAPE",
new PDRectangle(PDRectangle.A4.getHeight(), PDRectangle.A4.getWidth()));
sizeMap.put(
"A5_LANDSCAPE",
new PDRectangle(PDRectangle.A5.getHeight(), PDRectangle.A5.getWidth()));
sizeMap.put(
"A6_LANDSCAPE",
new PDRectangle(PDRectangle.A6.getHeight(), PDRectangle.A6.getWidth()));
// Portrait US sizes
sizeMap.put("LETTER", PDRectangle.LETTER);
sizeMap.put("LEGAL", PDRectangle.LEGAL);
// Landscape US sizes
sizeMap.put(
"LETTER_LANDSCAPE",
new PDRectangle(PDRectangle.LETTER.getHeight(), PDRectangle.LETTER.getWidth()));
sizeMap.put(
"LEGAL_LANDSCAPE",
new PDRectangle(PDRectangle.LEGAL.getHeight(), PDRectangle.LEGAL.getWidth()));
return sizeMap;
}
@@ -128,13 +96,14 @@ public class ScalePagesController {
throws IOException {
MultipartFile file = request.getFileInput();
String targetPDRectangle = request.getPageSize();
String orientation = request.getOrientation();
float scaleFactor = request.getScaleFactor();
try (PDDocument sourceDocument = pdfDocumentFactory.load(file);
PDDocument outputDocument =
pdfDocumentFactory.createNewDocumentBasedOnOldDocument(sourceDocument)) {
PDRectangle targetSize = getTargetSize(targetPDRectangle, sourceDocument);
PDRectangle targetSize = getTargetSize(targetPDRectangle, orientation, sourceDocument);
// Create LayerUtility once outside the loop for better performance
LayerUtility layerUtility = new LayerUtility(outputDocument);
@@ -275,7 +275,10 @@ public class ConvertImgPDFController {
GeneralUtils.generateFilename(file[0].getOriginalFilename(), "_converted.pdf"));
}
@AutoJobPostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE, value = "/cbz/pdf")
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/cbz/pdf",
resourceWeight = ResourceWeight.MEDIUM_WEIGHT)
@Operation(
summary = "Convert CBZ comic book archive to PDF",
description =
@@ -301,7 +304,10 @@ public class ConvertImgPDFController {
return WebResponseUtils.pdfFileToWebResponse(pdfFile, filename);
}
@AutoJobPostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE, value = "/pdf/cbz")
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/pdf/cbz",
resourceWeight = ResourceWeight.LARGE_WEIGHT)
@Operation(
summary = "Convert PDF to CBZ comic book archive",
description =
@@ -324,7 +330,10 @@ public class ConvertImgPDFController {
return WebResponseUtils.zipFileToWebResponse(cbzFile, filename);
}
@AutoJobPostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE, value = "/cbr/pdf")
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/cbr/pdf",
resourceWeight = ResourceWeight.MEDIUM_WEIGHT)
@Operation(
summary = "Convert CBR comic book archive to PDF",
description =
@@ -350,7 +359,10 @@ public class ConvertImgPDFController {
return WebResponseUtils.bytesToWebResponse(pdfBytes, filename);
}
@AutoJobPostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE, value = "/pdf/cbr")
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/pdf/cbr",
resourceWeight = ResourceWeight.LARGE_WEIGHT)
@Operation(
summary = "Convert PDF to CBR comic book archive",
description =
@@ -141,7 +141,8 @@ public class AttachmentController {
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/extract-attachments")
value = "/extract-attachments",
resourceWeight = ResourceWeight.SMALL_WEIGHT)
@Operation(
summary = "Extract attachments from PDF",
description =
@@ -176,7 +177,10 @@ public class AttachmentController {
}
}
@AutoJobPostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE, value = "/list-attachments")
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/list-attachments",
resourceWeight = ResourceWeight.SMALL_WEIGHT)
@Operation(
summary = "List attachments in PDF",
description =
@@ -193,7 +197,8 @@ public class AttachmentController {
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/rename-attachment")
value = "/rename-attachment",
resourceWeight = ResourceWeight.SMALL_WEIGHT)
@StandardPdfResponse
@Operation(
summary = "Rename attachment in PDF",
@@ -228,7 +233,8 @@ public class AttachmentController {
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/delete-attachment")
value = "/delete-attachment",
resourceWeight = ResourceWeight.SMALL_WEIGHT)
@StandardPdfResponse
@Operation(
summary = "Delete attachment from PDF",
@@ -12,6 +12,8 @@ import org.springframework.web.bind.annotation.RequestParam;
import io.swagger.v3.oas.annotations.Hidden;
import jakarta.servlet.http.HttpServletRequest;
import lombok.extern.slf4j.Slf4j;
import stirling.software.SPDF.config.EndpointConfiguration;
@@ -91,6 +93,44 @@ public class ConfigController {
return null;
}
/**
* Resolve the frontend URL the client should advertise to phones / share-link recipients.
* Priority: explicit system.frontendUrl, then the Host the user is already using to reach this
* server (works for Docker, reverse proxies, and bare-metal LANs), then a detected site-local
* IPv4, then empty.
*/
// visible for testing
String resolveFrontendUrl(HttpServletRequest request, AppConfig appConfig) {
String configured = applicationProperties.getSystem().getFrontendUrl();
if (configured != null && !configured.isBlank()) {
return configured;
}
if (request != null) {
String host = request.getServerName();
if (host != null && !host.isBlank() && !isLoopbackHost(host)) {
String scheme = request.getScheme();
int port = request.getServerPort();
boolean defaultPort =
("http".equals(scheme) && port == 80)
|| ("https".equals(scheme) && port == 443);
return defaultPort ? scheme + "://" + host : scheme + "://" + host + ":" + port;
}
}
String localIp = GeneralUtils.getLocalNetworkIp();
if (localIp != null) {
String scheme = appConfig.getBackendUrl().startsWith("https") ? "https" : "http";
return scheme + "://" + localIp + ":" + appConfig.getServerPort();
}
return "";
}
private static boolean isLoopbackHost(String host) {
return "localhost".equalsIgnoreCase(host)
|| "127.0.0.1".equals(host)
|| "::1".equals(host)
|| "0:0:0:0:0:0:0:1".equals(host);
}
/** Check if running Enterprise edition dynamically. */
private Boolean isRunningEE() {
// Use LicenseService for fresh license status if available
@@ -107,7 +147,7 @@ public class ConfigController {
}
@GetMapping("/app-config")
public ResponseEntity<Map<String, Object>> getAppConfig() {
public ResponseEntity<Map<String, Object>> getAppConfig(HttpServletRequest request) {
Map<String, Object> configData = new HashMap<>();
try {
@@ -124,17 +164,7 @@ public class ConfigController {
configData.put("serverPort", appConfig.getServerPort());
String frontendUrl = applicationProperties.getSystem().getFrontendUrl();
if ((frontendUrl == null || frontendUrl.isBlank())
&& Boolean.parseBoolean(
System.getProperty("STIRLING_PDF_TAURI_MODE", "false"))) {
String localIp = GeneralUtils.getLocalNetworkIp();
if (localIp != null) {
String scheme =
appConfig.getBackendUrl().startsWith("https") ? "https" : "http";
frontendUrl = scheme + "://" + localIp + ":" + appConfig.getServerPort();
}
}
configData.put("frontendUrl", frontendUrl != null ? frontendUrl : "");
configData.put("frontendUrl", resolveFrontendUrl(request, appConfig));
// Add mobile scanner settings
configData.put(
@@ -277,6 +307,9 @@ public class ConfigController {
// Premium/Enterprise settings
configData.put("premiumEnabled", applicationProperties.getPremium().isEnabled());
// AI Engine settings
configData.put("aiEngineEnabled", applicationProperties.getAiEngine().isEnabled());
// Timestamp TSA settings — single source of truth for presets + admin URLs
ApplicationProperties.Security.Timestamp tsConfig =
applicationProperties.getSecurity().getTimestamp();
@@ -160,14 +160,19 @@ public class ReactRoutingController {
return ResponseEntity.ok().contentType(MediaType.TEXT_HTML).body(cachedCallbackHtml);
}
// `files` was historically a backend static-asset directory and was therefore
// in the exclusion list - removing it lets /files and /files/<folder-uuid>
// forward to the SPA index.html, which is what FileManagerView expects.
// (Real storage endpoints live under /api/v1/storage/files, already
// excluded by the leading `api` token in the same regex.)
@GetMapping(
"/{path:^(?!api|static|robots\\.txt|favicon\\.ico|manifest.*\\.json|pipeline|pdfjs|pdfjs-legacy|pdfium|vendor|fonts|images|files|css|js|assets|locales|modern-logo|classic-logo|Login|og_images|samples)[^\\.]*$}")
"/{path:^(?!api|static|robots\\.txt|favicon\\.ico|manifest.*\\.json|pipeline|pdfjs|pdfjs-legacy|pdfium|vendor|fonts|images|css|js|assets|locales|modern-logo|classic-logo|Login|og_images|samples)[^\\.]*$}")
public ResponseEntity<String> forwardRootPaths(HttpServletRequest request) throws IOException {
return serveIndexHtml(request);
}
@GetMapping(
"/{path:^(?!api|static|pipeline|pdfjs|pdfjs-legacy|pdfium|vendor|fonts|images|files|css|js|assets|locales|modern-logo|classic-logo|Login|og_images|samples)[^\\.]*}/{subpath:^(?!.*\\.).*$}")
"/{path:^(?!api|static|pipeline|pdfjs|pdfjs-legacy|pdfium|vendor|fonts|images|css|js|assets|locales|modern-logo|classic-logo|Login|og_images|samples)[^\\.]*}/{subpath:^(?!.*\\.).*$}")
public ResponseEntity<String> forwardNestedPaths(HttpServletRequest request)
throws IOException {
return serveIndexHtml(request);
@@ -22,6 +22,7 @@ import org.springframework.web.bind.annotation.ExceptionHandler;
import org.springframework.web.bind.annotation.RestControllerAdvice;
import org.springframework.web.multipart.MaxUploadSizeExceededException;
import org.springframework.web.multipart.support.MissingServletRequestPartException;
import org.springframework.web.server.ResponseStatusException;
import org.springframework.web.servlet.NoHandlerFoundException;
import jakarta.servlet.http.HttpServletRequest;
@@ -196,12 +197,12 @@ public class GlobalExceptionHandler {
/**
* Checks whether the given IOException indicates that the client disconnected before the
* response could be written (broken pipe, connection reset, etc.). When this happens there is
* no point in serialising a {@link ProblemDetail} body because the socket is already closed —
* no point in serialising a {@link ProblemDetail} body because the socket is already closed -
* and attempting to do so may trigger a secondary {@code HttpMessageNotWritableException} if
* the response Content-Type was already committed as a non-JSON type (e.g. image/png).
*/
private static boolean isClientDisconnectException(IOException ex) {
// Walk the causal chain — Jetty/Tomcat may wrap the low-level SocketException
// Walk the causal chain - Jetty/Tomcat may wrap the low-level SocketException
Throwable current = ex;
while (current != null) {
String msg = current.getMessage();
@@ -1040,6 +1041,43 @@ public class GlobalExceptionHandler {
* @param request the HTTP servlet request
* @return ProblemDetail with appropriate HTTP status
*/
/**
* Handle ResponseStatusException explicitly so its embedded HTTP status reaches the client
* instead of being swallowed by the {@code RuntimeException} catch-all (which would downgrade
* every controller-thrown 400/404/409 to a generic 500). Folder/file storage controllers and
* any other code that throws {@code ResponseStatusException} relies on this handler taking
* precedence.
*/
@ExceptionHandler(ResponseStatusException.class)
public ResponseEntity<ProblemDetail> handleResponseStatusException(
ResponseStatusException ex, HttpServletRequest request) {
HttpStatus status =
HttpStatus.resolve(ex.getStatusCode().value()) != null
? HttpStatus.valueOf(ex.getStatusCode().value())
: HttpStatus.INTERNAL_SERVER_ERROR;
String reason = ex.getReason() != null ? ex.getReason() : status.getReasonPhrase();
ProblemDetail problemDetail = createBaseProblemDetail(status, reason, request);
problemDetail.setType(URI.create("/errors/" + status.value()));
problemDetail.setTitle(status.getReasonPhrase());
problemDetail.setProperty("title", status.getReasonPhrase());
// 5xx is operator-relevant; 4xx is a normal client-rejection - log at the right level.
if (status.is5xxServerError()) {
log.error(
"ResponseStatusException {} at {}: {}",
status.value(),
request.getRequestURI(),
reason,
ex);
} else {
log.debug(
"ResponseStatusException {} at {}: {}",
status.value(),
request.getRequestURI(),
reason);
}
return ResponseEntity.status(status).contentType(PROBLEM_JSON).body(problemDetail);
}
@ExceptionHandler(RuntimeException.class)
public ResponseEntity<ProblemDetail> handleRuntimeException(
RuntimeException ex, HttpServletRequest request) {
@@ -18,4 +18,11 @@ public class PDFWithPageSize extends PDFFile {
requiredMode = Schema.RequiredMode.REQUIRED,
allowableValues = {"A0", "A1", "A2", "A3", "A4", "A5", "A6", "LETTER", "LEGAL", "KEEP"})
private String pageSize;
@Schema(
description =
"Orientation to apply to the target page size. Ignored when pageSize is KEEP.",
defaultValue = "PORTRAIT",
allowableValues = {"PORTRAIT", "LANDSCAPE"})
private String orientation = "PORTRAIT";
}
@@ -13,6 +13,7 @@ import lombok.RequiredArgsConstructor;
import stirling.software.SPDF.config.swagger.MarkdownConversionResponse;
import stirling.software.common.annotations.AutoJobPostMapping;
import stirling.software.common.annotations.api.ConvertApi;
import stirling.software.common.enumeration.ResourceWeight;
import stirling.software.common.model.api.PDFFile;
import stirling.software.common.util.PDFToFile;
import stirling.software.common.util.TempFileManager;
@@ -23,7 +24,10 @@ public class ConvertPDFToMarkdown {
private final TempFileManager tempFileManager;
@AutoJobPostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE, value = "/pdf/markdown")
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/pdf/markdown",
resourceWeight = ResourceWeight.MEDIUM_WEIGHT)
@MarkdownConversionResponse
@Operation(
summary = "Convert PDF to Markdown",
@@ -4,7 +4,6 @@ import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.MediaType;
@@ -23,10 +22,6 @@ import jakarta.servlet.http.HttpServletRequest;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.cluster.ClusterBackplane;
import stirling.software.common.cluster.JobStore;
import stirling.software.common.cluster.JobStoreEntry;
import stirling.software.common.cluster.StickyMissRecorder;
import stirling.software.common.model.job.JobResult;
import stirling.software.common.model.job.ResultFile;
import stirling.software.common.service.FileStorage;
@@ -47,22 +42,10 @@ public class JobController {
private final FileStorage fileStorage;
private final JobQueue jobQueue;
private final HttpServletRequest request;
private final ClusterBackplane clusterBackplane;
private final JobStore jobStore;
/**
* Process-local short-TTL cache fronting {@link JobStore#get(String)} on the sticky-410 path.
* Without this every result download / status poll fires a Valkey HGETALL which doubles RTT on
* the hot path when the same client re-requests the same job within seconds.
*/
private final JobOwnershipCache ownershipCache = new JobOwnershipCache();
@Autowired(required = false)
private JobOwnershipService jobOwnershipService;
@Autowired(required = false)
private StickyMissRecorder stickyMissRecorder;
/**
* Get the status of a job
*
@@ -72,14 +55,6 @@ public class JobController {
@GetMapping("/job/{jobId}")
@Operation(summary = "Get job status")
public ResponseEntity<?> getJobStatus(@PathVariable("jobId") String jobId) {
// Sticky-410 must precede user-auth (403): a non-owner node has no way to verify
// ownership for a job it doesn't own, and a 403 here would leak job existence to
// unauthorized callers. Return 410 first so the LB re-routes to the owner.
Optional<ResponseEntity<?>> peerOwned = guardNonOwner(jobId);
if (peerOwned.isPresent()) {
return peerOwned.get();
}
// Validate job ownership
if (!validateJobAccess(jobId)) {
log.warn("Unauthorized attempt to access job status: {}", jobId);
@@ -116,14 +91,6 @@ public class JobController {
@GetMapping("/job/{jobId}/result")
@Operation(summary = "Get job result")
public ResponseEntity<?> getJobResult(@PathVariable("jobId") String jobId) {
// Sticky-410 must precede user-auth (403): a non-owner node has no way to verify
// ownership for a job it doesn't own, and a 403 here would leak job existence to
// unauthorized callers. Return 410 first so the LB re-routes to the owner.
Optional<ResponseEntity<?>> peerOwned = guardNonOwner(jobId);
if (peerOwned.isPresent()) {
return peerOwned.get();
}
// Validate job ownership
if (!validateJobAccess(jobId)) {
log.warn("Unauthorized attempt to access job result: {}", jobId);
@@ -158,14 +125,11 @@ public class JobController {
result.getAllResultFiles()));
}
// Handle single file (download directly). Cross-node ownership was already resolved
// at the top of this method, so reaching here means we ARE the owner (or single-node)
// and the bytes live on our local disk.
// Handle single file (download directly)
if (result.hasFiles() && !result.hasMultipleFiles()) {
try {
List<ResultFile> files = result.getAllResultFiles();
ResultFile singleFile = files.get(0);
byte[] fileContent = fileStorage.retrieveBytes(singleFile.getFileId());
return ResponseEntity.ok()
.header("Content-Type", singleFile.getContentType())
@@ -199,15 +163,6 @@ public class JobController {
public ResponseEntity<?> cancelJob(@PathVariable("jobId") String jobId) {
log.debug("Request to cancel job: {}", jobId);
// Sticky-410 must precede user-auth (403): a non-owner node has no way to verify
// ownership for a job it doesn't own, and a 403 here would leak job existence to
// unauthorized callers. Return 410 first so the LB re-routes to the owner who can
// actually cancel.
Optional<ResponseEntity<?>> peerOwned = guardNonOwner(jobId);
if (peerOwned.isPresent()) {
return peerOwned.get();
}
// Validate job ownership
if (!validateJobAccess(jobId)) {
log.warn("Unauthorized attempt to cancel job: {}", jobId);
@@ -246,9 +201,7 @@ public class JobController {
"queuePosition",
queuePosition >= 0 ? queuePosition : "n/a"));
} else {
// Job not found or already complete. Cross-node ownership was already resolved at
// the top of this method (sticky-410 precedes user-auth), so any peer-owned case
// has been returned already; reaching here means we ARE the owner (or single-node).
// Job not found or already complete
JobResult result = taskManager.getJobResult(jobId);
if (result == null) {
return ResponseEntity.notFound().build();
@@ -271,14 +224,6 @@ public class JobController {
@GetMapping("/job/{jobId}/result/files")
@Operation(summary = "Get job result files")
public ResponseEntity<?> getJobFiles(@PathVariable("jobId") String jobId) {
// Sticky-410 must precede user-auth (403): a non-owner node has no way to verify
// ownership for a job it doesn't own, and a 403 here would leak job existence to
// unauthorized callers. Return 410 first so the LB re-routes to the owner.
Optional<ResponseEntity<?>> peerOwned = guardNonOwner(jobId);
if (peerOwned.isPresent()) {
return peerOwned.get();
}
// Validate job ownership
if (!validateJobAccess(jobId)) {
log.warn("Unauthorized attempt to access job files: {}", jobId);
@@ -322,14 +267,6 @@ public class JobController {
return ResponseEntity.notFound().build();
}
// Sticky-410 must precede user-auth (403): a non-owner node has no way to verify
// ownership for a job it doesn't own, and a 403 here would leak file existence to
// unauthorized callers. Return 410 first so the LB re-routes to the owner.
Optional<ResponseEntity<?>> notOwner = guardNonOwner(jobKey);
if (notOwner.isPresent()) {
return notOwner.get();
}
if (!validateJobAccess(jobKey)) {
log.warn("Unauthorized attempt to access file metadata: {}", fileId);
return ResponseEntity.status(403)
@@ -386,21 +323,15 @@ public class JobController {
return ResponseEntity.notFound().build();
}
// Sticky-410 must precede the user-auth (403) check: a non-owner node has no way to
// verify ownership for a job it doesn't own, and a 403 here would leak file existence
// to unauthorized callers. Return 410 first so the LB re-routes to the owner where
// the real auth check can run.
Optional<ResponseEntity<?>> notOwner = guardNonOwner(jobKey);
if (notOwner.isPresent()) {
return notOwner.get();
}
if (!validateJobAccess(jobKey)) {
log.warn("Unauthorized attempt to download file: {}", fileId);
return ResponseEntity.status(403)
.body(Map.of("message", "You are not authorized to access this file"));
}
// Retrieve file content
byte[] fileContent = fileStorage.retrieveBytes(fileId);
// Find the file metadata from any job that contains this file
// This is for getting the original filename and content type
ResultFile resultFile = taskManager.findResultFileByFileId(fileId);
@@ -411,9 +342,6 @@ public class JobController {
? resultFile.getContentType()
: MediaType.APPLICATION_OCTET_STREAM_VALUE;
// Retrieve file content from local disk
byte[] fileContent = fileStorage.retrieveBytes(fileId);
return ResponseEntity.ok()
.header("Content-Type", contentType)
.header("Content-Disposition", createContentDispositionHeader(fileName))
@@ -428,76 +356,6 @@ public class JobController {
return jobOwnershipService != null;
}
/**
* Returns {@code 410 Gone} with {@code {message, ownedBy, currentNode}} and {@code Retry-After:
* 0} when the job is owned by a peer node. Returns {@link Optional#empty()} when we are the
* owner, when cluster mode is off / JobStore has no entry, or when {@code owningNodeId} is
* blank (caller proceeds with its normal not-found / 200 path).
*
* <p>Wraps the {@link JobStore#get(String)} call in a short-TTL local cache and a defensive
* try/catch so that Valkey RTT cost is not multiplied by every download retry and so that a
* Valkey timeout falls through to the local-disk path instead of surfacing as 500.
*/
private Optional<ResponseEntity<?>> guardNonOwner(String jobId) {
if (clusterBackplane == null || jobStore == null) {
return Optional.empty();
}
Optional<JobStoreEntry> entry;
Optional<Optional<JobStoreEntry>> cached = ownershipCache.get(jobId);
if (cached.isPresent()) {
entry = cached.get();
} else {
try {
entry = jobStore.get(jobId);
} catch (RuntimeException ex) {
// Valkey unavailable / timeout: treat as "no cluster-visible entry" so the request
// can proceed to the local-disk path. Surfacing a 500 here would break every
// download attempt during a brief Valkey blip; the worst case if we miss a real
// peer-owned entry is one wasted round trip + a 404 from the local node.
log.warn(
"JobStore lookup failed for jobId={} - treating as not-found and falling"
+ " through to local path: {}",
jobId,
ex.getMessage());
return Optional.empty();
}
ownershipCache.put(jobId, entry);
}
if (entry.isEmpty()) {
return Optional.empty();
}
String owner = entry.get().owningNodeId();
if (owner == null || owner.isBlank()) {
return Optional.empty();
}
String localId = clusterBackplane.localNodeId();
if (owner.equals(localId)) {
return Optional.empty();
}
log.info(
"Sticky-session miss for jobId={} (owner={}, local={}); returning 410 so client"
+ " retries via LB affinity",
jobId,
owner,
localId);
if (stickyMissRecorder != null) {
stickyMissRecorder.recordStickyMiss();
}
return Optional.of(
ResponseEntity.status(410)
.header("Retry-After", "0")
.body(
Map.of(
"message",
"Result lives on another node. Retry to be routed there"
+ " by the load balancer's sticky-session"
+ " affinity, or re-run the job.",
"ownedBy",
owner,
"currentNode",
localId == null ? "" : localId)));
}
/**
* Create Content-Disposition header with UTF-8 filename support
*
@@ -1,51 +0,0 @@
package stirling.software.common.controller;
import java.util.Optional;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
import stirling.software.common.cluster.JobStoreEntry;
/**
* Process-local TTL cache for {@link JobStoreEntry} lookups to suppress redundant Valkey HGETALL
* round-trips on the hot result-download path (sticky-410 ownership check).
*
* <p>5 second TTL is short enough that a job's lifecycle transitions (RUNNING -> COMPLETE -> TTL
* expiry) propagate to all nodes within the LB's sticky-session window, and short enough that a
* mistakenly-cached "not found" recovers quickly when an entry actually shows up. Cap the map at
* 2048 entries to bound memory; eviction is best-effort (clear-and-restart) since the cache is
* advisory.
*/
final class JobOwnershipCache {
private static final long TTL_NANOS = 5L * 1_000_000_000L; // 5 s
private static final int MAX_ENTRIES = 2048;
private final ConcurrentMap<String, Entry> entries = new ConcurrentHashMap<>();
Optional<Optional<JobStoreEntry>> get(String jobId) {
Entry e = entries.get(jobId);
if (e == null) {
return Optional.empty();
}
if (System.nanoTime() - e.storedAtNanos > TTL_NANOS) {
entries.remove(jobId, e);
return Optional.empty();
}
return Optional.of(e.value);
}
void put(String jobId, Optional<JobStoreEntry> value) {
if (entries.size() >= MAX_ENTRIES) {
// Best-effort eviction; under burst the cache simply rebuilds.
entries.clear();
}
entries.put(jobId, new Entry(value, System.nanoTime()));
}
void invalidate(String jobId) {
entries.remove(jobId);
}
private record Entry(Optional<JobStoreEntry> value, long storedAtNanos) {}
}
@@ -19,10 +19,6 @@ logging.level.stirling.software.common.service.TaskManager=INFO
spring.jpa.open-in-view=false
server.forward-headers-strategy=NATIVE
# Prevent Spring Boot auto-configuring a session repository from spring-session-data-redis being on
# the classpath. ClusterSessionConfiguration enables Redis-backed sessions when cluster.enabled=true.
spring.session.store-type=none
# Enable HTTP/2 for improved performance (multiplexed streams, header compression)
server.http2.enabled=true
@@ -20,6 +20,7 @@ security:
password: "" # initial password for the first login
oauth2:
enabled: false # set to 'true' to enable login (Note: enableLogin must also be 'true' for this to work)
debugLogging: false # set to 'true' to log full ID token and UserInfo claims during OAuth2/OIDC login. Use this to diagnose claim issues (e.g. "Attribute value for 'email' cannot be null" with ADFS). WARNING: writes PII (sub, email, name) to logs; disable after troubleshooting.
client:
keycloak:
issuer: "" # URL of the Keycloak realm's OpenID Connect Discovery endpoint
@@ -93,8 +94,8 @@ premium:
key: 00000000-0000-0000-0000-000000000000
enabled: false # Enable license key checks for pro/enterprise features
proFeatures:
SSOAutoLogin: false
CustomMetadata:
ssoAutoLogin: false
customMetadata:
autoUpdateMetadata: false
author: username
creator: Stirling-PDF
@@ -245,6 +246,39 @@ storage:
provider: local # storage provider: 'local' for filesystem storage, 'database' for DB-backed storage
local:
basePath: './storage' # base directory for stored files
# ====================================================================================
# S3-COMPATIBLE OBJECT STORAGE - PRO / ENTERPRISE LICENSE REQUIRED
# storage.provider=s3, storage.provider=database, and cluster.artifactStore=s3 all
# require a valid Pro or Enterprise license.
# ====================================================================================
# Used when provider=s3 (persistent user uploads) and/or cluster.artifactStore=s3
# (transient cluster artifacts). The two consumers share this block.
# Vendor cheat sheet (set the highlighted flags to taste):
# AWS S3 -> endpoint='' region='<your-region>' pathStyleAccess=false
# Cloudflare R2 -> endpoint='https://<acct>.r2.cloudflarestorage.com' region='auto'
# pathStyleAccess=false; if uploads fail with 'unsupported header
# x-amz-checksum-*' set requestChecksumCalculation=WHEN_REQUIRED
# Supabase Storage -> endpoint='https://<project>.supabase.co/storage/v1/s3'
# region='<project-region>' pathStyleAccess=true
# (filenames with non-ASCII display fine - the storage key is opaque)
# MinIO (in-cluster) -> endpoint='http://minio:9000' region='us-east-1'
# pathStyleAccess=true allowPrivateEndpoints=true
# Backblaze B2 -> endpoint='https://s3.<region>.backblazeb2.com'
# If on a B2 deployment older than July-2025 and uploads return
# 'Unsupported header x-amz-checksum-crc32', set
# requestChecksumCalculation=WHEN_REQUIRED
# DigitalOcean Spaces -> endpoint='https://<region>.digitaloceanspaces.com'
# Note: 5GB per-object cap (regardless of multipart)
s3:
endpoint: "" # blank = use AWS regional default; otherwise full URL incl. https://
bucket: "" # required when provider=s3 or cluster.artifactStore=s3
region: us-east-1
accessKey: "" # blank = fall back to AWS DefaultCredentialsProvider (env / profile / IMDS)
secretKey: ""
pathStyleAccess: false # true for MinIO and Supabase; false for AWS/R2/most CDNs
allowPrivateEndpoints: false # true required when endpoint resolves to a private/loopback IP (e.g. in-cluster MinIO). SSRF guard - leave false for any internet-facing vendor.
requestChecksumCalculation: WHEN_SUPPORTED # WHEN_SUPPORTED|WHEN_REQUIRED|DISABLED. Set WHEN_REQUIRED if your vendor rejects auto-added x-amz-checksum-* headers (older Backblaze B2, some R2 corner cases).
responseChecksumValidation: WHEN_SUPPORTED # WHEN_SUPPORTED|WHEN_REQUIRED|DISABLED. Set WHEN_REQUIRED if you see false-positive checksum-mismatch errors on GET from a vendor that never returns checksum headers.
quotas:
maxStorageMbPerUser: -1 # Max storage per user in MB; -1 disables per-user cap
maxStorageMbTotal: -1 # Max storage across all users in MB; -1 disables total cap
@@ -335,6 +369,8 @@ cluster:
enabled: false # Master switch. 'false' (default) wires the in-process backplane and skips all cluster checks. Single-instance installs do not need to change anything here.
backplane: inprocess # Backplane implementation: 'inprocess' (single JVM only) or 'valkey' (multi-node via Valkey/Redis)
artifactStore: local # Transient cluster job-artifact backend: 'local' (per-node disk; single-node only) or 's3' (shared object store; required for multi-node). Distinct from 'storage.provider' which controls persistent user uploads - when both are 's3' they share the storage.s3.* credentials block. Multi-node deployments MUST set this to 's3'.
s3:
keyPrefix: transient/ # Bucket key prefix used by the cluster artifact store when artifactStore=s3. Trailing slash recommended. Lets a single bucket host both persistent uploads (storage.s3.*) and transient job artifacts under separate prefixes.
valkey:
url: "" # Valkey/Redis URL, e.g. 'redis://valkey:6379' or 'rediss://...' for TLS. Required when enabled=true and backplane=valkey.
tls:
@@ -0,0 +1,132 @@
package stirling.software.SPDF.config;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.io.IOException;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;
import org.junit.jupiter.api.Test;
import org.springframework.core.io.Resource;
import org.springframework.core.io.support.PathMatchingResourcePatternResolver;
import org.springframework.core.io.support.ResourcePatternResolver;
import org.springframework.core.type.classreading.CachingMetadataReaderFactory;
import org.springframework.core.type.classreading.MetadataReader;
import org.springframework.core.type.classreading.MetadataReaderFactory;
import org.springframework.core.type.filter.TypeFilter;
import stirling.software.common.annotations.AutoJobPostMapping;
/**
* Build-time guardrail: every {@link AutoJobPostMapping} method must declare an explicit {@code
* resourceWeight}.
*
* <p>The credits interceptor multiplies {@code resourceWeight} into the per-call charge. An
* endpoint that falls through to the annotation default produces a charge derived from a value
* nobody chose — silently under- or over-billing depending on the endpoint's true cost. Forcing
* each method to pick a value from {@link stirling.software.common.enumeration.ResourceWeight}
* keeps the choice deliberate.
*
* <p>The annotation's default is {@link Integer#MIN_VALUE} (a sentinel). Runtime readers clamp the
* value into {@code [1, 100]}, so a missed declaration can't crash production — this test is the
* contract, the clamp is the safety net.
*
* <p>Lives in {@code :stirling-pdf} (core) because that's the module whose compile classpath
* transitively sees every other module's controllers ({@code :common}, {@code :proprietary}, and
* {@code :saas} when enabled).
*/
class AutoJobPostMappingWeightTest {
private static final String SCAN_BASE_PACKAGE = "stirling.software";
@Test
void everyAutoJobPostMappingDeclaresExplicitResourceWeight() throws Exception {
List<String> offenders = findOffendingMethods();
assertTrue(
offenders.isEmpty(),
() ->
"The following @AutoJobPostMapping methods do not declare an explicit"
+ " resourceWeight. Pick a value from"
+ " stirling.software.common.enumeration.ResourceWeight (SMALL,"
+ " MEDIUM, LARGE, XLARGE) and add it to the annotation:\n - "
+ String.join("\n - ", offenders));
}
private List<String> findOffendingMethods() throws IOException, ClassNotFoundException {
List<String> offenders = new ArrayList<>();
for (Class<?> candidate : scanForCandidateClasses()) {
for (Method method : candidate.getDeclaredMethods()) {
AutoJobPostMapping annotation = method.getAnnotation(AutoJobPostMapping.class);
if (annotation == null) {
continue;
}
if (annotation.resourceWeight() == Integer.MIN_VALUE) {
offenders.add(candidate.getName() + "#" + method.getName());
}
}
}
return offenders;
}
/**
* Returns every class under {@link #SCAN_BASE_PACKAGE} that has an @AutoJobPostMapping method.
*/
private List<Class<?>> scanForCandidateClasses() throws IOException, ClassNotFoundException {
ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver();
MetadataReaderFactory metadataReaderFactory = new CachingMetadataReaderFactory(resolver);
String pattern = "classpath*:" + SCAN_BASE_PACKAGE.replace('.', '/') + "/**/*.class";
Resource[] resources = resolver.getResources(pattern);
// Pre-filter by reading annotation metadata from the class file so we don't have to load
// every class on the test classpath just to find the few that are annotated.
TypeFilter mentionsAutoJobPostMapping =
(reader, factory) ->
reader.getAnnotationMetadata()
.getAnnotatedMethods(AutoJobPostMapping.class.getName())
.size()
> 0;
List<Class<?>> matches = new ArrayList<>();
for (Resource resource : resources) {
if (!resource.isReadable()) {
continue;
}
MetadataReader reader = metadataReaderFactory.getMetadataReader(resource);
if (!mentionsAutoJobPostMapping.match(reader, metadataReaderFactory)) {
continue;
}
matches.add(Class.forName(reader.getClassMetadata().getClassName()));
}
return matches;
}
/**
* Sanity check that the classpath scan returns non-empty; otherwise the main test passes
* vacuously.
*/
@Test
void scannerFindsAtLeastOneAutoJobPostMapping() throws Exception {
long count =
scanForCandidateClasses().stream()
.flatMap(c -> java.util.Arrays.stream(c.getDeclaredMethods()))
.filter(m -> m.isAnnotationPresent(AutoJobPostMapping.class))
.count();
assertTrue(
count > 10,
() ->
"Expected the classpath scan to find many @AutoJobPostMapping methods but"
+ " found only "
+ count
+ ". Scanner regression?");
}
@SuppressWarnings("unused")
private static String describeCandidates(List<Class<?>> candidates) {
return candidates.stream().map(Class::getName).collect(Collectors.joining(", "));
}
}
@@ -237,7 +237,8 @@ class ScalePagesControllerTest {
ScalePagesRequest request = new ScalePagesRequest();
request.setFileInput(file);
request.setPageSize("A4_LANDSCAPE");
request.setPageSize("A4");
request.setOrientation("LANDSCAPE");
request.setScaleFactor(1.0f);
setupFactory();
@@ -15,10 +15,14 @@ import org.springframework.context.ApplicationContext;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import jakarta.servlet.http.HttpServletRequest;
import stirling.software.SPDF.config.EndpointConfiguration;
import stirling.software.SPDF.config.EndpointConfiguration.DisableReason;
import stirling.software.SPDF.config.EndpointConfiguration.EndpointAvailability;
import stirling.software.common.configuration.AppConfig;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.common.model.ApplicationProperties.System;
import stirling.software.common.service.LicenseServiceInterface;
import stirling.software.common.service.ServerCertificateServiceInterface;
import stirling.software.common.service.UserServiceInterface;
@@ -173,4 +177,71 @@ class ConfigControllerTest {
assertEquals(HttpStatus.OK, response.getStatusCode());
verify(endpointConfiguration).getAllEndpoints();
}
@Test
void resolveFrontendUrl_prefersExplicitConfiguredValue() {
System sys = mock(System.class);
when(applicationProperties.getSystem()).thenReturn(sys);
when(sys.getFrontendUrl()).thenReturn("https://pdf.example.com");
// Request would say something else, but configured wins.
HttpServletRequest req = mock(HttpServletRequest.class);
AppConfig appConfig = mock(AppConfig.class);
assertEquals(
"https://pdf.example.com", configController.resolveFrontendUrl(req, appConfig));
}
@Test
void resolveFrontendUrl_usesRequestHostWhenNotConfigured() {
System sys = mock(System.class);
when(applicationProperties.getSystem()).thenReturn(sys);
when(sys.getFrontendUrl()).thenReturn(null);
HttpServletRequest req = mock(HttpServletRequest.class);
when(req.getServerName()).thenReturn("192.168.1.100");
when(req.getScheme()).thenReturn("http");
when(req.getServerPort()).thenReturn(8080);
assertEquals(
"http://192.168.1.100:8080",
configController.resolveFrontendUrl(req, mock(AppConfig.class)));
}
@Test
void resolveFrontendUrl_elidesDefaultHttpsPort() {
System sys = mock(System.class);
when(applicationProperties.getSystem()).thenReturn(sys);
when(sys.getFrontendUrl()).thenReturn("");
HttpServletRequest req = mock(HttpServletRequest.class);
when(req.getServerName()).thenReturn("pdf.example.com");
when(req.getScheme()).thenReturn("https");
when(req.getServerPort()).thenReturn(443);
assertEquals(
"https://pdf.example.com",
configController.resolveFrontendUrl(req, mock(AppConfig.class)));
}
@Test
void resolveFrontendUrl_fallsThroughOnLoopbackHost() {
System sys = mock(System.class);
when(applicationProperties.getSystem()).thenReturn(sys);
when(sys.getFrontendUrl()).thenReturn(null);
HttpServletRequest req = mock(HttpServletRequest.class);
when(req.getServerName()).thenReturn("localhost");
AppConfig appConfig = mock(AppConfig.class);
when(appConfig.getBackendUrl()).thenReturn("http://localhost:8080");
when(appConfig.getServerPort()).thenReturn("8080");
// Detected IP (if any) wins over loopback request host. We can't assert the
// exact value (depends on the host running the test) but we can assert it
// never returns "localhost".
String result = configController.resolveFrontendUrl(req, appConfig);
assertNotNull(result);
assertFalse(result.contains("localhost"));
}
}
@@ -0,0 +1,93 @@
package stirling.software.common.configuration;
import static org.junit.jupiter.api.Assertions.*;
import static org.mockito.Mockito.mockStatic;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.mockito.MockedStatic;
import stirling.software.common.util.GeneralUtils;
import stirling.software.common.util.YamlHelper;
/**
* End-to-end check of the container-restart path. {@link ConfigInitializer#ensureConfigExists()} is
* what runs on every startup, merging the on-disk settings.yml with the bundled
* settings.yml.template. These tests exercise it against the real template on the classpath to
* prove admin-saved proFeatures values survive a restart - the bug behind "the SSO auto-login
* button resets every time the container resets".
*/
class ConfigInitializerRestartTest {
private static String read(Path settings, String... keyPath) throws IOException {
return String.valueOf(new YamlHelper(settings).getValueByExactKeyPath(keyPath));
}
@Test
void ssoAutoLoginAndCustomMetadata_persistAcrossRestart(@TempDir Path tmp) throws Exception {
Path settings = tmp.resolve("settings.yml");
Path custom = tmp.resolve("custom_settings.yml");
try (MockedStatic<InstallationPathConfig> paths =
mockStatic(InstallationPathConfig.class)) {
paths.when(InstallationPathConfig::getSettingsPath).thenReturn(settings.toString());
paths.when(InstallationPathConfig::getCustomSettingsPath).thenReturn(custom.toString());
ConfigInitializer init = new ConfigInitializer();
// First boot: settings.yml created from the bundled template (camelCase, default off).
init.ensureConfigExists();
assertEquals("false", read(settings, "premium", "proFeatures", "ssoAutoLogin"));
// Admin enables SSO auto-login and edits custom metadata via the exact save path the
// admin settings controller uses.
GeneralUtils.saveKeyToSettings("premium.proFeatures.ssoAutoLogin", true);
GeneralUtils.saveKeyToSettings("premium.proFeatures.customMetadata.author", "acme");
// Container restart: ensureConfigExists merges the saved file with the template again.
init.ensureConfigExists();
assertEquals("true", read(settings, "premium", "proFeatures", "ssoAutoLogin"));
assertEquals(
"acme", read(settings, "premium", "proFeatures", "customMetadata", "author"));
}
}
@Test
void legacyPascalCaseConfig_isMigratedAndPreservedOnRestart(@TempDir Path tmp)
throws Exception {
Path settings = tmp.resolve("settings.yml");
Path custom = tmp.resolve("custom_settings.yml");
try (MockedStatic<InstallationPathConfig> paths =
mockStatic(InstallationPathConfig.class)) {
paths.when(InstallationPathConfig::getSettingsPath).thenReturn(settings.toString());
paths.when(InstallationPathConfig::getCustomSettingsPath).thenReturn(custom.toString());
ConfigInitializer init = new ConfigInitializer();
// Seed a full settings.yml as an OLD install would have written it: PascalCase keys
// with
// SSO auto-login enabled.
init.ensureConfigExists();
String legacy =
Files.readString(settings)
.replace("ssoAutoLogin: false", "SSOAutoLogin: true")
.replace("customMetadata:", "CustomMetadata:");
Files.writeString(settings, legacy);
// Upgrade restart.
init.ensureConfigExists();
// Value carried forward onto the new camelCase key; the legacy PascalCase key is gone.
assertEquals("true", read(settings, "premium", "proFeatures", "ssoAutoLogin"));
assertNull(
new YamlHelper(settings)
.getValueByExactKeyPath("premium", "proFeatures", "SSOAutoLogin"));
}
}
}
@@ -1,502 +0,0 @@
package stirling.software.common.controller;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertInstanceOf;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.Mockito.lenient;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.when;
import java.time.Instant;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.stream.Stream;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.Arguments;
import org.junit.jupiter.params.provider.MethodSource;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.test.util.ReflectionTestUtils;
import jakarta.servlet.http.HttpServletRequest;
import stirling.software.common.cluster.ClusterBackplane;
import stirling.software.common.cluster.JobStore;
import stirling.software.common.cluster.JobStoreEntry;
import stirling.software.common.cluster.StickyMissRecorder;
import stirling.software.common.model.job.JobResult;
import stirling.software.common.service.FileStorage;
import stirling.software.common.service.JobOwnershipService;
import stirling.software.common.service.JobQueue;
import stirling.software.common.service.TaskManager;
/**
* Sticky-session ownership behavior for {@link JobController}.
*
* <p>Result PDFs live on the local disk of whichever node ran the job. When the load balancer's
* cookie/IP affinity *misses* and routes a download to a non-owner node, the controller must return
* {@code 410 Gone} with a structured payload that tells the client to retry (the LB will usually
* route them to the owner on the second attempt).
*
* <p>Contract verified here:
*
* <ul>
* <li>Owner == this node → file is read from local disk (normal 200).
* <li>Owner == another node → {@code 410 Gone} with {@code ownedBy} + {@code currentNode} fields,
* and {@code FileStorage} is <b>never</b> touched.
* <li>JobStore has no entry → behave as single-instance (no 410).
* <li>{@code owningNodeId} blank → behave as single-instance (no 410).
* <li>Single-instance install (no JobStore / no ClusterBackplane) → no 410, no NPE.
* </ul>
*
* <p>Manual mock construction (no {@code MockitoExtension}) so each test can wire its own
* controller with a different {@code ClusterBackplane} / {@code JobStore} combo without setUp stubs
* leaking across cases.
*/
class JobControllerOwnershipTest {
private TaskManager taskManager;
private FileStorage fileStorage;
private JobQueue jobQueue;
private HttpServletRequest request;
private JobOwnershipService jobOwnershipService;
private ClusterBackplane clusterBackplane;
private JobStore jobStore;
private StickyMissRecorder stickyMissRecorder;
private static final String JOB_ID = "job-42";
private static final String FILE_ID = "file-abc";
private static final String LOCAL_NODE = "node-self";
private static final String PEER_NODE = "node-peer";
@BeforeEach
void setUp() {
taskManager = mock(TaskManager.class);
fileStorage = mock(FileStorage.class);
jobQueue = mock(JobQueue.class);
request = mock(HttpServletRequest.class);
jobOwnershipService = mock(JobOwnershipService.class);
clusterBackplane = mock(ClusterBackplane.class);
jobStore = mock(JobStore.class);
stickyMissRecorder = mock(StickyMissRecorder.class);
}
private JobController makeController(ClusterBackplane backplane, JobStore store) {
JobController c =
new JobController(taskManager, fileStorage, jobQueue, request, backplane, store);
// jobOwnershipService is @Autowired(required=false) - field-injected. When non-null,
// validateJobAccess delegates to it. We leave it null by default so the security
// check is a no-op (backwards compat path) and the test focuses on sticky-410.
// stickyMissRecorder is also field-injected; wire by default so the metric assertions
// work without per-test setup.
ReflectionTestUtils.setField(c, "stickyMissRecorder", stickyMissRecorder);
return c;
}
private JobController makeController() {
return makeController(clusterBackplane, jobStore);
}
private JobStoreEntry entryOwnedBy(String ownerNodeId) {
return new JobStoreEntry(
JOB_ID,
JobStoreEntry.JobState.COMPLETE,
ownerNodeId,
Instant.now(),
Instant.now(),
null,
List.of(FILE_ID),
Map.of());
}
private JobResult completedJobWithFile() {
JobResult result = new JobResult();
result.setJobId(JOB_ID);
// completeWithSingleFile populates the resultFiles list, sets complete=true,
// and sets completedAt - all required for the getJobResult single-file branch.
result.completeWithSingleFile(FILE_ID, "out.pdf", "application/pdf", 7L);
return result;
}
/**
* Full sticky-410 contract for {@code downloadFile} when the requested job is owned by a peer.
* Asserts everything in one place (status, Retry-After header, payload shape, no
* implementation-detail leak, metric incremented, storage never touched).
*
* <p>Other tests cover the edge cases independently (locally-owned, no JobStore entry, blank
* owner, etc.) so a single failure here points at exactly one missing or broken contract
* property.
*/
@Test
@DisplayName(
"downloadFile peer-owned → full sticky-410 contract"
+ " (status + Retry-After + payload + metric + storage untouched)")
void downloadFile_peerOwned_fullStickyContract() throws Exception {
when(clusterBackplane.localNodeId()).thenReturn(LOCAL_NODE);
when(taskManager.findJobKeyByFileId(FILE_ID)).thenReturn(JOB_ID);
when(jobStore.get(JOB_ID)).thenReturn(Optional.of(entryOwnedBy(PEER_NODE)));
ResponseEntity<?> response = makeController().downloadFile(FILE_ID);
// 1. Status + Retry-After header (immediate-retry hint).
assertEquals(HttpStatus.GONE, response.getStatusCode());
assertEquals("0", response.getHeaders().getFirst("Retry-After"));
// 2. Payload shape: exactly { message, ownedBy, currentNode }, with no leaked secrets.
assertInstanceOf(Map.class, response.getBody());
Map<?, ?> body = (Map<?, ?>) response.getBody();
assertEquals(3, body.size(), "exactly: message, ownedBy, currentNode");
assertEquals(PEER_NODE, body.get("ownedBy"));
assertEquals(LOCAL_NODE, body.get("currentNode"));
assertNotNull(body.get("message"));
assertTrue(((String) body.get("message")).toLowerCase().contains("retry"));
assertNull(body.get("internalSecret"));
assertNull(body.get("filePath"));
// 3. Operator-alert metric incremented exactly once.
verify(stickyMissRecorder).recordStickyMiss();
// 4. Storage layer NEVER touched - bytes don't live here, so reading would be wrong.
verify(fileStorage, never()).retrieveBytes(FILE_ID);
}
// --------------------------------------------------------------------------------------
// Happy-path ownership matrix: any non-peer signal (local owner, no entry, blank owner)
// must produce a 200 from FileStorage with NO sticky-miss metric increment.
// --------------------------------------------------------------------------------------
private static Stream<Arguments> downloadHappyPathScenarios() {
return Stream.of(
Arguments.of("locallyOwned", LOCAL_NODE, true),
Arguments.of("noJobStoreEntry", null, false),
Arguments.of("blankOwningNodeId", "", true));
}
@ParameterizedTest(name = "downloadFile {0} -> 200, no sticky-miss")
@MethodSource("downloadHappyPathScenarios")
void downloadFile_happyPath_returnsOkAndNoMetric(
String scenario, String ownerNodeId, boolean entryPresent) throws Exception {
when(clusterBackplane.localNodeId()).thenReturn(LOCAL_NODE);
when(taskManager.findJobKeyByFileId(FILE_ID)).thenReturn(JOB_ID);
when(jobStore.get(JOB_ID))
.thenReturn(
entryPresent ? Optional.of(entryOwnedBy(ownerNodeId)) : Optional.empty());
when(fileStorage.retrieveBytes(FILE_ID)).thenReturn("payload".getBytes());
ResponseEntity<?> response = makeController().downloadFile(FILE_ID);
assertEquals(HttpStatus.OK, response.getStatusCode(), scenario);
verify(fileStorage).retrieveBytes(FILE_ID);
verify(stickyMissRecorder, never()).recordStickyMiss();
}
@Test
@DisplayName("getJobResult: locally-owned single-file result → reads from FileStorage, 200 OK")
void getJobResult_singleFile_locallyOwned_readsFromStorage() throws Exception {
when(clusterBackplane.localNodeId()).thenReturn(LOCAL_NODE);
when(taskManager.getJobResult(JOB_ID)).thenReturn(completedJobWithFile());
when(jobStore.get(JOB_ID)).thenReturn(Optional.of(entryOwnedBy(LOCAL_NODE)));
when(fileStorage.retrieveBytes(FILE_ID)).thenReturn("payload".getBytes());
ResponseEntity<?> response = makeController().getJobResult(JOB_ID);
assertEquals(HttpStatus.OK, response.getStatusCode());
}
// --------------------------------------------------------------------------------------
// Peer-owned 410 matrix across endpoints. Cross-cutting contract:
// - status = 410, Retry-After = 0
// - body.ownedBy = peer node, body.currentNode = local node
// - sticky-miss metric incremented exactly once per request
// The contract test above covers ALL of these properties for downloadFile in detail; this
// parameterized matrix asserts the same status + ownedBy + metric signals on every endpoint.
// --------------------------------------------------------------------------------------
private enum Endpoint {
DOWNLOAD_FILE,
GET_JOB_RESULT,
GET_JOB_STATUS,
CANCEL_JOB
}
private static Stream<Arguments> peerOwned410Scenarios() {
return Stream.of(
Arguments.of(Endpoint.DOWNLOAD_FILE),
Arguments.of(Endpoint.GET_JOB_RESULT),
Arguments.of(Endpoint.GET_JOB_STATUS),
Arguments.of(Endpoint.CANCEL_JOB));
}
@ParameterizedTest(name = "{0} peer-owned -> 410, ownedBy=peer, metric++")
@MethodSource("peerOwned410Scenarios")
void endpoint_peerOwned_returns410(Endpoint endpoint) throws Exception {
when(clusterBackplane.localNodeId()).thenReturn(LOCAL_NODE);
when(jobStore.get(JOB_ID)).thenReturn(Optional.of(entryOwnedBy(PEER_NODE)));
// Endpoint-specific wiring: the per-endpoint code path needs different mock setup
// before it reaches the sticky-410 guard.
switch (endpoint) {
case DOWNLOAD_FILE -> when(taskManager.findJobKeyByFileId(FILE_ID)).thenReturn(JOB_ID);
case GET_JOB_RESULT ->
when(taskManager.getJobResult(JOB_ID)).thenReturn(completedJobWithFile());
case GET_JOB_STATUS -> when(taskManager.getJobResult(JOB_ID)).thenReturn(null);
case CANCEL_JOB -> {
when(jobQueue.isJobQueued(JOB_ID)).thenReturn(false);
when(taskManager.getJobResult(JOB_ID)).thenReturn(null);
}
}
ResponseEntity<?> response =
switch (endpoint) {
case DOWNLOAD_FILE -> makeController().downloadFile(FILE_ID);
case GET_JOB_RESULT -> makeController().getJobResult(JOB_ID);
case GET_JOB_STATUS -> makeController().getJobStatus(JOB_ID);
case CANCEL_JOB -> makeController().cancelJob(JOB_ID);
};
assertEquals(HttpStatus.GONE, response.getStatusCode());
Map<?, ?> body = (Map<?, ?>) response.getBody();
assertEquals(PEER_NODE, body.get("ownedBy"));
assertEquals(LOCAL_NODE, body.get("currentNode"));
verify(stickyMissRecorder).recordStickyMiss();
// Storage / mutation must never be touched by a peer-routed request.
verify(fileStorage, never()).retrieveBytes(FILE_ID);
if (endpoint == Endpoint.CANCEL_JOB) {
verify(taskManager, never()).setError(JOB_ID, "Job was cancelled by user");
}
}
// --------------------------------------------------------------------------------------
// Unknown-job 404 matrix: when neither TaskManager nor JobStore knows the jobId, the
// controller must return 404 (not 410) and must NOT count it as a sticky miss.
// --------------------------------------------------------------------------------------
private static Stream<Arguments> unknownJob404Scenarios() {
return Stream.of(Arguments.of(Endpoint.GET_JOB_STATUS), Arguments.of(Endpoint.CANCEL_JOB));
}
@ParameterizedTest(name = "{0} unknown jobId -> 404 (not 410), no metric")
@MethodSource("unknownJob404Scenarios")
void endpoint_unknownJob_returns404(Endpoint endpoint) {
when(taskManager.getJobResult(JOB_ID)).thenReturn(null);
when(jobStore.get(JOB_ID)).thenReturn(Optional.empty());
if (endpoint == Endpoint.CANCEL_JOB) {
when(jobQueue.isJobQueued(JOB_ID)).thenReturn(false);
}
ResponseEntity<?> response =
switch (endpoint) {
case GET_JOB_STATUS -> makeController().getJobStatus(JOB_ID);
case CANCEL_JOB -> makeController().cancelJob(JOB_ID);
default -> throw new IllegalArgumentException(endpoint.name());
};
assertEquals(HttpStatus.NOT_FOUND, response.getStatusCode());
verify(stickyMissRecorder, never()).recordStickyMiss();
}
// --------------------------------------------------------------------------------------
// Single-instance / null-bean wiring: no NPE, no 410, no metric. These test SPECIFIC
// wiring permutations and so stay as discrete tests rather than rows.
// --------------------------------------------------------------------------------------
@Test
@DisplayName("Single-instance install (no ClusterBackplane bean): no 410, no NPE")
void singleInstance_noClusterBackplane_noGoneResponse() throws Exception {
when(taskManager.findJobKeyByFileId(FILE_ID)).thenReturn(JOB_ID);
when(fileStorage.retrieveBytes(FILE_ID)).thenReturn("payload".getBytes());
ResponseEntity<?> response = makeController(null, jobStore).downloadFile(FILE_ID);
assertEquals(HttpStatus.OK, response.getStatusCode());
verify(fileStorage).retrieveBytes(FILE_ID);
}
@Test
@DisplayName("Single-instance install (no JobStore bean): no 410, no NPE")
void singleInstance_noJobStore_noGoneResponse() throws Exception {
when(taskManager.findJobKeyByFileId(FILE_ID)).thenReturn(JOB_ID);
when(fileStorage.retrieveBytes(FILE_ID)).thenReturn("payload".getBytes());
ResponseEntity<?> response = makeController(clusterBackplane, null).downloadFile(FILE_ID);
assertEquals(HttpStatus.OK, response.getStatusCode());
verify(fileStorage).retrieveBytes(FILE_ID);
}
@Test
@DisplayName("Single-instance (no StickyMissRecorder bean) → no NPE, still 200 OK")
void noStickyMissRecorder_works() throws Exception {
when(clusterBackplane.localNodeId()).thenReturn(LOCAL_NODE);
when(taskManager.findJobKeyByFileId(FILE_ID)).thenReturn(JOB_ID);
when(jobStore.get(JOB_ID)).thenReturn(Optional.of(entryOwnedBy(LOCAL_NODE)));
when(fileStorage.retrieveBytes(FILE_ID)).thenReturn("payload".getBytes());
JobController c = makeController();
ReflectionTestUtils.setField(c, "stickyMissRecorder", null);
ResponseEntity<?> response = c.downloadFile(FILE_ID);
assertEquals(HttpStatus.OK, response.getStatusCode());
}
@Test
@DisplayName(
"cluster-mode but localNodeId is null → no NPE; 410 because owner is set and"
+ " differs from blank")
void clusterBackplanePresent_butLocalNodeIdNull_falsBackGracefully() throws Exception {
when(clusterBackplane.localNodeId()).thenReturn(null);
when(taskManager.findJobKeyByFileId(FILE_ID)).thenReturn(JOB_ID);
when(jobStore.get(JOB_ID)).thenReturn(Optional.of(entryOwnedBy(PEER_NODE)));
// We still 410: owner is "node-peer", local is null → they don't match. Rather than
// silently 200-from-wrong-disk (which would serve garbage), we surface the mismatch.
ResponseEntity<?> response = makeController().downloadFile(FILE_ID);
assertEquals(HttpStatus.GONE, response.getStatusCode());
Map<?, ?> body = (Map<?, ?>) response.getBody();
assertEquals("", body.get("currentNode"), "blank when localNodeId is null");
assertEquals(PEER_NODE, body.get("ownedBy"));
}
// --------------------------------------------------------------------------------------
// Ownership-service interaction: sticky-410 must take precedence over per-user auth so
// we never 403 on a peer-owned resource (which would leak existence + defeat the redirect).
// --------------------------------------------------------------------------------------
@Test
@DisplayName("Owner returns 410 even when JobOwnershipService allows access (orthogonal)")
void ownershipService_passes_butStickyStillReturns410() throws Exception {
when(clusterBackplane.localNodeId()).thenReturn(LOCAL_NODE);
when(taskManager.findJobKeyByFileId(FILE_ID)).thenReturn(JOB_ID);
when(jobStore.get(JOB_ID)).thenReturn(Optional.of(entryOwnedBy(PEER_NODE)));
lenient().when(jobOwnershipService.validateJobAccess(JOB_ID)).thenReturn(true);
JobController c = makeController();
ReflectionTestUtils.setField(c, "jobOwnershipService", jobOwnershipService);
ResponseEntity<?> response = c.downloadFile(FILE_ID);
// OwnershipService is about *user* auth; sticky-410 is about *node* topology.
// Both must pass for a 200, and node-ownership is checked first so a non-owner
// never even runs the user-auth check.
assertEquals(HttpStatus.GONE, response.getStatusCode());
}
@Test
@DisplayName(
"downloadFile: peer-owned + ownership-denied → 410 (NOT 403) so we don't leak"
+ " file existence")
void downloadFile_peerOwned_ownershipDenied_returns410NotForbidden() throws Exception {
// Guard ordering: sticky-410 must run before user-auth. If user-auth ran first, a
// peer-owned-file request on the wrong node would fail user-auth (this node cannot
// verify access to a job it does not own) and return 403, which leaks file existence
// AND defeats the sticky-410 design (the frontend can't retry-with-affinity off a 403).
// Guard first so the user is redirected to the owner where the real auth check happens.
when(clusterBackplane.localNodeId()).thenReturn(LOCAL_NODE);
when(taskManager.findJobKeyByFileId(FILE_ID)).thenReturn(JOB_ID);
when(jobStore.get(JOB_ID)).thenReturn(Optional.of(entryOwnedBy(PEER_NODE)));
lenient().when(jobOwnershipService.validateJobAccess(JOB_ID)).thenReturn(false);
JobController c = makeController();
ReflectionTestUtils.setField(c, "jobOwnershipService", jobOwnershipService);
ResponseEntity<?> response = c.downloadFile(FILE_ID);
assertEquals(HttpStatus.GONE, response.getStatusCode());
Map<?, ?> body = (Map<?, ?>) response.getBody();
assertEquals(PEER_NODE, body.get("ownedBy"));
// Crucially: never reached fileStorage, never returned 403.
verify(fileStorage, never()).retrieveBytes(FILE_ID);
}
@Test
@DisplayName(
"getJobStatus: peer-owned + ownership-denied → 410 (NOT 403) so we don't leak"
+ " job existence")
void getJobStatus_peerOwned_ownershipDenied_returns410NotForbidden() {
when(clusterBackplane.localNodeId()).thenReturn(LOCAL_NODE);
when(jobStore.get(JOB_ID)).thenReturn(Optional.of(entryOwnedBy(PEER_NODE)));
lenient().when(jobOwnershipService.validateJobAccess(JOB_ID)).thenReturn(false);
JobController c = makeController();
ReflectionTestUtils.setField(c, "jobOwnershipService", jobOwnershipService);
ResponseEntity<?> response = c.getJobStatus(JOB_ID);
assertEquals(HttpStatus.GONE, response.getStatusCode());
Map<?, ?> body = (Map<?, ?>) response.getBody();
assertEquals(PEER_NODE, body.get("ownedBy"));
}
@Test
@DisplayName(
"cancelJob: peer-owned + ownership-denied → 410 (NOT 403) so we don't leak job"
+ " existence")
void cancelJob_peerOwned_ownershipDenied_returns410NotForbidden() {
when(clusterBackplane.localNodeId()).thenReturn(LOCAL_NODE);
when(jobQueue.isJobQueued(JOB_ID)).thenReturn(false);
when(jobStore.get(JOB_ID)).thenReturn(Optional.of(entryOwnedBy(PEER_NODE)));
lenient().when(jobOwnershipService.validateJobAccess(JOB_ID)).thenReturn(false);
JobController c = makeController();
ReflectionTestUtils.setField(c, "jobOwnershipService", jobOwnershipService);
ResponseEntity<?> response = c.cancelJob(JOB_ID);
assertEquals(HttpStatus.GONE, response.getStatusCode());
Map<?, ?> body = (Map<?, ?>) response.getBody();
assertEquals(PEER_NODE, body.get("ownedBy"));
verify(taskManager, never()).setError(JOB_ID, "Job was cancelled by user");
}
// --------------------------------------------------------------------------------------
// S2: JobStore lookup hardening (local cache + Valkey-fault graceful-degrade)
// --------------------------------------------------------------------------------------
@Test
@DisplayName(
"guardNonOwner caches JobStore.get within TTL window: second call same jobId hits"
+ " cache, not Valkey")
void guardNonOwner_cachesJobStoreLookupWithinTtl() throws Exception {
when(clusterBackplane.localNodeId()).thenReturn(LOCAL_NODE);
when(taskManager.findJobKeyByFileId(FILE_ID)).thenReturn(JOB_ID);
when(jobStore.get(JOB_ID)).thenReturn(Optional.of(entryOwnedBy(LOCAL_NODE)));
when(fileStorage.retrieveBytes(FILE_ID)).thenReturn("payload".getBytes());
JobController c = makeController();
c.downloadFile(FILE_ID);
c.downloadFile(FILE_ID);
c.downloadFile(FILE_ID);
// Three downloads of the same fileId == one HGETALL. Without the cache this would have
// been three Valkey round-trips on the hot download path.
verify(jobStore, times(1)).get(JOB_ID);
}
@Test
@DisplayName(
"guardNonOwner: JobStore.get throws (Valkey timeout) → falls through to local-disk"
+ " path, no 500 leaks to caller")
void guardNonOwner_jobStoreException_fallsThroughToLocalPath() throws Exception {
when(taskManager.findJobKeyByFileId(FILE_ID)).thenReturn(JOB_ID);
// Simulate a Valkey timeout. spring-data-redis surfaces these as runtime exceptions
// wrapping Lettuce errors; the controller must not let any RuntimeException leak out.
when(jobStore.get(JOB_ID)).thenThrow(new RuntimeException("Valkey command timeout"));
when(fileStorage.retrieveBytes(FILE_ID)).thenReturn("payload".getBytes());
ResponseEntity<?> response = makeController().downloadFile(FILE_ID);
// The download must succeed via the local-disk path. A brief Valkey blip cannot break
// every download attempt with 500 - we cleanly degrade to single-node behavior.
assertEquals(HttpStatus.OK, response.getStatusCode());
verify(fileStorage).retrieveBytes(FILE_ID);
// The exception did NOT count as a sticky-miss (it wasn't one; we just couldn't see).
verify(stickyMissRecorder, never()).recordStickyMiss();
}
}
@@ -19,8 +19,6 @@ import org.springframework.test.util.ReflectionTestUtils;
import jakarta.servlet.http.HttpServletRequest;
import stirling.software.common.cluster.ClusterBackplane;
import stirling.software.common.cluster.JobStore;
import stirling.software.common.model.job.JobResult;
import stirling.software.common.service.FileStorage;
import stirling.software.common.service.JobOwnershipService;
@@ -39,10 +37,6 @@ class JobControllerTest {
@Mock private JobOwnershipService jobOwnershipService;
@Mock private ClusterBackplane clusterBackplane;
@Mock private JobStore jobStore;
private MockHttpSession session;
@InjectMocks private JobController controller;
+2
View File
@@ -123,6 +123,8 @@ SwaggerDoc.json
*.tar.gz
*.rar
*.db
# Whitelist the H2 fixtures that feed the version-migration CI smoke test.
!src/test/resources/db-migration-fixtures/*.mv.db
/build
/app/proprietary/build/
+9 -11
View File
@@ -5,6 +5,8 @@ repositories {
ext {
jwtVersion = '0.13.0'
awsSdkVersion = '2.44.12'
testcontainersMinioVersion = '1.21.4'
}
bootRun {
@@ -44,7 +46,6 @@ dependencies {
api 'org.springframework:spring-jdbc'
api 'org.springframework:spring-webmvc'
api 'org.springframework.session:spring-session-core'
implementation 'org.springframework.session:spring-session-data-redis'
api "org.springframework.security:spring-security-core:$springSecuritySamlVersion"
api "org.springframework.security:spring-security-saml2-service-provider:$springSecuritySamlVersion"
api 'org.springframework.boot:spring-boot-starter-jetty'
@@ -54,13 +55,8 @@ dependencies {
api 'org.springframework.boot:spring-boot-starter-mail'
api 'org.springframework.boot:spring-boot-starter-cache'
api 'com.github.ben-manes.caffeine:caffeine'
implementation 'org.springframework.boot:spring-boot-starter-data-redis'
api 'io.swagger.core.v3:swagger-core-jakarta:2.2.46'
implementation 'com.bucket4j:bucket4j_jdk17-core:8.19.0'
// Lettuce-backed Bucket4j ProxyManager used by ValkeyRateLimitStore for cluster-wide
// token-bucket rate limiting (parity with in-process Bucket4j semantics; no fixed-window
// boundary doubling).
implementation 'com.bucket4j:bucket4j_jdk17-lettuce:8.19.0'
implementation 'com.bucket4j:bucket4j_jdk17-core:8.18.0'
// https://mvnrepository.com/artifact/com.bucket4j/bucket4j_jdk17
implementation "org.bouncycastle:bcprov-jdk18on:$bouncycastleVersion"
@@ -78,10 +74,12 @@ dependencies {
exclude group: 'org.opensaml', module: 'opensaml-core'
}
// Testcontainers: spins up a real Valkey for LiveValkeyIntegrationTest in CI without
// needing a manually-started instance. Tests skip cleanly when Docker is unavailable.
testImplementation 'org.testcontainers:testcontainers:1.21.4'
testImplementation 'org.testcontainers:junit-jupiter:1.21.4'
implementation "software.amazon.awssdk:s3:$awsSdkVersion"
implementation "software.amazon.awssdk:url-connection-client:$awsSdkVersion"
testImplementation "org.testcontainers:minio:$testcontainersMinioVersion"
testImplementation "org.testcontainers:junit-jupiter:$testcontainersMinioVersion"
testImplementation "org.testcontainers:localstack:$testcontainersMinioVersion"
}
tasks.register('prepareKotlinBuildScriptModel') {}
@@ -1,47 +0,0 @@
package stirling.software.proprietary.cluster;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.Ordered;
import org.springframework.core.annotation.Order;
import jakarta.annotation.PostConstruct;
import lombok.extern.slf4j.Slf4j;
/**
* Runtime license gate for cluster mode. Cluster mode requires a SERVER or ENTERPRISE license; the
* SaaS flavor bypasses (no {@code runningProOrHigher} bean is published). Fires before any Valkey
* bean construction via {@link Ordered#HIGHEST_PRECEDENCE}.
*
* <p>There is no testing/development bypass. Live e2e tests that need cluster mode must inject a
* valid {@code stirling.premium.key} for a test-tier SERVER/ENTERPRISE license. Unit tests stub the
* {@code runningProOrHigher} bean directly.
*/
@Configuration
@ConditionalOnProperty(name = "cluster.enabled", havingValue = "true")
@Order(Ordered.HIGHEST_PRECEDENCE)
@Slf4j
public class ClusterLicenseGate {
@Autowired(required = false)
@Qualifier("runningProOrHigher")
private Boolean runningProOrHigher;
@PostConstruct
void verifyLicense() {
if (runningProOrHigher == null) {
return; // saas flavor - licensed via Stripe elsewhere
}
if (!runningProOrHigher) {
throw new IllegalStateException(
"Cluster mode (cluster.enabled=true) requires a SERVER or"
+ " ENTERPRISE license. Configure stirling.premium.key with a valid"
+ " license key (contact sales@stirlingpdf.com to obtain one), or set"
+ " cluster.enabled=false.");
}
log.info("Cluster license gate: SERVER/ENTERPRISE license verified, cluster mode allowed.");
}
}
@@ -1,123 +0,0 @@
package stirling.software.proprietary.cluster;
import java.util.List;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.AtomicLong;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.stereotype.Component;
import io.micrometer.core.instrument.Counter;
import io.micrometer.core.instrument.Gauge;
import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Timer;
import stirling.software.common.cluster.StickyMissRecorder;
import stirling.software.common.model.ApplicationProperties;
/**
* Cluster operation metrics exposed via {@code /actuator/prometheus}. Registered only when cluster
* mode is on.
*/
@Component
@ConditionalOnProperty(name = "cluster.enabled", havingValue = "true")
public class ClusterMetrics implements StickyMissRecorder {
private final MeterRegistry registry;
private final ApplicationProperties applicationProperties;
private final Counter stickyMissTotal;
private final Counter rateLimitRejected;
private final Timer backplaneLatency;
private final Timer jobWaitSeconds;
// Per-lane queue depth gauges. Lanes are a fixed enum (FAST, SLOW, AI), so we register all
// three eagerly so dashboards never have a missing series.
private static final List<String> KNOWN_LANES = List.of("FAST", "SLOW", "AI");
private final ConcurrentHashMap<String, AtomicLong> queueDepth = new ConcurrentHashMap<>();
// In-flight job count for THIS node.
private final AtomicLong jobsInflight = new AtomicLong();
public ClusterMetrics(MeterRegistry registry, ApplicationProperties applicationProperties) {
this.registry = registry;
this.applicationProperties = applicationProperties;
this.stickyMissTotal =
Counter.builder("stirling_cluster_sticky_miss_total")
.description(
"Sticky-session misses: a download for a job whose result lives on"
+ " a peer node landed on this node. High sustained value means"
+ " LB affinity is broken.")
.register(registry);
this.rateLimitRejected =
Counter.builder("stirling_cluster_ratelimit_rejected_total")
.description("Cluster-wide rate limit rejections")
.register(registry);
this.backplaneLatency =
Timer.builder("stirling_cluster_backplane_latency_seconds")
.description("Backplane round-trip latency")
.register(registry);
this.jobWaitSeconds =
Timer.builder("stirling_cluster_job_wait_seconds")
.description("Time jobs spend queued before execution")
.register(registry);
Gauge.builder("stirling_cluster_jobs_inflight", jobsInflight, AtomicLong::doubleValue)
.description("Jobs currently in flight on this node")
.tag("node", applicationProperties.getCluster().resolvedNodeId())
.register(registry);
for (String lane : KNOWN_LANES) {
ensureLaneGauge(lane);
}
}
/**
* Increment when {@code JobController} returns 410 Gone because the requested job's owner is a
* peer node. Surfaces to {@code stirling_cluster_sticky_miss_total}.
*/
@Override
public void recordStickyMiss() {
stickyMissTotal.increment();
}
public void recordRateLimitReject() {
rateLimitRejected.increment();
}
public Timer backplaneLatency() {
return backplaneLatency;
}
public Timer jobWaitSeconds() {
return jobWaitSeconds;
}
public void incrementInflight() {
jobsInflight.incrementAndGet();
}
public void decrementInflight() {
jobsInflight.decrementAndGet();
}
/**
* Publish (or update) the queue depth gauge for {@code lane}. Idempotent - safe to call hot.
* Known lanes (FAST, SLOW, AI) are pre-registered at construction; unknown lanes register on
* first call.
*/
public void setQueueDepth(String lane, long depth) {
ensureLaneGauge(lane).set(depth);
}
private AtomicLong ensureLaneGauge(String lane) {
return queueDepth.computeIfAbsent(
lane,
l -> {
AtomicLong holder = new AtomicLong();
Gauge.builder("stirling_cluster_queue_depth", holder, AtomicLong::doubleValue)
.description("Pending items in a job queue lane")
.tag("lane", l)
.register(registry);
return holder;
});
}
}
@@ -1,201 +0,0 @@
package stirling.software.proprietary.cluster;
import java.net.InetAddress;
import java.net.UnknownHostException;
import java.time.Duration;
import java.time.Instant;
import java.util.Locale;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.SmartLifecycle;
import org.springframework.context.event.EventListener;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.cluster.ClusterNode;
import stirling.software.common.cluster.InstanceRegistry;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.common.model.ApplicationProperties.Cluster;
/**
* Registers the local node with {@link InstanceRegistry} on startup, refreshes the entry at 1/3 of
* the TTL, and deregisters cleanly on shutdown.
*
* <p>Implements {@link SmartLifecycle} with {@code getPhase() == Integer.MAX_VALUE} so Spring tears
* this bean down before {@code LettuceConnectionFactory} - deregister therefore runs while the
* Valkey connection is still alive.
*/
@Component
@Slf4j
@ConditionalOnProperty(name = "cluster.enabled", havingValue = "true")
public class ClusterNodeBootstrap implements SmartLifecycle {
/** TTL of the node entry in the registry. Set to 3x the heartbeat interval. */
private final Duration heartbeatTtl;
private final ApplicationProperties applicationProperties;
private final InstanceRegistry instanceRegistry;
@Value("${server.port:8080}")
private int serverPort;
private volatile String nodeId;
private volatile String internalAddress;
private volatile boolean running = false;
public ClusterNodeBootstrap(
ApplicationProperties applicationProperties, InstanceRegistry instanceRegistry) {
this.applicationProperties = applicationProperties;
this.instanceRegistry = instanceRegistry;
Cluster cluster = applicationProperties.getCluster();
long heartbeatMs =
cluster.getNode() == null ? 10_000L : cluster.getNode().getHeartbeatIntervalMs();
// TTL = 3x heartbeat: tolerate one missed tick before the node drops out of the registry.
this.heartbeatTtl = Duration.ofMillis(heartbeatMs * 3);
}
@EventListener(ApplicationReadyEvent.class)
public void registerOnStartup() {
nodeId = applicationProperties.getCluster().resolvedNodeId();
internalAddress = resolveInternalAddress();
registerSelf("register");
}
@Scheduled(fixedDelayString = "${cluster.node.heartbeat-interval-ms:10000}")
public void heartbeat() {
// Heartbeat-after-stop race: SmartLifecycle.stop() deregisters, but the @Scheduled
// tick keeps firing during a slow drain. Without this guard, the next tick re-registers
// the dead node and the entry resurfaces in the registry until TTL expiry.
if (!running) {
return;
}
if (nodeId == null) {
return; // not yet registered (startup race)
}
// Self-healing: register() is idempotent and re-populates every field, so a wiped
// Valkey (FLUSHALL, hash eviction) recovers on the next tick without operator action.
registerSelf("heartbeat");
}
private void registerSelf(String reason) {
try {
instanceRegistry.register(
new ClusterNode(nodeId, internalAddress, Instant.now(), role()), heartbeatTtl);
if ("register".equals(reason)) {
log.info(
"Cluster node registered: nodeId={}, internalAddress={}, role={}, ttl={}s",
nodeId,
internalAddress,
role(),
heartbeatTtl.toSeconds());
}
} catch (RuntimeException e) {
log.debug("Cluster {} failed for {}", reason, nodeId, e);
}
}
// ---------- SmartLifecycle (see class javadoc for ordering rationale) ----------
@Override
public void start() {
running = true;
}
@Override
public void stop() {
running = false;
if (nodeId == null) {
return;
}
try {
instanceRegistry.deregister(nodeId);
log.info("Cluster node deregistered: {}", nodeId);
} catch (RuntimeException e) {
// Registry entry will TTL-expire within heartbeatTtl anyway.
log.warn(
"Cluster deregister failed for {} (will TTL-expire within {}s): {}",
nodeId,
heartbeatTtl.toSeconds(),
e.getMessage());
}
}
@Override
public boolean isRunning() {
return running;
}
@Override
public int getPhase() {
return Integer.MAX_VALUE; // stopped first; LettuceConnectionFactory's phase is 0
}
@Override
public boolean isAutoStartup() {
return true;
}
/**
* Resolve the address peers should hit. Order: explicit config -> {@code POD_IP} env (K8s
* downward API) -> JDK hostname -> fail loud (never silently fall back to a loopback).
*
* <p>Scheme is taken from {@code cluster.node.scheme} (default {@code http}). Set to {@code
* https} when nodes terminate TLS themselves; leave as {@code http} when an upstream LB
* terminates TLS and intra-cluster traffic is plain HTTP.
*/
private String resolveInternalAddress() {
Cluster cluster = applicationProperties.getCluster();
String configured =
cluster.getNode() == null ? null : cluster.getNode().getInternalAddress();
if (configured != null && !configured.isBlank()) {
return ensurePort(configured);
}
String podIp = System.getenv("POD_IP");
if (podIp != null && !podIp.isBlank()) {
return scheme() + "://" + podIp + ":" + serverPort;
}
try {
return scheme()
+ "://"
+ InetAddress.getLocalHost().getHostAddress()
+ ":"
+ serverPort;
} catch (UnknownHostException e) {
throw new IllegalStateException(
"Could not resolve this host's address for cluster registration; set"
+ " cluster.node.internal-address explicitly (or set POD_IP"
+ " in the Kubernetes downward API).",
e);
}
}
private String ensurePort(String addr) {
if (addr.startsWith("http://") || addr.startsWith("https://")) {
return addr;
}
if (addr.contains(":")) {
return scheme() + "://" + addr;
}
return scheme() + "://" + addr + ":" + serverPort;
}
private String scheme() {
Cluster cluster = applicationProperties.getCluster();
if (cluster.getNode() == null
|| cluster.getNode().getScheme() == null
|| cluster.getNode().getScheme().isBlank()) {
return "http";
}
String s = cluster.getNode().getScheme().trim().toLowerCase(Locale.ROOT);
return "https".equals(s) ? "https" : "http";
}
private String role() {
Cluster.NodeRole r = applicationProperties.getCluster().resolvedRole();
return r == null ? "BOTH" : r.name();
}
}
@@ -1,55 +0,0 @@
package stirling.software.proprietary.cluster;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.serializer.GenericJackson2JsonRedisSerializer;
import org.springframework.data.redis.serializer.RedisSerializer;
import org.springframework.session.data.redis.config.annotation.web.http.EnableRedisHttpSession;
import org.springframework.session.web.http.CookieSerializer;
import org.springframework.session.web.http.DefaultCookieSerializer;
/**
* Enables Spring Session backed by Valkey when cluster mode is on AND a Lettuce connection factory
* exists (backplane=valkey). {@link ConditionalOnBean} prevents {@code @EnableRedisHttpSession}
* from wiring its filter before the required connection factory is present; without it the bean
* graph fails with "No qualifying bean of type 'SessionRepository'".
*/
@Configuration
@ConditionalOnProperty(name = "cluster.enabled", havingValue = "true")
@ConditionalOnBean(LettuceConnectionFactory.class)
@EnableRedisHttpSession
public class ClusterSessionConfiguration {
/**
* The bean name {@code springSessionDefaultRedisSerializer} is the exact hook Spring Session
* uses to override JDK serialization. JDK serialization is a deserialization-gadget RCE surface
* for anyone with Valkey write access; Jackson JSON is not.
*
* <p>Migrate to {@code GenericJacksonJsonRedisSerializer} when the Spring Session reference doc
* does (https://docs.spring.io/spring-session/reference/spring-security.html#config-redis).
*/
@Bean
@SuppressWarnings(
"removal") // GenericJackson2JsonRedisSerializer is the documented recipe name; migrate
// when upstream does
public RedisSerializer<Object> springSessionDefaultRedisSerializer() {
return new GenericJackson2JsonRedisSerializer();
}
/**
* Harden the Spring Session cookie: Spring Session omits {@code Secure} and {@code SameSite} by
* default, leaving the session id susceptible to plaintext leakage and CSRF. {@code Lax} allows
* top-level navigations (login redirects) while blocking cross-site sub-resource requests.
* HttpOnly is on by default - asserted in the regression test.
*/
@Bean
public CookieSerializer cookieSerializer() {
DefaultCookieSerializer serializer = new DefaultCookieSerializer();
serializer.setUseSecureCookie(true);
serializer.setSameSite("Lax");
return serializer;
}
}
@@ -0,0 +1,200 @@
package stirling.software.proprietary.cluster.s3;
import java.net.InetAddress;
import java.net.URI;
import java.net.URISyntaxException;
import java.net.UnknownHostException;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.model.ApplicationProperties;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.checksums.RequestChecksumCalculation;
import software.amazon.awssdk.core.checksums.ResponseChecksumValidation;
import software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3ClientBuilder;
import software.amazon.awssdk.services.s3.S3Configuration;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
/**
* Shared factory for {@link S3Client} and {@link S3Presigner} instances used by both {@code
* S3StorageProvider} and {@code S3FileStore}, so endpoint/region/credentials wiring lives in
* exactly one place.
*/
@Slf4j
public final class S3Clients {
private S3Clients() {}
/** Paired client and presigner with coordinated lifecycle. */
public record Bundle(S3Client client, S3Presigner presigner) implements AutoCloseable {
@Override
public void close() {
try {
presigner.close();
} catch (Exception e) {
log.warn("Error closing S3 presigner", e);
}
try {
client.close();
} catch (Exception e) {
log.warn("Error closing S3 client", e);
}
}
}
/** Build a client+presigner pair from the shared S3 config block. */
public static Bundle build(ApplicationProperties.Storage.S3 cfg, String usage) {
if (cfg == null) {
throw new IllegalStateException(
usage + " requires storage.s3.* configuration to be set");
}
if (cfg.getBucket() == null || cfg.getBucket().isBlank()) {
throw new IllegalStateException(usage + " requires storage.s3.bucket to be set");
}
String region =
cfg.getRegion() == null || cfg.getRegion().isBlank()
? "us-east-1"
: cfg.getRegion();
S3Configuration s3Configuration =
S3Configuration.builder().pathStyleAccessEnabled(cfg.isPathStyleAccess()).build();
RequestChecksumCalculation requestChecksum =
parseRequestChecksum(cfg.getRequestChecksumCalculation());
ResponseChecksumValidation responseChecksum =
parseResponseChecksum(cfg.getResponseChecksumValidation());
S3ClientBuilder clientBuilder =
S3Client.builder()
.httpClient(UrlConnectionHttpClient.create())
.region(Region.of(region))
.serviceConfiguration(s3Configuration)
.requestChecksumCalculation(requestChecksum)
.responseChecksumValidation(responseChecksum);
S3Presigner.Builder presignerBuilder =
S3Presigner.builder()
.region(Region.of(region))
.serviceConfiguration(s3Configuration);
if (cfg.getEndpoint() != null && !cfg.getEndpoint().isBlank()) {
URI endpoint;
try {
endpoint = new URI(cfg.getEndpoint());
} catch (URISyntaxException e) {
throw new IllegalStateException(
"Invalid storage.s3.endpoint: " + cfg.getEndpoint(), e);
}
validateEndpointHost(endpoint, cfg.isAllowPrivateEndpoints());
clientBuilder.endpointOverride(endpoint);
presignerBuilder.endpointOverride(endpoint);
}
boolean hasStaticCreds =
cfg.getAccessKey() != null
&& !cfg.getAccessKey().isBlank()
&& cfg.getSecretKey() != null
&& !cfg.getSecretKey().isBlank();
if (hasStaticCreds) {
AwsBasicCredentials credentials =
AwsBasicCredentials.create(cfg.getAccessKey(), cfg.getSecretKey());
StaticCredentialsProvider provider = StaticCredentialsProvider.create(credentials);
clientBuilder.credentialsProvider(provider);
presignerBuilder.credentialsProvider(provider);
} else {
clientBuilder.credentialsProvider(DefaultCredentialsProvider.create());
presignerBuilder.credentialsProvider(DefaultCredentialsProvider.create());
}
log.debug(
"Configured S3 {}: bucket={}, region={}, endpoint={}, pathStyle={}",
usage,
cfg.getBucket(),
region,
cfg.getEndpoint() == null || cfg.getEndpoint().isBlank()
? "<aws-default>"
: cfg.getEndpoint(),
cfg.isPathStyleAccess());
return new Bundle(clientBuilder.build(), presignerBuilder.build());
}
/**
* Block SSRF via the S3 endpoint setting. An admin who can edit config could otherwise point
* the SDK at the cloud metadata service (e.g. {@code http://169.254.169.254/}) and exfiltrate
* instance-role credentials. Reject any endpoint whose host resolves to a loopback, link-local,
* or RFC1918 private address unless the operator has explicitly opted in via {@code
* storage.s3.allow-private-endpoints=true}.
*/
static void validateEndpointHost(URI endpoint, boolean allowPrivate) {
if (allowPrivate) {
return;
}
String host = endpoint.getHost();
if (host == null || host.isBlank()) {
throw new IllegalStateException("storage.s3.endpoint must include a host: " + endpoint);
}
InetAddress[] addresses;
try {
addresses = InetAddress.getAllByName(host);
} catch (UnknownHostException e) {
throw new IllegalStateException(
"Unable to resolve storage.s3.endpoint host '" + host + "'", e);
}
for (InetAddress address : addresses) {
if (isPrivateOrLocal(address)) {
throw new IllegalStateException(
"storage.s3.endpoint host '"
+ host
+ "' resolves to private/link-local address "
+ address.getHostAddress()
+ "; set storage.s3.allow-private-endpoints=true to opt in"
+ " (e.g. for MinIO or in-cluster S3).");
}
}
}
private static boolean isPrivateOrLocal(InetAddress address) {
return address.isLoopbackAddress()
|| address.isLinkLocalAddress()
|| address.isSiteLocalAddress()
|| address.isAnyLocalAddress()
|| address.isMulticastAddress();
}
static RequestChecksumCalculation parseRequestChecksum(String value) {
if (value == null || value.isBlank()) {
return RequestChecksumCalculation.WHEN_SUPPORTED;
}
try {
return RequestChecksumCalculation.valueOf(
value.trim().toUpperCase(java.util.Locale.ROOT));
} catch (IllegalArgumentException ex) {
log.warn(
"Unknown storage.s3.request-checksum-calculation value '{}', falling back to WHEN_SUPPORTED",
value);
return RequestChecksumCalculation.WHEN_SUPPORTED;
}
}
static ResponseChecksumValidation parseResponseChecksum(String value) {
if (value == null || value.isBlank()) {
return ResponseChecksumValidation.WHEN_SUPPORTED;
}
try {
return ResponseChecksumValidation.valueOf(
value.trim().toUpperCase(java.util.Locale.ROOT));
} catch (IllegalArgumentException ex) {
log.warn(
"Unknown storage.s3.response-checksum-validation value '{}', falling back to WHEN_SUPPORTED",
value);
return ResponseChecksumValidation.WHEN_SUPPORTED;
}
}
}
@@ -0,0 +1,226 @@
package stirling.software.proprietary.cluster.s3;
import java.io.BufferedInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.util.Optional;
import java.util.UUID;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.cluster.FileStore;
import software.amazon.awssdk.core.ResponseInputStream;
import software.amazon.awssdk.core.exception.SdkException;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.DeleteObjectRequest;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.GetObjectResponse;
import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
import software.amazon.awssdk.services.s3.model.HeadObjectResponse;
import software.amazon.awssdk.services.s3.model.NoSuchKeyException;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.S3Exception;
/**
* S3-backed {@link FileStore} for transient job-result files. Objects are namespaced under a
* configurable key prefix (default {@code transient/}) and can coexist in the same bucket as {@code
* S3StorageProvider}.
*/
@Slf4j
public class S3FileStore implements FileStore, AutoCloseable {
public static final String DEFAULT_KEY_PREFIX = "transient/";
private final S3Client s3Client;
private final String bucket;
private final String keyPrefix;
private final boolean ownsClient;
public S3FileStore(S3Client s3Client, String bucket) {
this(s3Client, bucket, DEFAULT_KEY_PREFIX, true);
}
public S3FileStore(S3Client s3Client, String bucket, String keyPrefix) {
this(s3Client, bucket, keyPrefix, true);
}
/**
* @param ownsClient when true, {@link #close()} will close the supplied client. Set to false in
* tests that share the client with another consumer.
*/
public S3FileStore(S3Client s3Client, String bucket, String keyPrefix, boolean ownsClient) {
if (bucket == null || bucket.isBlank()) {
throw new IllegalArgumentException("S3 bucket must be configured");
}
this.s3Client = s3Client;
this.bucket = bucket;
this.keyPrefix = normalizePrefix(keyPrefix);
this.ownsClient = ownsClient;
}
@Override
public Stored store(InputStream in, String originalName) throws IOException {
String fileId = UUID.randomUUID().toString();
// S3 PUT requires a known content-length; spool to a temp file first so memory stays
// bounded for large payloads, then stream the file to S3 via RequestBody.fromFile.
Path tempFile = Files.createTempFile("s3-upload-", ".bin");
long size;
try {
try (InputStream src = in) {
Files.copy(src, tempFile, StandardCopyOption.REPLACE_EXISTING);
}
size = Files.size(tempFile);
PutObjectRequest request =
PutObjectRequest.builder().bucket(bucket).key(resolveKey(fileId)).build();
try {
s3Client.putObject(request, RequestBody.fromFile(tempFile));
} catch (SdkException e) {
throw new IOException("Failed to upload object to S3", e);
}
} finally {
try {
Files.deleteIfExists(tempFile);
} catch (IOException cleanupError) {
log.warn("Failed to delete S3 upload temp file: {}", tempFile, cleanupError);
}
}
return new Stored(fileId, size);
}
@Override
public InputStream retrieve(String fileId) throws IOException {
validateFileId(fileId);
GetObjectRequest request =
GetObjectRequest.builder().bucket(bucket).key(resolveKey(fileId)).build();
try {
ResponseInputStream<GetObjectResponse> stream = s3Client.getObject(request);
return new BufferedInputStream(stream);
} catch (NoSuchKeyException e) {
throw new IOException("File not found with ID: " + fileId, e);
} catch (SdkException e) {
throw new IOException("Failed to load object from S3", e);
}
}
@Override
public byte[] retrieveBytes(String fileId) throws IOException {
validateFileId(fileId);
GetObjectRequest request =
GetObjectRequest.builder().bucket(bucket).key(resolveKey(fileId)).build();
try (ResponseInputStream<GetObjectResponse> stream = s3Client.getObject(request)) {
return stream.readAllBytes();
} catch (NoSuchKeyException e) {
throw new IOException("File not found with ID: " + fileId, e);
} catch (SdkException e) {
throw new IOException("Failed to load object from S3", e);
}
}
@Override
public long size(String fileId) throws IOException {
validateFileId(fileId);
HeadObjectRequest request =
HeadObjectRequest.builder().bucket(bucket).key(resolveKey(fileId)).build();
try {
HeadObjectResponse response = s3Client.headObject(request);
return Optional.ofNullable(response.contentLength()).orElse(0L);
} catch (NoSuchKeyException e) {
throw new IOException("File not found with ID: " + fileId, e);
} catch (S3Exception e) {
if (e.statusCode() == 404) {
throw new IOException("File not found with ID: " + fileId, e);
}
throw new IOException("Failed to head object in S3", e);
} catch (SdkException e) {
throw new IOException("Failed to head object in S3", e);
}
}
@Override
public boolean delete(String fileId) {
try {
validateFileId(fileId);
} catch (IllegalArgumentException e) {
log.warn("Refusing to delete invalid file id: {}", fileId);
return false;
}
try {
s3Client.deleteObject(
DeleteObjectRequest.builder().bucket(bucket).key(resolveKey(fileId)).build());
return true;
} catch (SdkException e) {
log.error("Error deleting file with ID: {}", fileId, e);
return false;
}
}
@Override
public boolean exists(String fileId) {
try {
validateFileId(fileId);
} catch (IllegalArgumentException e) {
return false;
}
HeadObjectRequest request =
HeadObjectRequest.builder().bucket(bucket).key(resolveKey(fileId)).build();
try {
s3Client.headObject(request);
return true;
} catch (NoSuchKeyException e) {
return false;
} catch (S3Exception e) {
if (e.statusCode() == 404) {
return false;
}
log.warn("Error checking existence for file ID: {}", fileId, e);
return false;
} catch (SdkException e) {
log.warn("Error checking existence for file ID: {}", fileId, e);
return false;
}
}
@Override
public void close() {
if (!ownsClient) {
return;
}
try {
s3Client.close();
} catch (Exception e) {
log.warn("Error closing S3 client", e);
}
}
String resolveKey(String fileId) {
return keyPrefix + fileId;
}
private static void validateFileId(String fileId) {
if (fileId == null || fileId.isBlank()) {
throw new IllegalArgumentException("File ID must not be blank");
}
if (fileId.contains("..") || fileId.contains("/") || fileId.contains("\\")) {
throw new IllegalArgumentException("Invalid file ID");
}
}
private static String normalizePrefix(String prefix) {
if (prefix == null || prefix.isBlank()) {
return "";
}
String trimmed = prefix.trim();
if (trimmed.startsWith("/")) {
trimmed = trimmed.substring(1);
}
if (!trimmed.isEmpty() && !trimmed.endsWith("/")) {
trimmed = trimmed + "/";
}
return trimmed;
}
}
@@ -0,0 +1,37 @@
package stirling.software.proprietary.cluster.s3;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.cluster.FileStore;
import stirling.software.common.model.ApplicationProperties;
/** Activates the S3-backed transient {@link FileStore} when {@code cluster.artifactStore=s3}. */
@Slf4j
@Configuration
@RequiredArgsConstructor
@ConditionalOnProperty(prefix = "cluster", name = "artifactStore", havingValue = "s3")
public class S3FileStoreConfiguration {
private final ApplicationProperties applicationProperties;
@Bean(destroyMethod = "close")
@ConditionalOnMissingBean
public FileStore fileStore(@Value("${cluster.s3.keyPrefix:transient/}") String keyPrefix) {
ApplicationProperties.Storage.S3 cfg = applicationProperties.getStorage().getS3();
S3Clients.Bundle bundle = S3Clients.build(cfg, "cluster file store");
// FileStore has no signed-URL contract; close the unused presigner immediately.
try {
bundle.presigner().close();
} catch (Exception ignored) {
}
log.info("Cluster FileStore: s3 (bucket={}, keyPrefix={})", cfg.getBucket(), keyPrefix);
return new S3FileStore(bundle.client(), cfg.getBucket(), keyPrefix, true);
}
}
@@ -1,25 +0,0 @@
package stirling.software.proprietary.cluster.valkey;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
import org.springframework.boot.autoconfigure.condition.ConditionalOnExpression;
/**
* Composite condition: cluster mode is on AND the configured backplane is Valkey.
*
* <p>Either condition alone is insufficient to load a Valkey bean. With {@code enabled=true} but
* {@code backplane=inprocess}, loading the Valkey beans would crash at boot because there's no
* {@code StringRedisTemplate}; with {@code enabled=false} the whole cluster mode is off. Combining
* the two stops both footguns.
*
* <p>Spring's {@code @ConditionalOnProperty} cannot be applied twice on the same class, so we use
* {@code @ConditionalOnExpression} via this meta-annotation.
*/
@Target({ElementType.TYPE, ElementType.METHOD})
@Retention(RetentionPolicy.RUNTIME)
@ConditionalOnExpression(
"${cluster.enabled:false} and '${cluster.backplane:inprocess}'.equals('valkey')")
public @interface ConditionalOnValkeyBackplane {}
@@ -1,62 +0,0 @@
package stirling.software.proprietary.cluster.valkey;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.data.redis.core.RedisCallback;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.cluster.ClusterBackplane;
import stirling.software.common.model.ApplicationProperties;
@Slf4j
@Component
@RequiredArgsConstructor
@ConditionalOnProperty(name = "cluster.enabled", havingValue = "true")
@org.springframework.boot.autoconfigure.condition.ConditionalOnProperty(
name = "cluster.backplane",
havingValue = "valkey")
public class ValkeyClusterBackplane implements ClusterBackplane {
private final ApplicationProperties applicationProperties;
private final StringRedisTemplate template;
@Override
public boolean isHealthy() {
try {
// template.execute() borrows from the pool and returns the connection in a finally
// block - critical because isHealthy() is hit on every k8s liveness/readiness probe
// tick. Calling getConnectionFactory().getConnection() directly leaks the connection
// and exhausts the pool under monitoring load.
String pong = template.execute((RedisCallback<String>) connection -> connection.ping());
return "PONG".equalsIgnoreCase(pong);
} catch (RuntimeException ex) {
log.warn("Valkey backplane health check failed: {}", ex.getMessage());
return false;
}
}
@Override
public String backplaneType() {
return "valkey";
}
@Override
public String localNodeId() {
return applicationProperties.getCluster().resolvedNodeId();
}
/**
* Disable the local {@code TaskManager#cleanupOldJobs} loop on Valkey-backed clusters: {@link
* ValkeyJobStore} stores every entry with a TTL pExpire and the reverse-index entries share
* that TTL, so Valkey itself evicts expired job state. Running the local cleanup loop on top of
* that would only delete per-node in-memory {@code TaskManager} caches that the cluster-visible
* {@code JobStore} has already authoritative state for.
*/
@Override
public boolean shouldRunLocalCleanup() {
return false;
}
}
@@ -1,222 +0,0 @@
package stirling.software.proprietary.cluster.valkey;
import java.net.URI;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.DependsOn;
import org.springframework.data.redis.connection.RedisPassword;
import org.springframework.data.redis.connection.RedisStandaloneConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceClientConfiguration;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.core.StringRedisTemplate;
import io.lettuce.core.RedisCommandExecutionException;
import io.lettuce.core.SslVerifyMode;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.common.model.ApplicationProperties.Cluster;
/** Wires the LettuceConnectionFactory and StringRedisTemplate for cluster mode. */
@Slf4j
@Configuration
@RequiredArgsConstructor
@ConditionalOnProperty(name = "cluster.enabled", havingValue = "true")
@DependsOn("clusterLicenseGate")
public class ValkeyConnectionConfiguration {
private final ApplicationProperties applicationProperties;
@Bean(destroyMethod = "destroy")
@ConditionalOnProperty(name = "cluster.backplane", havingValue = "valkey")
public LettuceConnectionFactory valkeyConnectionFactory() {
Cluster cluster = applicationProperties.getCluster();
String url = cluster.getValkey().getUrl();
if (url == null || url.isBlank()) {
throw new IllegalStateException("cluster.valkey.url must be set when backplane=valkey");
}
URI uri = URI.create(url);
boolean tls = "rediss".equalsIgnoreCase(uri.getScheme());
int port = uri.getPort() <= 0 ? 6379 : uri.getPort();
RedisStandaloneConfiguration cfg = new RedisStandaloneConfiguration(uri.getHost(), port);
if (uri.getUserInfo() != null) {
String[] parts = uri.getUserInfo().split(":", 2);
if (parts.length == 2) {
cfg.setUsername(parts[0]);
cfg.setPassword(RedisPassword.of(parts[1]));
} else if (parts.length == 1 && !parts[0].isBlank()) {
cfg.setPassword(RedisPassword.of(parts[0]));
}
}
boolean skipCertVerification =
cluster.getValkey().getTls() != null
&& cluster.getValkey().getTls().isSkipCertVerification();
LettuceClientConfiguration clientConfig =
buildClientConfiguration(tls, skipCertVerification);
LettuceConnectionFactory factory = new LettuceConnectionFactory(cfg, clientConfig);
factory.afterPropertiesSet();
// Eager handshake with retry tolerates docker-compose DNS races; fails boot loudly
// if Valkey is genuinely unreachable.
eagerHandshake(factory, uri.getHost(), port, tls);
log.info(
"Valkey connection configured: {}:{} tls={} verifyPeer={}",
uri.getHost(),
port,
tls,
tls ? clientConfig.getVerifyMode() : "n/a");
return factory;
}
/**
* Builds the Lettuce client configuration with TLS verification pinned. Package-private so unit
* tests can verify the {@code verifyPeer} mode without standing up a real Valkey.
*
* <p>{@code verifyPeer(FULL)} is pinned explicitly so a future Spring Data Redis default change
* cannot silently weaken our TLS handshake. {@code FULL} = X.509 chain + hostname check (per
* Lettuce's {@link SslVerifyMode}). The {@code skipCertVerification} opt-out is for local dev
* with self-signed certs only; production deployments MUST leave it false.
*/
static LettuceClientConfiguration buildClientConfiguration(
boolean tls, boolean skipCertVerification) {
LettuceClientConfiguration.LettuceClientConfigurationBuilder clientBuilder =
LettuceClientConfiguration.builder();
if (tls) {
clientBuilder
.useSsl()
.verifyPeer(skipCertVerification ? SslVerifyMode.NONE : SslVerifyMode.FULL);
if (skipCertVerification) {
log.warn(
"Valkey TLS hostname/chain verification DISABLED via"
+ " cluster.valkey.tls.skip-cert-verification=true"
+ " - insecure, dev-only");
}
}
return clientBuilder.build();
}
/**
* 10 x 3s = 30s of retry. Boot-time only.
*
* <p>Auth-class failures (WRONGPASS / NOAUTH / NOPERM) are unrecoverable and surfaced
* immediately on the first attempt; only transport-level errors (connection refused, timeout,
* host unreachable) get the retry loop.
*
* <p>Package-private so unit tests can drive it with a mocked connection factory.
*/
static void eagerHandshake(
LettuceConnectionFactory factory, String host, int port, boolean tls) {
RuntimeException last = null;
for (int attempt = 1; attempt <= 10; attempt++) {
try {
String pong = factory.getConnection().ping();
if (!"PONG".equalsIgnoreCase(pong)) {
throw new IllegalStateException(
"Valkey PING returned '" + pong + "' (expected PONG)");
}
if (attempt > 1) {
log.info("Valkey reachable after {} attempts", attempt);
}
return;
} catch (RuntimeException ex) {
if (isAuthFailure(ex)) {
factory.destroy();
throw new IllegalStateException(
"Valkey authentication failed for "
+ host
+ ":"
+ port
+ " (tls="
+ tls
+ "): "
+ rootAuthMessage(ex)
+ ". Check cluster.valkey.url credentials"
+ " (user/password and ACL permissions).",
ex);
}
last = ex;
log.warn(
"Valkey PING attempt {}/10 failed ({}:{}, tls={}): {}",
attempt,
host,
port,
tls,
ex.getMessage());
try {
Thread.sleep(3000);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
break;
}
}
}
factory.destroy();
throw new IllegalStateException(
"Valkey unreachable at boot after 10 attempts ("
+ host
+ ":"
+ port
+ ", tls="
+ tls
+ "): "
+ (last == null ? "no detail" : last.getMessage()),
last);
}
/**
* Walks the cause chain for a Lettuce {@link RedisCommandExecutionException} whose message
* starts with an auth-class server reply (WRONGPASS, NOAUTH, NOPERM). Spring Data Redis wraps
* Lettuce errors in a {@code RedisSystemException}, so the auth signal usually lives one level
* down from the thrown exception.
*
* <p>Checked for a typed alternative: neither Spring Data Redis 4.0.5 nor Lettuce 6.8.2 ships a
* {@code RedisAuthenticationException} on the classpath, so we keep the message-prefix match.
* Revisit when upgrading Spring Data Redis if a typed exception lands upstream.
*/
static boolean isAuthFailure(Throwable t) {
for (Throwable cur = t; cur != null; cur = cur.getCause()) {
if (cur instanceof RedisCommandExecutionException && hasAuthPrefix(cur.getMessage())) {
return true;
}
// Defensive: some translations preserve the original message on the wrapper itself.
if (hasAuthPrefix(cur.getMessage())) {
return true;
}
if (cur.getCause() == cur) {
break;
}
}
return false;
}
private static boolean hasAuthPrefix(String message) {
if (message == null) {
return false;
}
String upper = message.toUpperCase(java.util.Locale.ROOT).stripLeading();
return upper.startsWith("WRONGPASS")
|| upper.startsWith("NOAUTH")
|| upper.startsWith("NOPERM");
}
private static String rootAuthMessage(Throwable t) {
for (Throwable cur = t; cur != null; cur = cur.getCause()) {
if (cur instanceof RedisCommandExecutionException && cur.getMessage() != null) {
return cur.getMessage();
}
if (cur.getCause() == cur) {
break;
}
}
return t.getMessage();
}
@Bean
@ConditionalOnProperty(name = "cluster.backplane", havingValue = "valkey")
public StringRedisTemplate valkeyTemplate(LettuceConnectionFactory factory) {
return new StringRedisTemplate(factory);
}
}
@@ -1,92 +0,0 @@
package stirling.software.proprietary.cluster.valkey;
import java.time.Duration;
import java.util.Collections;
import java.util.Optional;
import java.util.UUID;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.data.redis.core.script.DefaultRedisScript;
import org.springframework.data.redis.core.script.RedisScript;
import org.springframework.stereotype.Component;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.cluster.DistributedLock;
@Component
@RequiredArgsConstructor
@ConditionalOnValkeyBackplane
@Slf4j
public class ValkeyDistributedLock implements DistributedLock {
private static final String PREFIX = "stirling:lock:";
private static final RedisScript<Long> RELEASE_SCRIPT =
new DefaultRedisScript<>(
"if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) else return 0 end",
Long.class);
private static final RedisScript<Long> RENEW_SCRIPT =
new DefaultRedisScript<>(
"if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('pexpire', KEYS[1], ARGV[2]) else return 0 end",
Long.class);
private final StringRedisTemplate template;
@Override
public Optional<LockHandle> tryAcquire(String lockKey, Duration leaseTime) {
String key = PREFIX + lockKey;
String value = UUID.randomUUID().toString();
Boolean ok = template.opsForValue().setIfAbsent(key, value, leaseTime);
if (Boolean.TRUE.equals(ok)) {
return Optional.of(new ValkeyHandle(template, key, value));
}
return Optional.empty();
}
private static final class ValkeyHandle implements LockHandle {
private final StringRedisTemplate template;
private final String key;
private final String value;
private boolean released;
ValkeyHandle(StringRedisTemplate template, String key, String value) {
this.template = template;
this.key = key;
this.value = value;
}
@Override
public synchronized void release() {
if (released) {
return;
}
released = true;
template.execute(RELEASE_SCRIPT, Collections.singletonList(key), value);
}
@Override
public synchronized boolean renew(Duration leaseTime) {
if (released) {
return false;
}
try {
Long result =
template.execute(
RENEW_SCRIPT,
Collections.singletonList(key),
value,
Long.toString(leaseTime.toMillis()));
return result != null && result == 1L;
} catch (RuntimeException ex) {
log.warn(
"Lock renew failed for {} (treated as lost lease): {}",
key,
ex.getMessage());
return false;
}
}
}
}
@@ -1,116 +0,0 @@
package stirling.software.proprietary.cluster.valkey;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Collection;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import org.springframework.data.redis.core.Cursor;
import org.springframework.data.redis.core.RedisCallback;
import org.springframework.data.redis.core.ScanOptions;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;
import lombok.RequiredArgsConstructor;
import stirling.software.common.cluster.ClusterNode;
import stirling.software.common.cluster.InstanceRegistry;
/**
* Valkey-backed {@link InstanceRegistry}. Each node is stored as a hash with a TTL equal to the
* configured heartbeat TTL; the heartbeat re-arms the TTL.
*/
@Component
@RequiredArgsConstructor
@ConditionalOnValkeyBackplane
public class ValkeyInstanceRegistry implements InstanceRegistry {
private static final String PREFIX = "stirling:nodes:";
private final StringRedisTemplate template;
@Override
public void register(ClusterNode node, Duration heartbeatTtl) {
String key = PREFIX + node.nodeId();
long ttlMs = heartbeatTtl.toMillis();
Map<String, String> fields = new LinkedHashMap<>();
fields.put("nodeId", node.nodeId());
fields.put("internalAddress", node.internalAddress());
fields.put("role", node.role());
fields.put("lastHeartbeat", node.lastHeartbeat().toString());
// MULTI/EXEC so the hash fields and the TTL commit together. Without this, a crash
// between HSET and EXPIRE leaves the hash with no TTL: it never expires, masks the
// dead node as alive, and only a subsequent successful register() would re-arm it.
template.execute(
(RedisCallback<Object>)
connection -> {
connection.multi();
byte[] keyBytes = key.getBytes(StandardCharsets.UTF_8);
Map<byte[], byte[]> hashBytes = new LinkedHashMap<>();
for (Map.Entry<String, String> f : fields.entrySet()) {
hashBytes.put(
f.getKey().getBytes(StandardCharsets.UTF_8),
f.getValue().getBytes(StandardCharsets.UTF_8));
}
connection.hashCommands().hMSet(keyBytes, hashBytes);
connection.keyCommands().pExpire(keyBytes, ttlMs);
connection.exec();
return null;
});
}
@Override
public Optional<ClusterNode> lookup(String nodeId) {
return readNode(PREFIX + nodeId);
}
@Override
public Collection<ClusterNode> activeNodes() {
// SCAN, not KEYS - KEYS blocks the Valkey server for the duration of the walk.
ScanOptions options = ScanOptions.scanOptions().match(PREFIX + "*").count(256).build();
List<ClusterNode> nodes = new ArrayList<>();
try (Cursor<String> cursor = template.scan(options)) {
while (cursor.hasNext()) {
readNode(cursor.next()).ifPresent(nodes::add);
}
}
return nodes;
}
@Override
public void deregister(String nodeId) {
template.delete(PREFIX + nodeId);
}
private Optional<ClusterNode> readNode(String key) {
Map<Object, Object> entries = template.opsForHash().entries(key);
if (entries == null || entries.isEmpty()) {
return Optional.empty();
}
Object nodeId = entries.get("nodeId");
if (nodeId == null) {
return Optional.empty();
}
Instant heartbeat = Instant.now();
Object hb = entries.get("lastHeartbeat");
if (hb != null) {
try {
heartbeat = Instant.parse(hb.toString());
} catch (RuntimeException ignored) {
// keep default
}
}
return Optional.of(
new ClusterNode(
nodeId.toString(),
String.valueOf(entries.getOrDefault("internalAddress", "")),
heartbeat,
String.valueOf(entries.getOrDefault("role", "BOTH"))));
}
}
@@ -1,297 +0,0 @@
package stirling.software.proprietary.cluster.valkey;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import org.springframework.data.redis.core.Cursor;
import org.springframework.data.redis.core.RedisCallback;
import org.springframework.data.redis.core.ScanOptions;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.cluster.JobStore;
import stirling.software.common.cluster.JobStoreEntry;
/**
* Valkey-backed {@link JobStore}. Each job is one hash; a reverse index maps fileId to jobId.
*
* <p><b>put() atomicity:</b> the hash fields, the per-job TTL, and the reverse-index entries are
* issued inside a single pipelined Redis transaction (MULTI/EXEC). A partial failure cannot leave
* the hash without a TTL or with half the file→job index entries written.
*/
@Component
@RequiredArgsConstructor
@ConditionalOnValkeyBackplane
@Slf4j
public class ValkeyJobStore implements JobStore {
private static final String JOB_PREFIX = "stirling:job:";
private static final String FILE_INDEX_PREFIX = "stirling:file2job:";
private static final ObjectMapper MAPPER = new ObjectMapper();
private static final TypeReference<List<String>> LIST_STRING = new TypeReference<>() {};
private static final TypeReference<Map<String, String>> MAP_STRING = new TypeReference<>() {};
private final StringRedisTemplate template;
@Override
public void put(JobStoreEntry entry, Duration ttl) {
String key = JOB_PREFIX + entry.jobId();
long ttlMs = ttl.toMillis();
Map<String, String> fields = new LinkedHashMap<>();
fields.put("jobId", entry.jobId());
fields.put("state", entry.state().name());
fields.put("owningNodeId", entry.owningNodeId() == null ? "" : entry.owningNodeId());
if (entry.createdAt() != null) {
fields.put("createdAt", entry.createdAt().toString());
}
if (entry.completedAt() != null) {
fields.put("completedAt", entry.completedAt().toString());
}
if (entry.error() != null) {
fields.put("error", entry.error());
}
fields.put("fileIds", writeJson(entry.fileIds() == null ? List.of() : entry.fileIds()));
fields.put(
"resultMeta",
writeJson(entry.resultMeta() == null ? Map.of() : entry.resultMeta()));
// Build pipelined MULTI/EXEC so the hash, its TTL, and every reverse-index entry
// commit atomically.
template.execute(
(RedisCallback<Object>)
connection -> {
connection.multi();
byte[] keyBytes = key.getBytes(StandardCharsets.UTF_8);
Map<byte[], byte[]> hashBytes = new LinkedHashMap<>();
for (Map.Entry<String, String> f : fields.entrySet()) {
hashBytes.put(
f.getKey().getBytes(StandardCharsets.UTF_8),
f.getValue().getBytes(StandardCharsets.UTF_8));
}
connection.hashCommands().hMSet(keyBytes, hashBytes);
connection.keyCommands().pExpire(keyBytes, ttlMs);
if (entry.fileIds() != null) {
for (String fileId : entry.fileIds()) {
byte[] idxKey =
(FILE_INDEX_PREFIX + fileId)
.getBytes(StandardCharsets.UTF_8);
connection
.stringCommands()
.set(
idxKey,
entry.jobId().getBytes(StandardCharsets.UTF_8));
connection.keyCommands().pExpire(idxKey, ttlMs);
}
}
connection.exec();
return null;
});
}
@Override
public Optional<JobStoreEntry> get(String jobId) {
return readEntry(JOB_PREFIX + jobId);
}
@Override
public void delete(String jobId) {
// WATCH/MULTI/EXEC: read fileIds INSIDE the watched scope so a concurrent put() that
// adds new fileIds between our read and EXEC aborts the transaction. Without this guard,
// an interleaved put() that grows fileIds would leave orphaned reverse-index entries
// pointing at the deleted jobId until their TTL expires. One retry handles the common
// case; further contention falls through to lazy TTL cleanup (acceptable - this is an
// eviction path, not a correctness primitive).
String jobKey = JOB_PREFIX + jobId;
byte[] jobKeyBytes = jobKey.getBytes(StandardCharsets.UTF_8);
for (int attempt = 0; attempt < 2; attempt++) {
Boolean committed =
template.execute(
(RedisCallback<Boolean>)
connection -> {
connection.watch(jobKeyBytes);
Map<byte[], byte[]> hash =
connection.hashCommands().hGetAll(jobKeyBytes);
List<byte[]> keysToDelete = new ArrayList<>();
keysToDelete.add(jobKeyBytes);
if (hash != null) {
byte[] fileIdsBytes =
hash.get(
"fileIds"
.getBytes(
StandardCharsets
.UTF_8));
if (fileIdsBytes != null) {
List<String> fileIds =
readJsonList(
new String(
fileIdsBytes,
StandardCharsets.UTF_8),
jobKey);
for (String fileId : fileIds) {
keysToDelete.add(
(FILE_INDEX_PREFIX + fileId)
.getBytes(
StandardCharsets
.UTF_8));
}
}
}
connection.multi();
for (byte[] key : keysToDelete) {
connection.keyCommands().del(key);
}
List<Object> results = connection.exec();
// exec() returns null when WATCH detected a concurrent
// write; spring-data-redis surfaces this as either null
// or empty depending on the driver path.
return results != null && !results.isEmpty();
});
if (Boolean.TRUE.equals(committed)) {
return;
}
}
log.warn(
"JobStore.delete({}) lost two WATCH races to concurrent put(); reverse-index"
+ " entries may linger until TTL expiry",
jobId);
}
@Override
public boolean exists(String jobId) {
Boolean exists = template.hasKey(JOB_PREFIX + jobId);
return Boolean.TRUE.equals(exists);
}
@Override
public Optional<String> findJobIdByFileId(String fileId) {
return Optional.ofNullable(template.opsForValue().get(FILE_INDEX_PREFIX + fileId));
}
@Override
public Collection<JobStoreEntry> all() {
// SCAN, not KEYS - KEYS blocks the Valkey server for the duration of the walk.
ScanOptions options = ScanOptions.scanOptions().match(JOB_PREFIX + "*").count(256).build();
List<JobStoreEntry> result = new ArrayList<>();
try (Cursor<String> cursor = template.scan(options)) {
while (cursor.hasNext()) {
readEntry(cursor.next()).ifPresent(result::add);
}
}
return result;
}
private Optional<JobStoreEntry> readEntry(String key) {
Map<Object, Object> entries = template.opsForHash().entries(key);
if (entries == null || entries.isEmpty()) {
return Optional.empty();
}
Object jobId = entries.get("jobId");
if (jobId == null) {
return Optional.empty();
}
Instant createdAt = parseInstant(entries.get("createdAt"), key, "createdAt");
Instant completedAt = parseInstant(entries.get("completedAt"), key, "completedAt");
List<String> fileIds = parseList(entries.get("fileIds"), key);
Map<String, String> resultMeta = parseMap(entries.get("resultMeta"), key);
String stateName =
String.valueOf(
entries.getOrDefault("state", JobStoreEntry.JobState.PENDING.name()));
JobStoreEntry.JobState state;
try {
state = JobStoreEntry.JobState.valueOf(stateName);
} catch (IllegalArgumentException ex) {
log.warn("Unrecognised job state '{}' in {}, defaulting to PENDING", stateName, key);
state = JobStoreEntry.JobState.PENDING;
}
String owningNodeId = String.valueOf(entries.getOrDefault("owningNodeId", ""));
String error = entries.get("error") == null ? null : entries.get("error").toString();
return Optional.of(
new JobStoreEntry(
jobId.toString(),
state,
owningNodeId,
createdAt,
completedAt,
error,
fileIds,
resultMeta));
}
private Instant parseInstant(Object v, String key, String field) {
if (v == null) {
return null;
}
try {
return Instant.parse(v.toString());
} catch (RuntimeException e) {
log.warn(
"JobStore {} field '{}' has malformed timestamp '{}' - treating as missing",
key,
field,
v);
return null;
}
}
private List<String> parseList(Object v, String key) {
if (v == null) {
return new ArrayList<>();
}
return readJsonList(v.toString(), key);
}
private Map<String, String> parseMap(Object v, String key) {
if (v == null) {
return new HashMap<>();
}
try {
return MAPPER.readValue(v.toString(), MAP_STRING);
} catch (JsonProcessingException e) {
log.warn(
"JobStore {} field 'resultMeta' is not valid JSON '{}' - treating as empty",
key,
v);
return new HashMap<>();
}
}
private static String writeJson(Object value) {
try {
return MAPPER.writeValueAsString(value);
} catch (JsonProcessingException e) {
// The shapes we serialize are simple List<String> / Map<String,String>; Jackson
// can encode these without escapes that fail. Surface anything unexpected loud and
// early rather than persisting a half-serialized field that would re-throw on read.
throw new IllegalStateException("Failed to JSON-serialize JobStore field", e);
}
}
private List<String> readJsonList(String json, String key) {
try {
List<String> parsed = MAPPER.readValue(json, LIST_STRING);
return parsed == null ? new ArrayList<>() : parsed;
} catch (JsonProcessingException e) {
log.warn(
"JobStore {} field 'fileIds' is not valid JSON '{}' - treating as empty",
key,
json);
return new ArrayList<>();
}
}
}
@@ -1,62 +0,0 @@
package stirling.software.proprietary.cluster.valkey;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
import java.util.concurrent.TimeUnit;
import org.springframework.data.redis.core.Cursor;
import org.springframework.data.redis.core.ScanOptions;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.stereotype.Component;
import lombok.RequiredArgsConstructor;
import stirling.software.common.cluster.KeyValueCache;
@Component
@RequiredArgsConstructor
@ConditionalOnValkeyBackplane
public class ValkeyKeyValueCache implements KeyValueCache {
private static final String PREFIX = "stirling:kv:";
private final StringRedisTemplate template;
@Override
public void put(String namespace, String key, String value, Duration ttl) {
template.opsForValue()
.set(buildKey(namespace, key), value, ttl.toMillis(), TimeUnit.MILLISECONDS);
}
@Override
public Optional<String> get(String namespace, String key) {
return Optional.ofNullable(template.opsForValue().get(buildKey(namespace, key)));
}
@Override
public void evict(String namespace, String key) {
template.delete(buildKey(namespace, key));
}
@Override
public void evictNamespace(String namespace) {
// SCAN, not KEYS: KEYS blocks the server until it has walked the entire keyspace.
ScanOptions options =
ScanOptions.scanOptions().match(PREFIX + namespace + ":*").count(256).build();
List<String> keys = new ArrayList<>();
try (Cursor<String> cursor = template.scan(options)) {
while (cursor.hasNext()) {
keys.add(cursor.next());
}
}
if (!keys.isEmpty()) {
template.delete(keys);
}
}
private String buildKey(String namespace, String key) {
return PREFIX + namespace + ":" + key;
}
}
@@ -1,82 +0,0 @@
package stirling.software.proprietary.cluster.valkey;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.stereotype.Component;
import io.github.bucket4j.BucketConfiguration;
import io.github.bucket4j.ConsumptionProbe;
import io.github.bucket4j.distributed.BucketProxy;
import io.github.bucket4j.distributed.proxy.ProxyManager;
import io.github.bucket4j.redis.lettuce.Bucket4jLettuce;
import io.lettuce.core.AbstractRedisClient;
import io.lettuce.core.RedisClient;
import jakarta.annotation.PostConstruct;
import jakarta.annotation.PreDestroy;
import stirling.software.common.cluster.RateLimitStore;
/**
* Valkey-backed token-bucket rate limiting via Bucket4j's Lettuce ProxyManager.
*
* <p>Replaces the earlier hand-rolled INCR+EXPIRE Lua fixed-window script. The fixed-window impl
* could allow a caller to spend the full bucket at second 59 of one window and the full bucket
* again at second 1 of the next window (effective burst of 2x capacity at boundaries). The Bucket4j
* token bucket refills continuously and removes that boundary doubling, giving cross-node parity
* with the in-process {@code InProcessRateLimitStore} which already uses Bucket4j.
*/
@Component
@ConditionalOnValkeyBackplane
public class ValkeyRateLimitStore implements RateLimitStore {
private static final String PREFIX = "stirling:rl:";
private final LettuceConnectionFactory connectionFactory;
private ProxyManager<byte[]> proxyManager;
public ValkeyRateLimitStore(LettuceConnectionFactory connectionFactory) {
this.connectionFactory = connectionFactory;
}
@PostConstruct
void initProxyManager() {
AbstractRedisClient client = connectionFactory.getNativeClient();
if (!(client instanceof RedisClient redisClient)) {
throw new IllegalStateException(
"ValkeyRateLimitStore requires a standalone Lettuce RedisClient; got "
+ (client == null ? "null" : client.getClass().getName())
+ " (cluster client not yet supported by this rate limit impl)");
}
this.proxyManager = Bucket4jLettuce.casBasedBuilder(redisClient).build();
}
@PreDestroy
void shutdown() {
// Lettuce client lifecycle is owned by Spring (LettuceConnectionFactory#destroy), so we
// only drop the proxy reference. No explicit close needed.
proxyManager = null;
}
@Override
public RateLimitDecision tryConsume(String bucketKey, long capacity, Duration refillPeriod) {
byte[] key = (PREFIX + bucketKey).getBytes(StandardCharsets.UTF_8);
// Greedy refill of capacity tokens per refillPeriod, matching InProcessRateLimitStore
// semantics (continuously refilling, no fixed-window boundary doubling).
BucketConfiguration cfg =
BucketConfiguration.builder()
.addLimit(
stage ->
stage.capacity(capacity)
.refillGreedy(capacity, refillPeriod))
.build();
BucketProxy bucket = proxyManager.builder().build(key, () -> cfg);
ConsumptionProbe probe = bucket.tryConsumeAndReturnRemaining(1);
if (probe.isConsumed()) {
return new RateLimitDecision(true, probe.getRemainingTokens(), 0L);
}
return new RateLimitDecision(false, 0L, probe.getNanosToWaitForRefill());
}
}
@@ -11,7 +11,7 @@ import jakarta.validation.constraints.NotNull;
import lombok.Data;
@Data
@Schema(description = "Run an AI workflow against one or more PDF files")
@Schema(description = "Run an AI workflow")
public class AiWorkflowRequest {
@NotNull
@@ -49,7 +49,11 @@ public class EEAppConfig {
@Profile("security & !saas")
@Bean(name = "SSOAutoLogin")
public boolean ssoAutoLogin() {
return applicationProperties.getPremium().getProFeatures().isSsoAutoLogin();
boolean enabled = applicationProperties.getPremium().getProFeatures().isSsoAutoLogin();
if (enabled) {
licenseKeyChecker.requireProOrEnterprise("premium.proFeatures.ssoAutoLogin=true");
}
return enabled;
}
// TODO: Remove post migration
@@ -32,7 +32,10 @@ public class LicenseKeyChecker {
private final UserLicenseSettingsService licenseSettingsService;
private License premiumEnabledResult = License.NORMAL;
// volatile: written by evaluateLicense() on the @Scheduled refresh thread, read by request
// threads via getPremiumLicenseEnabledResult() / requireProOrEnterprise(). Ensures readers see
// the latest tier rather than a stale cached value.
private volatile License premiumEnabledResult = License.NORMAL;
public LicenseKeyChecker(
KeygenLicenseVerifier licenseService,
@@ -133,4 +136,16 @@ public class LicenseKeyChecker {
public License getPremiumLicenseEnabledResult() {
return premiumEnabledResult;
}
/**
* Throws {@link IllegalStateException} if the current license is not Pro or Enterprise. Used by
* boot-time gates to fail fast when an operator enables a premium-only setting without a valid
* license. {@code configuredAs} is the human-readable property path (e.g. {@code
* "storage.provider=s3"}) and appears in the exception message.
*/
public void requireProOrEnterprise(String configuredAs) {
if (premiumEnabledResult != License.SERVER && premiumEnabledResult != License.ENTERPRISE) {
throw new IllegalStateException(configuredAs + " requires a Pro or Enterprise license");
}
}
}
@@ -17,6 +17,7 @@ import lombok.extern.slf4j.Slf4j;
import stirling.software.common.annotations.AutoJobPostMapping;
import stirling.software.common.annotations.api.GeneralApi;
import stirling.software.common.enumeration.ResourceWeight;
import stirling.software.proprietary.security.model.api.Email;
import stirling.software.proprietary.security.service.EmailService;
@@ -39,7 +40,10 @@ public class EmailController {
* attachment.
* @return ResponseEntity with success or error message.
*/
@AutoJobPostMapping(consumes = MediaType.MULTIPART_FORM_DATA_VALUE, value = "/send-email")
@AutoJobPostMapping(
consumes = MediaType.MULTIPART_FORM_DATA_VALUE,
value = "/send-email",
resourceWeight = ResourceWeight.SMALL_WEIGHT)
@Operation(
summary = "Send an email with an attachment",
description =
@@ -2,9 +2,9 @@ package stirling.software.proprietary.security.filter;
import java.io.IOException;
import java.time.Duration;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import org.apache.commons.codec.digest.DigestUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Profile;
import org.springframework.http.HttpStatus;
@@ -15,6 +15,9 @@ import org.springframework.security.core.userdetails.UserDetails;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;
import io.github.bucket4j.Bandwidth;
import io.github.bucket4j.Bucket;
import io.github.bucket4j.ConsumptionProbe;
import io.github.pixee.security.Newlines;
import jakarta.servlet.FilterChain;
@@ -22,28 +25,22 @@ import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import stirling.software.common.cluster.RateLimitStore;
import stirling.software.common.cluster.RateLimitStore.RateLimitDecision;
import stirling.software.common.model.enumeration.Role;
import stirling.software.common.util.RegexPatternUtils;
import stirling.software.proprietary.cluster.ClusterMetrics;
@Component
@Profile("!saas")
public class UserBasedRateLimitingFilter extends OncePerRequestFilter {
private final RateLimitStore rateLimitStore;
private final Map<String, Bucket> apiBuckets = new ConcurrentHashMap<>();
private final Map<String, Bucket> webBuckets = new ConcurrentHashMap<>();
@Qualifier("rateLimit")
private final boolean rateLimit;
@Autowired(required = false)
private ClusterMetrics clusterMetrics;
public UserBasedRateLimitingFilter(
@Qualifier("rateLimit") boolean rateLimit, RateLimitStore rateLimitStore) {
public UserBasedRateLimitingFilter(@Qualifier("rateLimit") boolean rateLimit) {
this.rateLimit = rateLimit;
this.rateLimitStore = rateLimitStore;
}
@Override
@@ -51,44 +48,54 @@ public class UserBasedRateLimitingFilter extends OncePerRequestFilter {
HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)
throws ServletException, IOException {
if (!rateLimit) {
// If rateLimit is not enabled, just pass all requests without rate limiting
filterChain.doFilter(request, response);
return;
}
String method = request.getMethod();
if (!"POST".equalsIgnoreCase(method)) {
// If the request is not a POST, just pass it through without rate limiting
filterChain.doFilter(request, response);
return;
}
String identifier;
String identifier = null;
// Check for API key in the request headers
String apiKey = request.getHeader("X-API-KEY");
if (apiKey != null && !apiKey.trim().isEmpty()) {
// Hash the API key so the raw value never appears in any Valkey rate-limit bucket key.
identifier = "API_KEY_" + DigestUtils.sha256Hex(apiKey);
identifier = // Prefix to distinguish between API keys and usernames
"API_KEY_" + apiKey;
} else {
Authentication authentication = SecurityContextHolder.getContext().getAuthentication();
// AnonymousAuthenticationToken.isAuthenticated() == true but principal is
// "anonymousUser";
// guard the cast so anonymous requests fall through to the remote-addr branch.
if (authentication != null
&& authentication.isAuthenticated()
&& authentication.getPrincipal() instanceof UserDetails userDetails) {
if (authentication != null && authentication.isAuthenticated()) {
UserDetails userDetails = (UserDetails) authentication.getPrincipal();
identifier = userDetails.getUsername();
} else {
identifier = request.getRemoteAddr();
}
}
// If neither API key nor an authenticated user is present, use IP address
if (identifier == null) {
identifier = request.getRemoteAddr();
}
Role userRole =
getRoleFromAuthentication(SecurityContextHolder.getContext().getAuthentication());
String scope;
int limitPerDay;
if (request.getHeader("X-API-KEY") != null) {
scope = "api:";
limitPerDay = userRole.getApiCallsPerDay();
// It's an API call
processRequest(
userRole.getApiCallsPerDay(),
identifier,
apiBuckets,
request,
response,
filterChain);
} else {
scope = "web:";
limitPerDay = userRole.getWebCallsPerDay();
// It's a Web UI call
processRequest(
userRole.getWebCallsPerDay(),
identifier,
webBuckets,
request,
response,
filterChain);
}
processRequest(limitPerDay, scope + identifier, request, response, filterChain);
}
private Role getRoleFromAuthentication(Authentication authentication) {
@@ -101,50 +108,43 @@ public class UserBasedRateLimitingFilter extends OncePerRequestFilter {
}
}
}
return Role.WEB_ONLY_USER; // no matching authority - use most restrictive bucket
throw new IllegalStateException("User does not have a valid role.");
}
private void processRequest(
int limitPerDay,
String bucketKey,
String identifier,
Map<String, Bucket> buckets,
HttpServletRequest request,
HttpServletResponse response,
FilterChain filterChain)
throws IOException, ServletException {
RateLimitDecision probe;
try {
probe = rateLimitStore.tryConsume(bucketKey, limitPerDay, Duration.ofDays(1));
} catch (RuntimeException ex) {
// Fail OPEN: a rate-limit backend outage (e.g. Valkey unreachable in cluster mode)
// must not turn every POST into a 500. Availability beats strict enforcement here -
// allow the request through and log so the outage stays visible. The in-process store
// never throws, so single-node behaviour is unchanged.
logger.warn(
"Rate-limit backend unavailable for "
+ bucketKey
+ "; allowing request (fail-open): "
+ ex.getMessage());
filterChain.doFilter(request, response);
return;
}
if (probe.allowed()) {
Bucket userBucket = buckets.computeIfAbsent(identifier, k -> createUserBucket(limitPerDay));
ConsumptionProbe probe = userBucket.tryConsumeAndReturnRemaining(1);
if (probe.isConsumed()) {
response.setHeader(
"X-Rate-Limit-Remaining",
stripNewlines(Newlines.stripAll(Long.toString(probe.remainingTokens()))));
stripNewlines(Newlines.stripAll(Long.toString(probe.getRemainingTokens()))));
filterChain.doFilter(request, response);
} else {
long waitForRefill = probe.nanosToWaitForRefill() / 1_000_000_000;
long waitForRefill = probe.getNanosToWaitForRefill() / 1_000_000_000;
response.setStatus(HttpStatus.TOO_MANY_REQUESTS.value());
response.setHeader(
"X-Rate-Limit-Retry-After-Seconds",
Newlines.stripAll(String.valueOf(waitForRefill)));
response.getWriter().write("Rate limit exceeded for POST requests.");
if (clusterMetrics != null) {
clusterMetrics.recordRateLimitReject();
}
}
}
private Bucket createUserBucket(int limitPerDay) {
Bandwidth limit =
Bandwidth.builder()
.capacity(limitPerDay)
.refillIntervally(limitPerDay, Duration.ofDays(1))
.build();
return Bucket.builder().addLimit(limit).build();
}
private static String stripNewlines(final String s) {
return RegexPatternUtils.getInstance().getNewlineCharsPattern().matcher(s).replaceAll("");
}
@@ -1,6 +1,10 @@
package stirling.software.proprietary.security.service;
import java.util.Collections;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
import java.util.TreeSet;
import org.springframework.security.authentication.LockedException;
import org.springframework.security.oauth2.client.oidc.userinfo.OidcUserRequest;
@@ -8,6 +12,8 @@ import org.springframework.security.oauth2.client.oidc.userinfo.OidcUserService;
import org.springframework.security.oauth2.client.userinfo.OAuth2UserService;
import org.springframework.security.oauth2.core.OAuth2AuthenticationException;
import org.springframework.security.oauth2.core.OAuth2Error;
import org.springframework.security.oauth2.core.oidc.OidcIdToken;
import org.springframework.security.oauth2.core.oidc.OidcUserInfo;
import org.springframework.security.oauth2.core.oidc.user.DefaultOidcUser;
import org.springframework.security.oauth2.core.oidc.user.OidcUser;
@@ -39,20 +45,37 @@ public class CustomOAuth2UserService implements OAuth2UserService<OidcUserReques
@Override
public OidcUser loadUser(OidcUserRequest userRequest) throws OAuth2AuthenticationException {
String registrationId = userRequest.getClientRegistration().getRegistrationId();
boolean debugLogging = Boolean.TRUE.equals(oauth2Properties.getDebugLogging());
// Resolved inside the try so a bad/null useAsUsername (IllegalArgumentException from
// valueOf, or NPE on toUpperCase) is caught and wrapped as OAuth2AuthenticationException
// by the existing handlers below, matching the pre-debugLogging behaviour.
String usernameAttributeKey = null;
try {
OidcUser user = delegate.loadUser(userRequest);
String usernameAttributeKey =
usernameAttributeKey =
UsernameAttribute.valueOf(oauth2Properties.getUseAsUsername().toUpperCase())
.getName();
OidcUser user = delegate.loadUser(userRequest);
if (debugLogging) {
logClaimDump(
"OAuth2/OIDC login claims received",
registrationId,
usernameAttributeKey,
user.getIdToken(),
user.getUserInfo(),
user.getAttributes(),
false);
}
// Extract SSO provider information
String ssoProviderId = user.getSubject(); // Standard OIDC 'sub' claim
String ssoProvider = userRequest.getClientRegistration().getRegistrationId();
String username = user.getAttribute(usernameAttributeKey);
log.debug(
"OAuth2 login - Provider: {}, ProviderId: {}, Username: {}",
ssoProvider,
registrationId,
ssoProviderId,
username);
@@ -79,10 +102,154 @@ public class CustomOAuth2UserService implements OAuth2UserService<OidcUserReques
usernameAttributeKey);
} catch (IllegalArgumentException e) {
log.error("Error loading OIDC user: {}", e.getMessage());
// Only emit the claim dump if we successfully resolved usernameAttributeKey. A null
// value here means UsernameAttribute.valueOf rejected the configured useAsUsername
// before delegate.loadUser ran — that error message is self-explanatory and a claim
// dump would have no resolved-key to compare against.
if (debugLogging && usernameAttributeKey != null) {
// The DefaultOidcUser constructor (or our own checks) rejected the chosen
// username attribute. Dump the claims we DID receive so the operator can pick
// a different value for security.oauth2.useAsUsername.
logClaimDump(
"OAuth2/OIDC login FAILED - dumping received claims",
registrationId,
usernameAttributeKey,
userRequest.getIdToken(),
null,
userRequest.getIdToken() == null
? Collections.emptyMap()
: userRequest.getIdToken().getClaims(),
true);
}
throw new OAuth2AuthenticationException(new OAuth2Error(e.getMessage()), e);
} catch (Exception e) {
log.error("Unexpected error loading OIDC user", e);
if (debugLogging && usernameAttributeKey != null && userRequest.getIdToken() != null) {
logClaimDump(
"OAuth2/OIDC login FAILED (unexpected error) - dumping ID token claims",
registrationId,
usernameAttributeKey,
userRequest.getIdToken(),
null,
userRequest.getIdToken().getClaims(),
true);
}
throw new OAuth2AuthenticationException("Unexpected error during authentication");
}
}
/**
* Emits a multi-line diagnostic dump of the claims returned by the OAuth2/OIDC provider. Only
* invoked when {@code security.oauth2.debugLogging=true}.
*
* @param banner short title for the log block
* @param registrationId Spring client registration id (e.g. "demarest", "keycloak")
* @param usernameAttributeKey the claim key the application is configured to use as username
* @param idToken the decoded ID token, may be null on unexpected failures
* @param userInfo the decoded UserInfo response, may be null if the provider returned none
* @param mergedAttributes the merged attribute map Spring uses for {@code getAttribute()}
* @param failure true if logging in the error path (uses ERROR level), false for INFO
*/
private void logClaimDump(
String banner,
String registrationId,
String usernameAttributeKey,
OidcIdToken idToken,
OidcUserInfo userInfo,
Map<String, Object> mergedAttributes,
boolean failure) {
StringBuilder sb = new StringBuilder();
sb.append("\n========== [OAUTH2 DEBUG] ").append(banner).append(" ==========\n");
sb.append("Provider registrationId : ").append(registrationId).append('\n');
sb.append("Configured useAsUsername: ")
.append(oauth2Properties.getUseAsUsername())
.append(" (looks up claim key '")
.append(usernameAttributeKey)
.append("')\n");
if (idToken != null) {
Map<String, Object> idClaims = idToken.getClaims();
sb.append("\n-- ID token claims (")
.append(idClaims == null ? 0 : idClaims.size())
.append(") --\n");
appendClaims(sb, idClaims);
sb.append("ID token issued at : ").append(idToken.getIssuedAt()).append('\n');
sb.append("ID token expires at: ").append(idToken.getExpiresAt()).append('\n');
} else {
sb.append("\n-- ID token: <null> --\n");
}
if (userInfo != null && userInfo.getClaims() != null) {
sb.append("\n-- UserInfo endpoint claims (")
.append(userInfo.getClaims().size())
.append(") --\n");
appendClaims(sb, userInfo.getClaims());
} else {
sb.append("\n-- UserInfo endpoint claims: none returned --\n");
}
if (mergedAttributes != null) {
sb.append("\n-- Merged attribute keys available to useAsUsername: ")
.append(new TreeSet<>(mergedAttributes.keySet()))
.append("\n");
Object resolved = mergedAttributes.get(usernameAttributeKey);
sb.append("-- Value at '")
.append(usernameAttributeKey)
.append("' : ")
.append(resolved == null ? "<NULL — this is why login fails>" : resolved)
.append('\n');
if (resolved == null) {
Set<String> hints = suggestUsernameClaims(mergedAttributes.keySet());
if (!hints.isEmpty()) {
sb.append(
"-- Hint: the following claim(s) are present and map to a"
+ " known UsernameAttribute value — try setting"
+ " security.oauth2.useAsUsername to one of: ")
.append(hints)
.append('\n');
}
}
}
sb.append(
"\nWARNING: this block contains PII. Set security.oauth2.debugLogging=false once"
+ " troubleshooting is complete.\n");
sb.append("========== [/OAUTH2 DEBUG] ==========");
if (failure) {
log.error(sb.toString());
} else {
log.info(sb.toString());
}
}
private static void appendClaims(StringBuilder sb, Map<String, Object> claims) {
if (claims == null || claims.isEmpty()) {
sb.append(" (no claims)\n");
return;
}
// Sort for stable, scannable output
new TreeSet<>(claims.keySet())
.forEach(
key -> {
Object value = claims.get(key);
sb.append(" ").append(key).append(" = ").append(value).append('\n');
});
}
/**
* Returns the intersection of the claim keys the provider actually returned and the keys that
* {@link UsernameAttribute} accepts — i.e. valid values the operator could put in {@code
* security.oauth2.useAsUsername} to make this login work.
*/
private static Set<String> suggestUsernameClaims(Set<String> availableClaimKeys) {
Set<String> supported = new TreeSet<>();
for (UsernameAttribute attr : UsernameAttribute.values()) {
if (availableClaimKeys.contains(attr.getName())) {
supported.add(attr.getName());
}
}
return supported;
}
}
@@ -210,37 +210,20 @@ public class JwtService implements JwtServiceInterface {
if (specificKeyPair.isPresent()) {
keyPair = specificKeyPair.get();
} else {
Optional<PublicKey> peerKey = keyPersistenceService.resolvePublicKey(keyId);
if (peerKey.isPresent()) {
return Jwts.parser()
.verifyWith(peerKey.get())
.clockSkewSeconds(getAllowedClockSkewSeconds())
.build()
.parseSignedClaims(token)
.getPayload();
}
log.warn(
"Key ID {} not found in keystore, token may have been signed with an expired key",
keyId);
if (keyId.equals(keyPersistenceService.getActiveKey().getKeyId())) {
// Re-check local store before rotating: rotating a key still on disk
// invalidates every in-flight token signed with it.
Optional<KeyPair> localActivePair = keyPersistenceService.getKeyPair(keyId);
if (localActivePair.isPresent()) {
keyPair = localActivePair.get();
JwtVerificationKey verificationKey =
keyPersistenceService.refreshActiveKeyPair();
Optional<KeyPair> refreshedKeyPair =
keyPersistenceService.getKeyPair(verificationKey.getKeyId());
if (refreshedKeyPair.isPresent()) {
keyPair = refreshedKeyPair.get();
} else {
// Key missing everywhere - rotate to restore signing capability.
JwtVerificationKey verificationKey =
keyPersistenceService.refreshActiveKeyPair();
Optional<KeyPair> refreshedKeyPair =
keyPersistenceService.getKeyPair(verificationKey.getKeyId());
if (refreshedKeyPair.isPresent()) {
keyPair = refreshedKeyPair.get();
} else {
throw new AuthenticationFailureException(
"Failed to retrieve refreshed key pair");
}
throw new AuthenticationFailureException(
"Failed to retrieve refreshed key pair");
}
} else {
// Try to use active key as fallback
@@ -257,6 +240,7 @@ public class JwtService implements JwtServiceInterface {
}
} else {
log.debug("No key ID in token header, trying all available keys");
// Try all available keys when no keyId is present
return tryAllKeys(token, allowExpired);
}
@@ -315,6 +299,8 @@ public class JwtService implements JwtServiceInterface {
| NoSuchAlgorithmException
| InvalidKeySpecException activeKeyException) {
log.debug("Active key failed, trying all available keys from cache");
// If active key fails, try all available keys from cache
List<JwtVerificationKey> allKeys =
keyPersistenceService.getKeysEligibleForCleanup(
LocalDateTime.now().plusDays(1));
@@ -353,10 +339,13 @@ public class JwtService implements JwtServiceInterface {
@Override
public String extractToken(HttpServletRequest request) {
// Extract from Authorization header Bearer token
String authHeader = request.getHeader("Authorization");
if (authHeader != null && authHeader.startsWith("Bearer ")) {
return authHeader.substring(7);
String token = authHeader.substring(7); // Remove "Bearer " prefix
return token;
}
return null;
}
@@ -365,11 +354,22 @@ public class JwtService implements JwtServiceInterface {
return v2Enabled;
}
/** Return the {@code kid} claim from the JWT header, or null if absent or unparseable. */
/**
* Extract key ID from JWT header without validating the token.
*
* <p>Parses the Base64-encoded JWT header to retrieve the "kid" (key ID) claim. Returns null if
* the header cannot be parsed or does not contain a key ID.
*
* @param token the JWT token
* @return the key ID, or null if not found or parsing fails
*/
private String extractKeyId(String token) {
try {
String[] tokenParts = token.split("\\.");
if (tokenParts.length < 2) {
log.debug(
"Token does not have enough parts (expected at least 2, got {})",
tokenParts.length);
return null;
}
@@ -1,11 +1,9 @@
package stirling.software.proprietary.security.service;
import java.io.IOException;
import java.nio.file.AtomicMoveNotSupportedException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.security.KeyFactory;
import java.security.KeyPair;
import java.security.KeyPairGenerator;
@@ -17,7 +15,6 @@ import java.security.spec.InvalidKeySpecException;
import java.security.spec.PKCS8EncodedKeySpec;
import java.security.spec.RSAPublicKeySpec;
import java.security.spec.X509EncodedKeySpec;
import java.time.Duration;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.Base64;
@@ -31,21 +28,16 @@ import org.springframework.cache.CacheManager;
import org.springframework.cache.annotation.CacheEvict;
import org.springframework.cache.caffeine.CaffeineCache;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import jakarta.annotation.PostConstruct;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.cluster.KeyValueCache;
import stirling.software.common.configuration.InstallationPathConfig;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.proprietary.security.model.JwtVerificationKey;
/**
* SECURITY: the {@link #JWT_PUBKEY_NAMESPACE} cluster cache is trust-on-publish. Operators MUST
* restrict Valkey ACL writes to app pods, enable AUTH + TLS, and network-isolate the deployment.
* Future hardening: HMAC-signed broadcasts with a cluster master secret.
*/
@Slf4j
@Service
public class KeyPersistenceService implements KeyPersistenceServiceInterface {
@@ -53,28 +45,18 @@ public class KeyPersistenceService implements KeyPersistenceServiceInterface {
public static final String KEY_SUFFIX = ".key";
public static final String PUB_KEY_SUFFIX = ".pub";
private static final Duration JWT_PUBKEY_CLUSTER_TTL = Duration.ofHours(24);
/** Cluster KeyValueCache namespace used to broadcast public keys to peers. */
public static final String JWT_PUBKEY_NAMESPACE = "jwtkey";
private final ApplicationProperties.Security.Jwt jwtProperties;
private final CacheManager cacheManager;
private final Cache verifyingKeyCache;
private final KeyValueCache clusterKeyCache; // null in single-instance mode
private volatile JwtVerificationKey activeKey;
@Autowired
public KeyPersistenceService(
ApplicationProperties applicationProperties,
CacheManager cacheManager,
@Autowired(required = false) KeyValueCache clusterKeyCache) {
ApplicationProperties applicationProperties, CacheManager cacheManager) {
this.jwtProperties = applicationProperties.getSecurity().getJwt();
this.cacheManager = cacheManager;
this.verifyingKeyCache = cacheManager.getCache("verifyingKeys");
this.clusterKeyCache = clusterKeyCache;
}
@PostConstruct
@@ -92,6 +74,12 @@ public class KeyPersistenceService implements KeyPersistenceServiceInterface {
}
}
/**
* Load all existing JWT keys from disk into memory on startup.
*
* <p>This ensures tokens signed with previous keys remain valid after server restart. If no
* keys exist on disk, generates a new keypair.
*/
private void loadExistingKeysFromDisk() {
try {
Path keyDirectory = Paths.get(InstallationPathConfig.getPrivateKeyPath());
@@ -106,8 +94,11 @@ public class KeyPersistenceService implements KeyPersistenceServiceInterface {
try (var stream = Files.list(keyDirectory)) {
keyFiles =
stream.filter(path -> path.toString().endsWith(KEY_SUFFIX))
// most recent first
.sorted((a, b) -> b.getFileName().compareTo(a.getFileName()))
.sorted(
(a, b) ->
b.getFileName().compareTo(a.getFileName())) // Most
// recent
// first
.collect(Collectors.toList());
}
@@ -124,10 +115,11 @@ public class KeyPersistenceService implements KeyPersistenceServiceInterface {
try {
String keyId = keyFile.getFileName().toString().replace(KEY_SUFFIX, "");
// Load private key first
PrivateKey privateKey = loadPrivateKey(keyId);
// Try to load public key; generate from private key if missing (legacy
// migration).
// Try to load public key, or generate it from private key if missing
// (migration)
String encodedPublicKey;
try {
encodedPublicKey = loadPublicKey(keyId);
@@ -147,11 +139,13 @@ public class KeyPersistenceService implements KeyPersistenceServiceInterface {
log.info("Successfully migrated key: {}", keyId);
}
// Create verification key and add to cache
JwtVerificationKey verifyingKey =
new JwtVerificationKey(keyId, encodedPublicKey);
verifyingKeyCache.put(keyId, verifyingKey);
loadedCount++;
// Set the most recent key as active (first in sorted list)
if (activeKey == null) {
activeKey = verifyingKey;
log.info("Set active JWT signing key: {}", keyId);
@@ -185,6 +179,7 @@ public class KeyPersistenceService implements KeyPersistenceServiceInterface {
}
}
@Transactional
private JwtVerificationKey generateAndStoreKeypair() {
JwtVerificationKey verifyingKey = null;
@@ -193,12 +188,9 @@ public class KeyPersistenceService implements KeyPersistenceServiceInterface {
String keyId = generateKeyId();
storeKeyPair(keyId, keyPair);
String encodedPublicKey = encodePublicKey(keyPair.getPublic());
verifyingKey = new JwtVerificationKey(keyId, encodedPublicKey);
verifyingKey = new JwtVerificationKey(keyId, encodePublicKey(keyPair.getPublic()));
verifyingKeyCache.put(keyId, verifyingKey);
activeKey = verifyingKey;
// Broadcast so peer nodes can verify tokens we sign without waiting for restart.
publishToCluster(keyId, encodedPublicKey);
log.info("Generated and stored new JWT keypair: {}", keyId);
} catch (IOException e) {
log.error("Failed to generate and store keypair", e);
@@ -207,23 +199,6 @@ public class KeyPersistenceService implements KeyPersistenceServiceInterface {
return verifyingKey;
}
private void publishToCluster(String keyId, String encodedPublicKey) {
if (clusterKeyCache == null) {
return;
}
try {
clusterKeyCache.put(
JWT_PUBKEY_NAMESPACE, keyId, encodedPublicKey, JWT_PUBKEY_CLUSTER_TTL);
log.info("Broadcast JWT public key to cluster KeyValueCache: {}", keyId);
} catch (RuntimeException e) {
// Non-fatal: we still serve tokens locally; peers catch up on their next restart.
log.warn(
"Failed to broadcast JWT public key {} to cluster cache: {}",
keyId,
e.getMessage());
}
}
@Override
public JwtVerificationKey getActiveKey() {
if (activeKey == null) {
@@ -274,17 +249,6 @@ public class KeyPersistenceService implements KeyPersistenceServiceInterface {
condition = "#root.target.isKeystoreEnabled()")
public void removeKey(String keyId) {
verifyingKeyCache.evict(keyId);
// Evict cluster broadcast so peers don't keep serving the removed key for up to 24h.
if (clusterKeyCache != null && keyId != null) {
try {
clusterKeyCache.evict(JWT_PUBKEY_NAMESPACE, keyId);
} catch (RuntimeException e) {
log.warn(
"Failed to evict JWT public key {} from cluster cache: {}",
keyId,
e.getMessage());
}
}
}
@Override
@@ -341,29 +305,33 @@ public class KeyPersistenceService implements KeyPersistenceServiceInterface {
}
/**
* Store both private and public keys to disk using a temp-then-atomic-rename pattern.
* Store both private and public keys to disk.
*
* <p>The caller broadcasts the public key to peers immediately after this returns. If a peer
* learned about the keyId before the private key was fully durable, a crash mid-write followed
* by restart would lose the key while peers still serve tokens signed with it. Writing to
* {@code <file>.tmp} and moving with {@link StandardCopyOption#ATOMIC_MOVE} guarantees the
* final path either contains the fully-written payload or does not exist at all.
* <p>Private key stored as: keyId.key
*
* <p>Public key stored as: keyId.pub
*/
private void storeKeyPair(String keyId, KeyPair keyPair) throws IOException {
Path keyDirectory = Paths.get(InstallationPathConfig.getPrivateKeyPath());
// Store private key
Path privateKeyFile = keyDirectory.resolve(keyId + KEY_SUFFIX);
String encodedPrivateKey =
Base64.getEncoder().encodeToString(keyPair.getPrivate().getEncoded());
writeAtomically(privateKeyFile, encodedPrivateKey);
Files.writeString(privateKeyFile, encodedPrivateKey);
// Set read/write to only the owner (security)
privateKeyFile.toFile().setReadable(true, true);
privateKeyFile.toFile().setWritable(true, true);
privateKeyFile.toFile().setExecutable(false, false);
// Store public key
Path publicKeyFile = keyDirectory.resolve(keyId + PUB_KEY_SUFFIX);
String encodedPublicKey =
Base64.getEncoder().encodeToString(keyPair.getPublic().getEncoded());
writeAtomically(publicKeyFile, encodedPublicKey);
Files.writeString(publicKeyFile, encodedPublicKey);
// Public key can be more permissive but still restrict to owner
publicKeyFile.toFile().setReadable(true, true);
publicKeyFile.toFile().setWritable(true, true);
publicKeyFile.toFile().setExecutable(false, false);
@@ -375,31 +343,6 @@ public class KeyPersistenceService implements KeyPersistenceServiceInterface {
publicKeyFile.getFileName());
}
/**
* Write {@code contents} to {@code finalPath} so that the final path either contains the full
* payload or does not exist. Writes to a sibling {@code .tmp} file first and renames it.
* Returns silently after falling back to a non-atomic move if the filesystem does not support
* {@link StandardCopyOption#ATOMIC_MOVE}.
*/
static void writeAtomically(Path finalPath, String contents) throws IOException {
Path tmp = finalPath.resolveSibling(finalPath.getFileName().toString() + ".tmp");
Files.writeString(tmp, contents);
try {
Files.move(
tmp,
finalPath,
StandardCopyOption.REPLACE_EXISTING,
StandardCopyOption.ATOMIC_MOVE);
} catch (AtomicMoveNotSupportedException e) {
log.warn(
"Filesystem does not support atomic move for {}; falling back to non-atomic"
+ " replace. A crash between rename and fsync may leave the key partially"
+ " written.",
finalPath);
Files.move(tmp, finalPath, StandardCopyOption.REPLACE_EXISTING);
}
}
private PrivateKey loadPrivateKey(String keyId)
throws IOException, NoSuchAlgorithmException, InvalidKeySpecException {
Path keyFile =
@@ -417,6 +360,13 @@ public class KeyPersistenceService implements KeyPersistenceServiceInterface {
return keyFactory.generatePrivate(keySpec);
}
/**
* Load public key from disk.
*
* @param keyId the key identifier
* @return Base64-encoded public key string
* @throws IOException if the public key file is not found
*/
private String loadPublicKey(String keyId) throws IOException {
Path publicKeyFile =
Paths.get(InstallationPathConfig.getPrivateKeyPath())
@@ -429,12 +379,29 @@ public class KeyPersistenceService implements KeyPersistenceServiceInterface {
return Files.readString(publicKeyFile).trim();
}
/**
* Reconstruct a KeyPair from a PrivateKey.
*
* <p>For RSA keys, derives the public key from the private key.
*
* @param privateKey the RSA private key
* @return reconstructed KeyPair
* @throws NoSuchAlgorithmException if RSA algorithm is not available
* @throws InvalidKeySpecException if the key specification is invalid
*/
private KeyPair reconstructKeyPair(PrivateKey privateKey)
throws NoSuchAlgorithmException, InvalidKeySpecException {
// For RSA, we can derive the public key from the private key
KeyFactory keyFactory = KeyFactory.getInstance("RSA");
// Get the private key spec
RSAPrivateCrtKey rsaPrivateKey = (RSAPrivateCrtKey) privateKey;
// Create public key spec from private key parameters
RSAPublicKeySpec publicKeySpec =
new RSAPublicKeySpec(rsaPrivateKey.getModulus(), rsaPrivateKey.getPublicExponent());
// Generate public key
PublicKey publicKey = keyFactory.generatePublic(publicKeySpec);
return new KeyPair(publicKey, privateKey);
@@ -451,50 +418,4 @@ public class KeyPersistenceService implements KeyPersistenceServiceInterface {
KeyFactory keyFactory = KeyFactory.getInstance("RSA");
return keyFactory.generatePublic(keySpec);
}
@Override
public Optional<PublicKey> resolvePublicKey(String keyId) {
if (keyId == null || keyId.isBlank()) {
return Optional.empty();
}
// 1. Local in-memory cache (warm path - same node that signed, or already learnt).
JwtVerificationKey local = verifyingKeyCache.get(keyId, JwtVerificationKey.class);
if (local != null) {
return decodeQuietly(local.getVerifyingKey(), keyId);
}
// 2. Local disk (cold path on restart).
try {
String onDisk = loadPublicKey(keyId);
JwtVerificationKey rebuilt = new JwtVerificationKey(keyId, onDisk);
verifyingKeyCache.put(keyId, rebuilt);
return decodeQuietly(onDisk, keyId);
} catch (IOException ignored) {
// not on this node's disk - try the cluster cache
}
// 3. Cluster cache. NOT written to local cache: expireAfterWrite would outlive the
// broadcast TTL and serve stale keys after peer rotation. Valkey faults return empty.
if (clusterKeyCache != null) {
try {
Optional<String> remote = clusterKeyCache.get(JWT_PUBKEY_NAMESPACE, keyId);
if (remote.isPresent()) {
String encoded = remote.get();
log.debug("Resolved JWT public key {} from cluster KeyValueCache", keyId);
return decodeQuietly(encoded, keyId);
}
} catch (RuntimeException e) {
log.warn("Cluster key cache unavailable for keyId {}: {}", keyId, e.getMessage());
return Optional.empty();
}
}
return Optional.empty();
}
private Optional<PublicKey> decodeQuietly(String encoded, String keyId) {
try {
return Optional.of(decodePublicKey(encoded));
} catch (NoSuchAlgorithmException | InvalidKeySpecException e) {
log.warn("Could not decode public key for {}: {}", keyId, e.getMessage());
return Optional.empty();
}
}
}
@@ -26,10 +26,4 @@ public interface KeyPersistenceServiceInterface {
PublicKey decodePublicKey(String encodedKey)
throws NoSuchAlgorithmException, InvalidKeySpecException;
/**
* Resolve a public key by id, consulting the cluster cache when the key is unknown locally.
* Returns empty if the keyId is unknown anywhere.
*/
Optional<PublicKey> resolvePublicKey(String keyId);
}
@@ -5,10 +5,7 @@ import static stirling.software.proprietary.security.service.MfaService.MFA_LAST
import static stirling.software.proprietary.security.service.MfaService.MFA_REQUIRED_KEY;
import static stirling.software.proprietary.security.service.MfaService.MFA_SECRET_KEY;
import java.nio.charset.StandardCharsets;
import java.security.MessageDigest;
import java.sql.SQLException;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
@@ -19,7 +16,6 @@ import java.util.Optional;
import java.util.UUID;
import java.util.function.Supplier;
import org.apache.commons.codec.digest.DigestUtils;
import org.slf4j.MDC;
import org.springframework.context.MessageSource;
import org.springframework.context.i18n.LocaleContextHolder;
@@ -38,7 +34,6 @@ import org.springframework.transaction.annotation.Transactional;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.cluster.KeyValueCache;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.common.model.enumeration.Role;
import stirling.software.common.model.exception.UnsupportedProviderException;
@@ -84,13 +79,6 @@ public class UserService implements UserServiceInterface {
private final ApplicationProperties.Security.OAUTH2 oAuth2;
private final KeyValueCache keyValueCache;
private static final String API_KEY_CACHE_NS = "apikey";
private static final Duration API_KEY_TTL = Duration.ofSeconds(60);
private static final Duration API_KEY_NEGATIVE_TTL = Duration.ofSeconds(10);
private static final String API_KEY_NEGATIVE_MARKER = "__none__";
private final PersistentLoginRepository persistentLoginRepository;
private final UserServerCertificateService userServerCertificateService;
private final WorkflowParticipantRepository workflowParticipantRepository;
@@ -155,7 +143,11 @@ public class UserService implements UserServiceInterface {
if (user.isEmpty()) {
throw new UsernameNotFoundException("API key is not valid");
}
return new UsernamePasswordAuthenticationToken(user, null, getAuthorities(user.get()));
// Convert the user into an Authentication object
return new UsernamePasswordAuthenticationToken( // principal (typically the user)
user, // credentials (we don't expose the password or API key here)
null, // user's authorities (roles/permissions)
getAuthorities(user.get()));
}
private Collection<? extends GrantedAuthority> getAuthorities(User user) {
@@ -184,19 +176,14 @@ public class UserService implements UserServiceInterface {
private User saveUser(Optional<User> user, String apiKey) {
if (user.isPresent()) {
String previousKey = user.get().getApiKey();
user.get().setApiKey(apiKey);
User saved = userRepository.save(user.get());
// Evict the previously cached entry so peers see the rotation within negative TTL.
if (previousKey != null && !previousKey.isBlank()) {
evictApiKeyCache(previousKey);
}
return saved;
return userRepository.save(user.get());
}
throw new UsernameNotFoundException("User not found");
}
public User refreshApiKeyForUser(String username) {
// reuse the add API key method for refreshing
return addApiKeyToUser(username);
}
@@ -221,71 +208,25 @@ public class UserService implements UserServiceInterface {
}
public boolean isValidApiKey(String apiKey) {
return getUserByApiKey(apiKey).isPresent();
return userRepository.findByApiKey(apiKey).isPresent();
}
public Optional<User> getUserByApiKey(String apiKey) {
return findByApiKeyCached(apiKey);
return userRepository.findByApiKey(apiKey);
}
public Optional<User> loadUserByApiKey(String apiKey) {
Optional<User> user = findByApiKeyCached(apiKey);
Optional<User> user = userRepository.findByApiKey(apiKey);
if (user.isPresent()) {
return user;
}
// or throw an exception
return null;
}
private Optional<User> findByApiKeyCached(String apiKey) {
if (apiKey == null || apiKey.isBlank() || keyValueCache == null) {
return userRepository.findByApiKey(apiKey);
}
String keyHash = DigestUtils.sha256Hex(apiKey);
Optional<String> cached = keyValueCache.get(API_KEY_CACHE_NS, keyHash);
if (cached.isPresent()) {
String value = cached.get();
if (API_KEY_NEGATIVE_MARKER.equals(value)) {
return Optional.empty();
}
Optional<User> user = userRepository.findByUsernameIgnoreCase(value);
if (user.isPresent() && constantTimeEquals(apiKey, user.get().getApiKey())) {
return user;
}
keyValueCache.evict(API_KEY_CACHE_NS, keyHash);
}
Optional<User> user = userRepository.findByApiKey(apiKey);
if (user.isPresent()) {
keyValueCache.put(API_KEY_CACHE_NS, keyHash, user.get().getUsername(), API_KEY_TTL);
} else {
keyValueCache.put(
API_KEY_CACHE_NS, keyHash, API_KEY_NEGATIVE_MARKER, API_KEY_NEGATIVE_TTL);
}
return user;
}
/** Invalidate the cached API-key entry on rotation. */
public void evictApiKeyCache(String apiKey) {
if (apiKey != null && !apiKey.isBlank() && keyValueCache != null) {
keyValueCache.evict(API_KEY_CACHE_NS, DigestUtils.sha256Hex(apiKey));
}
}
public boolean validateApiKeyForUser(String username, String apiKey) {
Optional<User> userOpt = findByUsernameIgnoreCase(username);
return userOpt.isPresent() && constantTimeEquals(apiKey, userOpt.get().getApiKey());
}
/**
* Constant-time comparison of two API keys. {@link MessageDigest#isEqual} short-circuits only
* on null/empty inputs; for equal-length and unequal-length non-empty strings it scans every
* byte, so a remote attacker cannot infer the stored key via response-time differences.
*/
private static boolean constantTimeEquals(String provided, String stored) {
if (provided == null || stored == null) {
return false;
}
return MessageDigest.isEqual(
provided.getBytes(StandardCharsets.UTF_8), stored.getBytes(StandardCharsets.UTF_8));
return userOpt.isPresent() && apiKey.equals(userOpt.get().getApiKey());
}
@Transactional
@@ -765,13 +706,8 @@ public class UserService implements UserServiceInterface {
User updatedUser = existingUser.get();
if (!customApiKey.equals(updatedUser.getApiKey())) {
// Capture before mutation so we can evict the prior cache entry.
String previousKey = updatedUser.getApiKey();
updatedUser.setApiKey(customApiKey);
userRepository.save(updatedUser);
if (previousKey != null && !previousKey.isBlank()) {
evictApiKeyCache(previousKey);
}
}
},
() -> {
@@ -0,0 +1,95 @@
package stirling.software.proprietary.storage.config;
import java.util.Locale;
import java.util.Optional;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Configuration;
import jakarta.annotation.PostConstruct;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.proprietary.security.configuration.ee.LicenseKeyChecker;
/**
* Fails fast at boot if cluster mode is enabled with node-local storage. Validates both {@code
* storage.provider} (persistent uploads) and {@code cluster.artifactStore} (transient job-result
* files): neither may be {@code local} when {@code cluster.enabled=true}. Additionally enforces
* that any S3-backed configuration ({@code storage.provider=s3} or {@code
* cluster.artifactStore=s3}) is accompanied by a valid Pro / Enterprise license.
*/
@Configuration
@RequiredArgsConstructor
@Slf4j
public class ClusterStorageGate {
private final ApplicationProperties applicationProperties;
private final LicenseKeyChecker licenseKeyChecker;
@Value("${cluster.enabled:false}")
private boolean clusterEnabled;
@Value("${cluster.artifactStore:local}")
private String clusterArtifactStore;
@PostConstruct
void validate() {
// License enforcement runs regardless of cluster.enabled: even a single-node setup that
// selects a remote backend must hold a Pro or higher license.
ApplicationProperties.Storage storage = applicationProperties.getStorage();
if (storage != null && storage.isEnabled()) {
String provider = normalize(storage.getProvider());
if ("s3".equals(provider) || "database".equals(provider)) {
licenseKeyChecker.requireProOrEnterprise("storage.provider=" + provider);
}
}
if ("s3".equals(normalize(clusterArtifactStore))) {
licenseKeyChecker.requireProOrEnterprise("cluster.artifactStore=s3");
}
if (!clusterEnabled) {
return;
}
if (storage != null && storage.isEnabled()) {
validate(
"storage.provider",
storage.getProvider(),
"Local filesystem storage cannot be shared across cluster nodes."
+ " Configure storage.provider=s3 (with storage.s3.bucket /"
+ " endpoint / credentials) or storage.provider=database before"
+ " enabling clustering.");
}
validate(
"cluster.artifactStore",
clusterArtifactStore,
"Per-node disk cannot back transient job-result files in a multi-node"
+ " deployment; downloads would 404 whenever the load balancer routes"
+ " a follow-up request to a different node. Configure"
+ " cluster.artifactStore=s3 (reuses storage.s3.* config)"
+ " before enabling clustering.");
}
private static String normalize(String value) {
return Optional.ofNullable(value).orElse("local").trim().toLowerCase(Locale.ROOT);
}
private static void validate(String propertyName, String configuredValue, String remediation) {
String normalized =
Optional.ofNullable(configuredValue)
.orElse("local")
.trim()
.toLowerCase(Locale.ROOT);
if ("local".equals(normalized)) {
throw new IllegalStateException(
"Cluster mode (cluster.enabled=true) is incompatible with "
+ propertyName
+ "=local. "
+ remediation);
}
log.info(
"Cluster storage gate: clusterEnabled=true, {}={} -> OK", propertyName, normalized);
}
}
@@ -15,8 +15,11 @@ import lombok.extern.slf4j.Slf4j;
import stirling.software.common.configuration.InstallationPathConfig;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.proprietary.cluster.s3.S3Clients;
import stirling.software.proprietary.security.configuration.ee.LicenseKeyChecker;
import stirling.software.proprietary.storage.provider.DatabaseStorageProvider;
import stirling.software.proprietary.storage.provider.LocalStorageProvider;
import stirling.software.proprietary.storage.provider.S3StorageProvider;
import stirling.software.proprietary.storage.provider.StorageProvider;
import stirling.software.proprietary.storage.repository.StoredFileBlobRepository;
@@ -27,8 +30,9 @@ public class StorageProviderConfig {
private final ApplicationProperties applicationProperties;
private final StoredFileBlobRepository storedFileBlobRepository;
private final LicenseKeyChecker licenseKeyChecker;
@Bean
@Bean(destroyMethod = "close")
public StorageProvider storageProvider() {
boolean storageEnabled = applicationProperties.getStorage().isEnabled();
String providerName =
@@ -37,8 +41,13 @@ public class StorageProviderConfig {
.trim()
.toLowerCase(Locale.ROOT);
if ("database".equals(providerName)) {
licenseKeyChecker.requireProOrEnterprise("storage.provider=database");
return new DatabaseStorageProvider(storedFileBlobRepository);
}
if ("s3".equals(providerName)) {
licenseKeyChecker.requireProOrEnterprise("storage.provider=s3");
return buildS3Provider(applicationProperties.getStorage().getS3());
}
if (!"local".equals(providerName)) {
throw new IllegalStateException("Storage provider not supported: " + providerName);
}
@@ -71,4 +80,9 @@ public class StorageProviderConfig {
}
return new LocalStorageProvider(basePath);
}
private S3StorageProvider buildS3Provider(ApplicationProperties.Storage.S3 cfg) {
S3Clients.Bundle bundle = S3Clients.build(cfg, "storage provider");
return new S3StorageProvider(bundle.client(), bundle.presigner(), cfg.getBucket());
}
}
@@ -0,0 +1,91 @@
package stirling.software.proprietary.storage.controller;
import java.util.List;
import java.util.UUID;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PatchMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import jakarta.validation.Valid;
import jakarta.validation.constraints.NotNull;
import jakarta.validation.constraints.Size;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.RequiredArgsConstructor;
import stirling.software.proprietary.storage.service.FolderService;
/**
* Folder placement endpoints for existing stored files. Thin adapter: validates the request shape,
* delegates the transaction to {@link FolderService}, then maps the result onto the HTTP status.
* Authentication, storage-gate, ownership checks, and the bulk cap all live on the service (where
* {@code @Transactional} also lives) so the JDBC connection isn't held through JSON serialization.
*/
@RestController
@RequestMapping("/api/v1/storage/files")
@RequiredArgsConstructor
public class FileFolderPlacementController {
private static final int BULK_MOVE_MAX_FILES = 1000;
private final FolderService folderService;
/** Move a single file to a folder (or to root when folderId is null). */
@PatchMapping("/{fileId}/folder")
public ResponseEntity<Void> moveFileToFolder(
@PathVariable Long fileId, @Valid @RequestBody FolderPlacement body) {
folderService.moveFileToFolder(fileId, body.getFolderId());
return ResponseEntity.noContent().build();
}
/**
* Bulk move - fewer round-trips than calling the single endpoint N times. Returns 200 on full
* success, 207 (Multi-Status) when some files were skipped (typically because they don't belong
* to the caller).
*/
@PatchMapping("/folder")
public ResponseEntity<BulkMoveResponse> bulkMove(@Valid @RequestBody BulkMoveRequest body) {
FolderService.BulkMoveResult result =
folderService.bulkMoveFilesToFolder(body.getFolderId(), body.getFileIds());
HttpStatus status =
result.skippedFileIds().isEmpty() ? HttpStatus.OK : HttpStatus.MULTI_STATUS;
return ResponseEntity.status(status)
.body(new BulkMoveResponse(result.movedFileIds(), result.skippedFileIds()));
}
@Data
@NoArgsConstructor
@AllArgsConstructor
public static class FolderPlacement {
private UUID folderId;
}
@Data
@NoArgsConstructor
@AllArgsConstructor
public static class BulkMoveRequest {
private UUID folderId;
@NotNull
@Size(
min = 1,
max = BULK_MOVE_MAX_FILES,
message = "fileIds must contain between 1 and 1000 entries")
private List<Long> fileIds;
}
@Data
@NoArgsConstructor
@AllArgsConstructor
public static class BulkMoveResponse {
private List<Long> movedFileIds;
private List<Long> skippedFileIds;
}
}
@@ -1,7 +1,11 @@
package stirling.software.proprietary.storage.controller;
import java.io.IOException;
import java.net.URI;
import java.time.Duration;
import java.util.List;
import java.util.Locale;
import java.util.Optional;
import org.springframework.http.ContentDisposition;
import org.springframework.http.HttpHeaders;
@@ -25,6 +29,7 @@ import org.springframework.web.server.ResponseStatusException;
import io.swagger.v3.oas.annotations.tags.Tag;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.proprietary.security.model.User;
import stirling.software.proprietary.storage.model.FileShare;
@@ -35,17 +40,22 @@ import stirling.software.proprietary.storage.model.api.ShareLinkMetadataResponse
import stirling.software.proprietary.storage.model.api.ShareLinkResponse;
import stirling.software.proprietary.storage.model.api.ShareWithUserRequest;
import stirling.software.proprietary.storage.model.api.StoredFileResponse;
import stirling.software.proprietary.storage.provider.StorageProvider;
import stirling.software.proprietary.storage.service.FileStorageService;
@RestController
@RequestMapping("/api/v1/storage")
@RequiredArgsConstructor
@Slf4j
@Tag(
name = "File Storage",
description = "Stored file management, sharing, and share link operations")
public class FileStorageController {
private static final Duration SIGNED_URL_TTL = Duration.ofMinutes(5);
private final FileStorageService fileStorageService;
private final StorageProvider storageProvider;
@PostMapping(
value = "/files",
@@ -91,7 +101,9 @@ public class FileStorageController {
User user = fileStorageService.requireAuthenticatedUser();
StoredFile file = fileStorageService.getAccessibleFile(user, fileId);
fileStorageService.requireReadAccess(user, file);
return buildFileResponse(file, inline);
Optional<ResponseEntity<org.springframework.core.io.Resource>> redirect =
tryRedirectToSignedUrl(file, inline);
return redirect.orElseGet(() -> buildFileResponse(file, inline));
}
@DeleteMapping("/files/{fileId}")
@@ -189,7 +201,9 @@ public class FileStorageController {
fileStorageService.requireReadAccess(share);
fileStorageService.recordShareAccess(share, authentication, inline);
StoredFile file = share.getFile();
return buildFileResponse(file, inline);
Optional<ResponseEntity<org.springframework.core.io.Resource>> redirect =
tryRedirectToSignedUrl(file, inline);
return redirect.orElseGet(() -> buildFileResponse(file, inline));
}
@GetMapping("/share-links/{token}/metadata")
@@ -272,4 +286,34 @@ public class FileStorageController {
&& authentication.isAuthenticated()
&& !"anonymousUser".equals(authentication.getPrincipal());
}
private Optional<ResponseEntity<org.springframework.core.io.Resource>> tryRedirectToSignedUrl(
StoredFile file, boolean inline) {
if (file == null || file.getStorageKey() == null || file.getStorageKey().isBlank()) {
return Optional.empty();
}
try {
Optional<URI> signed =
storageProvider.signedDownloadUrl(
file.getStorageKey(),
SIGNED_URL_TTL,
inline,
file.getOriginalFilename());
if (signed.isEmpty()) {
return Optional.empty();
}
HttpHeaders headers = new HttpHeaders();
headers.setLocation(signed.get());
ResponseEntity<org.springframework.core.io.Resource> response =
ResponseEntity.status(HttpStatus.FOUND).headers(headers).build();
return Optional.of(response);
} catch (IOException e) {
log.warn(
"Failed to create signed download URL for file {} (key: {}), falling back to streaming",
file.getId(),
file.getStorageKey(),
e);
return Optional.empty();
}
}
}
@@ -0,0 +1,70 @@
package stirling.software.proprietary.storage.controller;
import java.net.URI;
import java.util.List;
import java.util.UUID;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.DeleteMapping;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PatchMapping;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;
import jakarta.validation.Valid;
import lombok.RequiredArgsConstructor;
import stirling.software.proprietary.storage.model.api.CreateFolderRequest;
import stirling.software.proprietary.storage.model.api.FolderResponse;
import stirling.software.proprietary.storage.model.api.UpdateFolderRequest;
import stirling.software.proprietary.storage.service.FolderService;
/**
* REST endpoints for user-owned folders. Phase A - no folder-level sharing yet (Phase 3).
*
* <p>All operations are scoped to the authenticated user; existing single-file storage endpoints in
* {@link FileStorageController} are left alone so the cert-signing and standard upload flows are
* unaffected.
*/
@RestController
@RequestMapping("/api/v1/storage/folders")
@RequiredArgsConstructor
public class FolderController {
private final FolderService folderService;
@GetMapping
public List<FolderResponse> listFolders() {
return folderService.listFolders();
}
@PostMapping
public ResponseEntity<FolderResponse> createFolder(
@Valid @RequestBody CreateFolderRequest request) {
FolderResponse response = folderService.createFolder(request);
// 201 Created with Location header - conventional REST. The idempotent re-return path
// (same id resubmitted) also lands here; treating it as 201 keeps wire semantics simple.
return ResponseEntity.status(HttpStatus.CREATED)
.location(URI.create("/api/v1/storage/folders/" + response.id()))
.body(response);
}
@PatchMapping("/{folderId}")
public ResponseEntity<FolderResponse> updateFolder(
@PathVariable UUID folderId, @Valid @RequestBody UpdateFolderRequest request) {
return ResponseEntity.ok(folderService.updateFolder(folderId, request));
}
@DeleteMapping("/{folderId}")
public ResponseEntity<DeleteFolderResponse> deleteFolder(@PathVariable UUID folderId) {
List<UUID> removed = folderService.deleteFolder(folderId);
return ResponseEntity.ok(new DeleteFolderResponse(removed));
}
public record DeleteFolderResponse(List<UUID> removedFolderIds) {}
}
@@ -0,0 +1,104 @@
package stirling.software.proprietary.storage.model;
import java.io.Serializable;
import java.time.LocalDateTime;
import java.util.UUID;
import org.hibernate.annotations.CreationTimestamp;
import org.hibernate.annotations.OnDelete;
import org.hibernate.annotations.OnDeleteAction;
import org.hibernate.annotations.UpdateTimestamp;
import jakarta.persistence.Column;
import jakarta.persistence.Entity;
import jakarta.persistence.FetchType;
import jakarta.persistence.Id;
import jakarta.persistence.Index;
import jakarta.persistence.JoinColumn;
import jakarta.persistence.ManyToOne;
import jakarta.persistence.Table;
import jakarta.persistence.Version;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import stirling.software.proprietary.security.model.User;
/**
* A user-owned folder used by the file manager UI to organise stored files. Phase A entity - no
* folder-level sharing yet (Phase 3).
*
* <p>The id is a UUID rather than a numeric auto-increment so it round-trips with the
* client-generated {@code FolderId} and survives cross-device sync without re-keying.
*/
@Entity
@Table(
name = "folders",
indexes = {
@Index(name = "idx_folders_owner", columnList = "owner_id"),
@Index(name = "idx_folders_parent", columnList = "parent_folder_id"),
@Index(name = "idx_folders_owner_parent", columnList = "owner_id, parent_folder_id")
})
@NoArgsConstructor
@Getter
@Setter
public class Folder implements Serializable {
private static final long serialVersionUID = 1L;
/**
* Dialect-portable UUID column. The previous {@code columnDefinition = "uuid"} was
* Postgres-specific and broke on H2/MariaDB. Hibernate's {@code UUID} mapping picks the right
* native type per dialect (BINARY(16) on H2/MariaDB, uuid on Postgres) when no explicit
* columnDefinition is set.
*/
@Id
@Column(name = "folder_id", nullable = false)
private UUID id;
/**
* {@code OnDeleteAction.CASCADE} so deleting the owning {@code User} cascades to this row at
* the DB level - UserService.deleteUserRelatedData doesn't enumerate folders today, and leaving
* the FK without an action throws a constraint violation on user delete.
*/
@ManyToOne(fetch = FetchType.LAZY)
@JoinColumn(name = "owner_id", nullable = false)
@OnDelete(action = OnDeleteAction.CASCADE)
private User owner;
/**
* Parent folder; null = root. {@code OnDeleteAction.CASCADE} so a backend-side parent delete
* cleans children automatically, matching the service-layer recursive-delete contract.
*/
@ManyToOne(fetch = FetchType.LAZY)
@JoinColumn(name = "parent_folder_id")
@OnDelete(action = OnDeleteAction.CASCADE)
private Folder parent;
@Column(name = "name", nullable = false, length = 255)
private String name;
@Column(name = "color", length = 32)
private String color;
@Column(name = "icon", length = 64)
private String icon;
/**
* Optimistic-locking version. Cross-PC sync without this lets last-write-win silently. The
* column is nullable so existing rows from a pre-version deployment can be backfilled by
* Hibernate's update-on-write rather than failing the ddl-auto upgrade.
*/
@Version
@Column(name = "version")
private Long version;
@CreationTimestamp
@Column(name = "created_at", updatable = false)
private LocalDateTime createdAt;
@UpdateTimestamp
@Column(name = "updated_at")
private LocalDateTime updatedAt;
}
@@ -6,6 +6,8 @@ import java.util.HashSet;
import java.util.Set;
import org.hibernate.annotations.CreationTimestamp;
import org.hibernate.annotations.OnDelete;
import org.hibernate.annotations.OnDeleteAction;
import org.hibernate.annotations.UpdateTimestamp;
import jakarta.persistence.CascadeType;
@@ -35,7 +37,8 @@ import stirling.software.proprietary.workflow.model.WorkflowSession;
name = "stored_files",
indexes = {
@Index(name = "idx_stored_files_owner", columnList = "owner_id"),
@Index(name = "idx_stored_files_workflow", columnList = "workflow_session_id")
@Index(name = "idx_stored_files_workflow", columnList = "workflow_session_id"),
@Index(name = "idx_stored_files_folder", columnList = "folder_id")
})
@NoArgsConstructor
@Getter
@@ -106,6 +109,20 @@ public class StoredFile implements Serializable {
orphanRemoval = true)
private Set<FileShare> shares = new HashSet<>();
/**
* Optional folder placement for the file manager UI. Null = root. Hibernate ddl-auto will add
* this as a nullable column on upgrade so existing records continue to work untouched.
*
* <p>{@code OnDeleteAction.SET_NULL} so any backend that drops a folder row (admin script,
* future cleanup job, cascading user delete) cleanly orphans files to root rather than leaving
* dangling FK references. The application path ({@code FolderRepository.clearFolderForFiles})
* still runs first as a belt-and-braces.
*/
@ManyToOne(fetch = FetchType.LAZY)
@JoinColumn(name = "folder_id")
@OnDelete(action = OnDeleteAction.SET_NULL)
private Folder folder;
@CreationTimestamp
@Column(name = "created_at", updatable = false)
private LocalDateTime createdAt;
@@ -0,0 +1,43 @@
package stirling.software.proprietary.storage.model.api;
import java.util.UUID;
import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.Pattern;
import jakarta.validation.constraints.Size;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
@Data
@NoArgsConstructor
@AllArgsConstructor
public class CreateFolderRequest {
/**
* Client-generated UUID - lets the caller round-trip the same id it stored locally. Optional;
* the server generates one when missing.
*/
private UUID id;
@NotBlank
@Size(max = 255)
private String name;
private UUID parentFolderId;
/** Hex colour string (#rrggbb or #rrggbbaa) - matches the frontend palette format. */
@Size(max = 32)
@Pattern(
regexp = "^#[0-9a-fA-F]{6}([0-9a-fA-F]{2})?$",
message = "color must be a #RRGGBB or #RRGGBBAA hex value")
private String color;
/** Icon identifier - lowercase alphanumerics, hyphens, underscores only. */
@Size(max = 64)
@Pattern(
regexp = "^[a-z0-9_-]+$",
message = "icon must be a lowercase id (a-z, 0-9, '-' or '_')")
private String icon;
}
@@ -0,0 +1,38 @@
package stirling.software.proprietary.storage.model.api;
import java.time.LocalDateTime;
import java.util.UUID;
import stirling.software.proprietary.storage.model.Folder;
/**
* Outbound DTO for folder responses. Records are immutable, value-equality-based, and far less
* accident-prone than a {@code @Data} class with public setters.
*/
public record FolderResponse(
UUID id,
String name,
UUID parentFolderId,
String color,
String icon,
Long version,
LocalDateTime createdAt,
LocalDateTime updatedAt) {
public static FolderResponse from(Folder folder) {
// {@code folder.getParent().getId()} on a lazy proxy returns the FK value cached at the
// join column WITHOUT initialising the proxy under standard Hibernate, so this does
// not N+1. If a future Hibernate update changes that, switch the JPQL list query to a
// constructor projection.
UUID parentId = folder.getParent() == null ? null : folder.getParent().getId();
return new FolderResponse(
folder.getId(),
folder.getName(),
parentId,
folder.getColor(),
folder.getIcon(),
folder.getVersion(),
folder.getCreatedAt(),
folder.getUpdatedAt());
}
}
@@ -2,6 +2,7 @@ package stirling.software.proprietary.storage.model.api;
import java.time.LocalDateTime;
import java.util.List;
import java.util.UUID;
import lombok.Builder;
import lombok.Getter;
@@ -22,4 +23,10 @@ public class StoredFileResponse {
private final List<SharedUserResponse> sharedUsers;
private final List<ShareLinkResponse> shareLinks;
private final String filePurpose;
/**
* Optional folder placement (Phase A). Null when the file lives at the root or when the server
* build doesn't have the folders feature enabled - existing clients should treat null as root.
*/
private final UUID folderId;
}
@@ -0,0 +1,58 @@
package stirling.software.proprietary.storage.model.api;
import java.util.UUID;
import jakarta.validation.constraints.Pattern;
import jakarta.validation.constraints.Size;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
/**
* PATCH-style update - every field is optional. Send only the fields you want to change.
*
* <p>The {@code reparent} flag distinguishes "do not change parent" from "move to root" since
* {@code parentFolderId == null} alone is ambiguous in a sparse body. We use a boxed {@link
* Boolean} so a missing field deserialises to {@code null} (= "do not reparent") rather than to
* primitive {@code false}, removing a class of "I PATCHed only the name but the server reset my
* parent" footguns.
*
* <p>When the trimmed name is empty (e.g. {@code " "}) the service rejects the request with HTTP
* 400 - silent drops are too easy to mistake for a successful rename.
*/
@Data
@NoArgsConstructor
@AllArgsConstructor
public class UpdateFolderRequest {
/** When provided, must contain at least one non-whitespace character. */
@Size(max = 255)
@Pattern(regexp = "\\S.*", message = "name must not be blank")
private String name;
private Boolean reparent;
private UUID parentFolderId;
@Size(max = 32)
@Pattern(
regexp = "^(|#[0-9a-fA-F]{6}([0-9a-fA-F]{2})?)$",
message = "color must be empty or a #RRGGBB / #RRGGBBAA hex value")
private String color;
@Size(max = 64)
@Pattern(
regexp = "^([a-z0-9_-]+)?$",
message = "icon must be a lowercase id (a-z, 0-9, '-' or '_') or empty")
private String icon;
/**
* Convenience accessor - treats null as "do not reparent". Named differently from the
* Lombok-generated {@code getReparent()} so callers don't accidentally use one for the other
* (the getter is nullable {@code Boolean}; this method collapses to primitive).
*/
@com.fasterxml.jackson.annotation.JsonIgnore
public boolean shouldReparent() {
return Boolean.TRUE.equals(reparent);
}
}
@@ -24,6 +24,9 @@ public class LocalStorageProvider implements StorageProvider {
@Override
public StoredObject store(User owner, MultipartFile file) throws IOException {
if (owner == null || owner.getId() == null) {
throw new IllegalArgumentException("owner.id is required for local storage key");
}
String originalFilename = sanitizeFilename(file.getOriginalFilename());
String storageKey =
owner.getId()
@@ -77,6 +80,7 @@ public class LocalStorageProvider implements StorageProvider {
if (filename == null || filename.isBlank()) {
return "file";
}
return Paths.get(filename).getFileName().toString();
String stripped = Paths.get(filename).getFileName().toString().replaceAll("\\p{Cntrl}", "");
return stripped.isBlank() ? "file" : stripped;
}
}
@@ -0,0 +1,190 @@
package stirling.software.proprietary.storage.provider;
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.net.URISyntaxException;
import java.nio.file.Paths;
import java.time.Duration;
import java.util.Optional;
import java.util.UUID;
import org.springframework.core.io.InputStreamResource;
import org.springframework.core.io.Resource;
import org.springframework.web.multipart.MultipartFile;
import lombok.extern.slf4j.Slf4j;
import stirling.software.proprietary.security.model.User;
import software.amazon.awssdk.core.ResponseInputStream;
import software.amazon.awssdk.core.exception.SdkException;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.DeleteObjectRequest;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.GetObjectResponse;
import software.amazon.awssdk.services.s3.model.NoSuchKeyException;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;
import software.amazon.awssdk.services.s3.presigner.model.PresignedGetObjectRequest;
/** {@link StorageProvider} backed by an S3-compatible object store. */
@Slf4j
public class S3StorageProvider implements StorageProvider, AutoCloseable {
private final S3Client s3Client;
private final S3Presigner s3Presigner;
private final String bucket;
public S3StorageProvider(S3Client s3Client, S3Presigner s3Presigner, String bucket) {
if (bucket == null || bucket.isBlank()) {
throw new IllegalArgumentException("S3 bucket must be configured");
}
this.s3Client = s3Client;
this.s3Presigner = s3Presigner;
this.bucket = bucket;
}
@Override
public StoredObject store(User owner, MultipartFile file) throws IOException {
if (owner == null || owner.getId() == null) {
throw new IllegalArgumentException("owner.id is required for S3 storage key");
}
String originalFilename = sanitizeFilename(file.getOriginalFilename());
// Key is opaque ({ownerId}/{uuid}) so non-ASCII filenames don't break vendors that
// restrict key charset (e.g. Supabase Storage returns 400 Invalid key on unicode).
// The display name is preserved in StoredObject.originalFilename and the DB row.
String storageKey = owner.getId() + "/" + UUID.randomUUID();
PutObjectRequest.Builder request =
PutObjectRequest.builder().bucket(bucket).key(storageKey);
if (file.getContentType() != null && !file.getContentType().isBlank()) {
request.contentType(file.getContentType());
}
try (InputStream inputStream = file.getInputStream()) {
s3Client.putObject(
request.build(), RequestBody.fromInputStream(inputStream, file.getSize()));
} catch (SdkException e) {
throw new IOException("Failed to upload object to S3", e);
}
return StoredObject.builder()
.storageKey(storageKey)
.originalFilename(originalFilename)
.contentType(file.getContentType())
.sizeBytes(file.getSize())
.build();
}
@Override
public Resource load(String storageKey) throws IOException {
GetObjectRequest request =
GetObjectRequest.builder().bucket(bucket).key(storageKey).build();
try {
ResponseInputStream<GetObjectResponse> stream = s3Client.getObject(request);
long contentLength =
stream.response().contentLength() != null
? stream.response().contentLength()
: -1;
return new InputStreamResource(stream) {
@Override
public long contentLength() {
return contentLength;
}
};
} catch (NoSuchKeyException e) {
throw new IOException("File not found", e);
} catch (SdkException e) {
throw new IOException("Failed to load object from S3", e);
}
}
@Override
public void delete(String storageKey) throws IOException {
try {
s3Client.deleteObject(
DeleteObjectRequest.builder().bucket(bucket).key(storageKey).build());
} catch (SdkException e) {
throw new IOException("Failed to delete object from S3", e);
}
}
@Override
public Optional<URI> signedDownloadUrl(String storageKey, Duration ttl) throws IOException {
return signedDownloadUrl(storageKey, ttl, false, null);
}
@Override
public Optional<URI> signedDownloadUrl(
String storageKey, Duration ttl, boolean inline, String originalFilename)
throws IOException {
if (storageKey == null || storageKey.isBlank()) {
return Optional.empty();
}
Duration effectiveTtl =
ttl == null || ttl.isZero() || ttl.isNegative() ? Duration.ofMinutes(5) : ttl;
try {
GetObjectRequest.Builder getBuilder =
GetObjectRequest.builder().bucket(bucket).key(storageKey);
String disposition = buildContentDisposition(inline, originalFilename);
if (disposition != null) {
getBuilder.responseContentDisposition(disposition);
}
GetObjectPresignRequest presignRequest =
GetObjectPresignRequest.builder()
.signatureDuration(effectiveTtl)
.getObjectRequest(getBuilder.build())
.build();
PresignedGetObjectRequest presigned = s3Presigner.presignGetObject(presignRequest);
return Optional.of(presigned.url().toURI());
} catch (SdkException | URISyntaxException e) {
log.warn("Failed to create presigned S3 GET URL for key {}", storageKey, e);
return Optional.empty();
}
}
// Returns null when originalFilename is blank; S3 falls back to its own default in that case.
static String buildContentDisposition(boolean inline, String originalFilename) {
if (originalFilename == null || originalFilename.isBlank()) {
return null;
}
// Strip CR/LF and other control chars before path parsing (Paths.get throws on them on
// Windows, and they defeat header parsers).
String stripped = originalFilename.replaceAll("\\p{Cntrl}", "");
// Use only the basename to avoid leaking directory structure into the header.
int lastSeparator = Math.max(stripped.lastIndexOf('/'), stripped.lastIndexOf('\\'));
if (lastSeparator >= 0) {
stripped = stripped.substring(lastSeparator + 1);
}
if (stripped.isBlank()) {
return null;
}
// Escape per RFC 6266 quoted-string rules.
String escaped = stripped.replace("\\", "\\\\").replace("\"", "\\\"");
return (inline ? "inline" : "attachment") + "; filename=\"" + escaped + "\"";
}
@Override
public void close() {
try {
s3Presigner.close();
} catch (Exception e) {
log.warn("Error closing S3 presigner", e);
}
try {
s3Client.close();
} catch (Exception e) {
log.warn("Error closing S3 client", e);
}
}
private String sanitizeFilename(String filename) {
if (filename == null || filename.isBlank()) {
return "file";
}
String stripped = Paths.get(filename).getFileName().toString().replaceAll("\\p{Cntrl}", "");
return stripped.isBlank() ? "file" : stripped;
}
}
@@ -1,16 +1,45 @@
package stirling.software.proprietary.storage.provider;
import java.io.IOException;
import java.net.URI;
import java.time.Duration;
import java.util.Optional;
import org.springframework.core.io.Resource;
import org.springframework.web.multipart.MultipartFile;
import stirling.software.proprietary.security.model.User;
public interface StorageProvider {
public interface StorageProvider extends AutoCloseable {
StoredObject store(User owner, MultipartFile file) throws IOException;
Resource load(String storageKey) throws IOException;
void delete(String storageKey) throws IOException;
/**
* Releases any backend-specific resources. Default no-op so {@link LocalStorageProvider} and
* {@link DatabaseStorageProvider} (which hold no closeable handles) satisfy Spring's
* {@code @Bean(destroyMethod = "close")} signature requirement without ceremony. {@code
* S3StorageProvider} overrides this to close the underlying SDK client + presigner.
*/
@Override
default void close() {}
/**
* Returns a presigned download URL valid for {@code ttl}, or {@link Optional#empty()} if the
* provider does not support signed URLs (callers fall back to {@link #load(String)}).
*/
default Optional<URI> signedDownloadUrl(String storageKey, Duration ttl) throws IOException {
return signedDownloadUrl(storageKey, ttl, false, null);
}
/**
* Like {@link #signedDownloadUrl(String, Duration)} with explicit Content-Disposition control.
*/
default Optional<URI> signedDownloadUrl(
String storageKey, Duration ttl, boolean inline, String originalFilename)
throws IOException {
return Optional.empty();
}
}
@@ -0,0 +1,35 @@
package stirling.software.proprietary.storage.repository;
import java.util.List;
import java.util.Optional;
import java.util.UUID;
import org.springframework.data.jpa.repository.JpaRepository;
import org.springframework.data.jpa.repository.Modifying;
import org.springframework.data.jpa.repository.Query;
import org.springframework.data.repository.query.Param;
import stirling.software.proprietary.security.model.User;
import stirling.software.proprietary.storage.model.Folder;
public interface FolderRepository extends JpaRepository<Folder, UUID> {
Optional<Folder> findByIdAndOwner(UUID id, User owner);
List<Folder> findAllByOwnerOrderByName(User owner);
long countByOwner(User owner);
/**
* Clear the folder reference on every file currently inside any of the given folders. Used when
* a folder subtree is deleted - files fall back to the root rather than dangling.
*
* <p>{@code flushAutomatically + clearAutomatically} forces Hibernate to flush any cached dirty
* {@code StoredFile} entities before the bulk UPDATE runs, and clears the persistence context
* afterwards so a subsequent {@code deleteAllByIdInBatch} on the parent folders doesn't see
* stale entity state referencing the about-to-be-deleted folder.
*/
@Modifying(flushAutomatically = true, clearAutomatically = true)
@Query("UPDATE StoredFile sf SET sf.folder = null WHERE sf.folder.id IN :folderIds")
void clearFolderForFiles(@Param("folderIds") List<UUID> folderIds);
}
@@ -59,6 +59,13 @@ public interface StoredFileRepository extends JpaRepository<StoredFile, Long> {
List<StoredFile> findAllByOwner(User owner);
/**
* Bulk lookup used by the folder-placement controller. Returns only files owned by {@code
* owner}; ids that don't exist or that belong to another user are silently dropped so the
* caller can compute the "skipped" set by subtraction.
*/
List<StoredFile> findAllByIdInAndOwner(List<Long> ids, User owner);
@Modifying
@Transactional
@Query(
@@ -459,6 +459,7 @@ public class FileStorageService {
file.getPurpose() != null
? file.getPurpose().name().toLowerCase(Locale.ROOT)
: null)
.folderId(file.getFolder() != null ? file.getFolder().getId() : null)
.build();
}
@@ -0,0 +1,417 @@
package stirling.software.proprietary.storage.service;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Deque;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.UUID;
import org.springframework.dao.DataIntegrityViolationException;
import org.springframework.http.HttpStatus;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.web.server.ResponseStatusException;
import lombok.RequiredArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.proprietary.security.model.User;
import stirling.software.proprietary.storage.model.Folder;
import stirling.software.proprietary.storage.model.StoredFile;
import stirling.software.proprietary.storage.model.api.CreateFolderRequest;
import stirling.software.proprietary.storage.model.api.FolderResponse;
import stirling.software.proprietary.storage.model.api.UpdateFolderRequest;
import stirling.software.proprietary.storage.repository.FolderRepository;
import stirling.software.proprietary.storage.repository.StoredFileRepository;
/**
* Phase A folder operations. Each call is scoped to the authenticated user - folders are private to
* their owner. Folder-level sharing is a Phase 3 feature.
*/
@Service
@RequiredArgsConstructor
@Slf4j
public class FolderService {
/**
* Hard cap on folders per user. Beyond this {@link #createFolder} rejects with 409 - guards
* against per-account folder-explosion DoS and bounds the in-memory subtree walk in {@link
* #deleteFolder}.
*/
private static final long MAX_FOLDERS_PER_USER = 5_000L;
/**
* Hard cap on chain depth from the root to any folder. Bounds the lazy-proxy walk in {@link
* #enforceDepthAndCycle} - otherwise a user could build a chain up to MAX_FOLDERS_PER_USER deep
* and force one Hibernate SELECT per ancestor on every reparent (5,000+ SELECTs == seconds of
* DB time per request, per-account weaponizable as DoS).
*/
private static final int MAX_FOLDER_DEPTH = 64;
/**
* Hard cap on bulk-move payload size, mirroring the request-validation cap on {@code
* FileFolderPlacementController.BulkMoveRequest.fileIds}. Re-asserted at the service layer
* because controller-level @Valid bounds aren't enforced when the service is called directly
* (e.g. by future internal callers or tests).
*/
private static final int BULK_MOVE_MAX_FILES = 1000;
private final FolderRepository folderRepository;
private final StoredFileRepository storedFileRepository;
private final ApplicationProperties applicationProperties;
/**
* Gate every public method on storage being enabled, mirroring {@code
* FileStorageService.ensureStorageEnabled}. Without this, folder CRUD still works when {@code
* storage.enabled=false} or {@code security.enableLogin=false}, defeating the operator's intent
* to disable storage end-to-end.
*/
private void ensureStorageEnabled() {
if (!applicationProperties.getSecurity().isEnableLogin()) {
throw new ResponseStatusException(
HttpStatus.FORBIDDEN, "Storage requires login to be enabled");
}
if (!applicationProperties.getStorage().isEnabled()) {
throw new ResponseStatusException(HttpStatus.FORBIDDEN, "Storage is disabled");
}
}
/** List every folder owned by the current user, alphabetical. */
@Transactional(readOnly = true)
public List<FolderResponse> listFolders() {
ensureStorageEnabled();
User user = requireAuthenticatedUser();
return folderRepository.findAllByOwnerOrderByName(user).stream()
.map(FolderResponse::from)
.toList();
}
@Transactional
public FolderResponse createFolder(CreateFolderRequest request) {
ensureStorageEnabled();
User user = requireAuthenticatedUser();
// Reject self-parenting up-front. Without this, a client posting
// {id: X, parentFolderId: X} for a folder X they already own would silently
// get the existing folder back (idempotent path) and never learn that the
// parentFolderId they sent was ignored. For new ids the parent lookup would
// 404, but the message is misleading.
if (request.getId() != null && request.getId().equals(request.getParentFolderId())) {
throw new ResponseStatusException(
HttpStatus.BAD_REQUEST, "A folder cannot be its own parent");
}
Folder parent = resolveParent(request.getParentFolderId(), user, null);
UUID id = request.getId() != null ? request.getId() : UUID.randomUUID();
// Idempotent: if this user already owns a folder with the supplied id, return it
// unchanged. Single fetch (the previous code did findByIdAndOwner twice with a race
// window between the two lookups).
java.util.Optional<Folder> existing = folderRepository.findByIdAndOwner(id, user);
if (existing.isPresent()) {
return FolderResponse.from(existing.get());
}
// The id is a global primary key. If the id exists for a *different* user, surfacing 500
// with a constraint-violation stack trace leaks far too much; convert to 409 Conflict so
// the caller can pick a fresh id.
if (folderRepository.existsById(id)) {
throw new ResponseStatusException(
HttpStatus.CONFLICT,
"A folder with this id already exists; choose a different id");
}
if (folderRepository.countByOwner(user) >= MAX_FOLDERS_PER_USER) {
throw new ResponseStatusException(
HttpStatus.CONFLICT,
"Folder limit reached (max " + MAX_FOLDERS_PER_USER + " per user)");
}
Folder folder = new Folder();
folder.setId(id);
folder.setOwner(user);
folder.setParent(parent);
folder.setName(request.getName().trim());
folder.setColor(request.getColor());
folder.setIcon(request.getIcon());
// saveAndFlush forces the INSERT now so @CreationTimestamp populates
// createdAt/updatedAt before we build the response. Plain save defers
// the SQL until @Transactional commit, and the response would carry
// null timestamps that the frontend trust-boundary parser then rejects.
Folder saved = folderRepository.saveAndFlush(folder);
log.info(
"Folder created: user={} id={} parent={}",
user.getId(),
saved.getId(),
parent == null ? "root" : parent.getId());
return FolderResponse.from(saved);
}
@Transactional
public FolderResponse updateFolder(UUID id, UpdateFolderRequest request) {
ensureStorageEnabled();
User user = requireAuthenticatedUser();
Folder folder = requireOwnedFolder(id, user);
if (request.getName() != null) {
String trimmed = request.getName().trim();
if (trimmed.isEmpty()) {
// Bean validation should already catch this via @Pattern, but be explicit so
// an empty-after-trim payload reaches the user as a 400 instead of being
// silently dropped.
throw new ResponseStatusException(
HttpStatus.BAD_REQUEST, "Folder name cannot be blank");
}
folder.setName(trimmed);
}
if (request.shouldReparent()) {
Folder newParent = resolveParent(request.getParentFolderId(), user, folder.getId());
folder.setParent(newParent);
}
if (request.getColor() != null) {
folder.setColor(request.getColor().isEmpty() ? null : request.getColor());
}
if (request.getIcon() != null) {
folder.setIcon(request.getIcon().isEmpty() ? null : request.getIcon());
}
// saveAndFlush so @UpdateTimestamp populates updatedAt before the
// response is serialized (same reason as createFolder).
return FolderResponse.from(folderRepository.saveAndFlush(folder));
}
/**
* Recursive delete. Returns the ids of every folder that was removed so the caller can purge
* them from its local cache. Files inside those folders are detached (folder_id set to null) -
* never deleted.
*/
@Transactional
public List<UUID> deleteFolder(UUID id) {
ensureStorageEnabled();
User user = requireAuthenticatedUser();
Folder folder = requireOwnedFolder(id, user);
// Build the parent → children map once. Project to id-only via the
// existing entity list (Hibernate already has the column loaded -
// we only access f.getParent().getId() on a managed proxy, which
// does NOT initialize the proxy because Hibernate has the FK
// value cached at the join column).
Map<UUID, List<UUID>> childIdsByParent = new HashMap<>();
for (Folder f : folderRepository.findAllByOwnerOrderByName(user)) {
UUID parentId = f.getParent() == null ? null : f.getParent().getId();
childIdsByParent.computeIfAbsent(parentId, k -> new ArrayList<>()).add(f.getId());
}
// Iterative subtree collection - prior recursive form blew the JVM
// stack on deeply nested chains a malicious caller could create.
List<UUID> removed = new ArrayList<>();
Set<UUID> seen = new HashSet<>();
Deque<UUID> stack = new ArrayDeque<>();
stack.push(folder.getId());
while (!stack.isEmpty()) {
UUID cur = stack.pop();
if (!seen.add(cur)) continue;
removed.add(cur);
List<UUID> children = childIdsByParent.get(cur);
if (children != null) {
for (UUID childId : children) stack.push(childId);
}
}
if (!removed.isEmpty()) {
folderRepository.clearFolderForFiles(removed);
folderRepository.deleteAllByIdInBatch(removed);
log.info(
"Folder subtree deleted: user={} root={} count={}",
user.getId(),
folder.getId(),
removed.size());
}
return removed;
}
/**
* Move a single owned file to a folder (or root when {@code folderId} is null). Owns its
* own @Transactional rather than relying on the caller so the JDBC connection is released as
* soon as the writes commit, not held through controller-side JSON serialization.
*/
@Transactional
public void moveFileToFolder(Long fileId, UUID folderId) {
ensureStorageEnabled();
User user = requireAuthenticatedUser();
StoredFile file =
storedFileRepository
.findByIdAndOwner(fileId, user)
.orElseThrow(
() ->
new ResponseStatusException(
HttpStatus.NOT_FOUND,
"File not found or not owned by current user"));
file.setFolder(resolveOwnedFolder(folderId, user));
storedFileRepository.save(file);
}
/**
* Bulk move that returns the moved + skipped split. Skipped == file ids the caller doesn't own
* (or that no longer exist); the controller surfaces this as 207 Multi-Status.
*/
@Transactional
public BulkMoveResult bulkMoveFilesToFolder(UUID folderId, List<Long> fileIds) {
ensureStorageEnabled();
if (fileIds == null || fileIds.isEmpty()) {
return new BulkMoveResult(List.of(), List.of());
}
if (fileIds.size() > BULK_MOVE_MAX_FILES) {
throw new ResponseStatusException(
HttpStatus.BAD_REQUEST,
"fileIds must contain between 1 and " + BULK_MOVE_MAX_FILES + " entries");
}
User user = requireAuthenticatedUser();
Folder target = resolveOwnedFolder(folderId, user);
List<StoredFile> owned = storedFileRepository.findAllByIdInAndOwner(fileIds, user);
Set<Long> ownedIds = new HashSet<>(owned.size());
for (StoredFile f : owned) {
f.setFolder(target);
ownedIds.add(f.getId());
}
// If the target folder was deleted concurrently between resolveOwnedFolder and the
// flush, the FK constraint fires as DataIntegrityViolationException. Surface that as
// 409 Conflict so the caller sees an actionable error instead of a 500 stack.
try {
storedFileRepository.saveAll(owned);
storedFileRepository.flush();
} catch (DataIntegrityViolationException ex) {
throw new ResponseStatusException(
HttpStatus.CONFLICT,
"Target folder no longer exists; refresh and try again",
ex);
}
List<Long> moved = owned.stream().map(StoredFile::getId).toList();
List<Long> skipped = fileIds.stream().filter(id -> !ownedIds.contains(id)).toList();
if (!skipped.isEmpty()) {
log.warn(
"bulkMove: user {} skipped {} of {} files (not owned or missing)",
user.getId(),
skipped.size(),
fileIds.size());
}
return new BulkMoveResult(moved, skipped);
}
/** Result of {@link #bulkMoveFilesToFolder}. Records are immutable + auto-serializable. */
public record BulkMoveResult(List<Long> movedFileIds, List<Long> skippedFileIds) {}
// ─── helpers ────────────────────────────────────────────────────
/**
* Resolve a placement-target folder. Distinct from {@link #resolveParent} because move targets
* don't carry the parent-cycle semantics - we only need the folder to exist AND belong to the
* caller. Returns null for null input (root).
*/
private Folder resolveOwnedFolder(UUID folderId, User user) {
if (folderId == null) return null;
return folderRepository
.findByIdAndOwner(folderId, user)
.orElseThrow(
() ->
new ResponseStatusException(
HttpStatus.BAD_REQUEST,
"Folder does not exist or is not owned by you"));
}
private Folder requireOwnedFolder(UUID id, User user) {
return folderRepository
.findByIdAndOwner(id, user)
.orElseThrow(
() ->
new ResponseStatusException(
HttpStatus.NOT_FOUND,
"Folder not found or not owned by current user"));
}
private Folder resolveParent(UUID parentId, User user, UUID forbidId) {
if (parentId == null) return null;
if (forbidId != null && parentId.equals(forbidId)) {
throw new ResponseStatusException(
HttpStatus.BAD_REQUEST, "A folder cannot be its own parent");
}
Folder parent =
folderRepository
.findByIdAndOwner(parentId, user)
.orElseThrow(
() ->
new ResponseStatusException(
HttpStatus.BAD_REQUEST,
"Parent folder does not exist or is not owned by you"));
// Reject before the child is created/moved if attaching it would push the chain past the
// depth cap. Done in one pass that also returns the cycle answer so we don't walk the
// lazy-proxy chain twice.
enforceDepthAndCycle(parent, user, forbidId);
return parent;
}
/**
* Single pass that walks the parent chain to root and (a) rejects if attaching a child here
* would exceed MAX_FOLDER_DEPTH, (b) rejects if {@code forbidId} appears in the chain (cycle on
* reparent), (c) rejects on a broken graph, and (d) rejects if any ancestor is owned by a
* different user (defense-in-depth: callers always pass a parent already ownership-checked, but
* the parent chain is followed via lazy proxy without re-checking ownership at each hop, so any
* stray cross-owner edge in the database would otherwise leak ancestor folder ids through the
* cycle error message). The walk is hard-bounded at MAX_FOLDER_DEPTH so a corrupted database
* (chain longer than the API would allow) can never produce an unbounded SELECT loop.
*/
private void enforceDepthAndCycle(Folder candidateParent, User user, UUID forbidId) {
Folder cursor = candidateParent;
Set<UUID> seen = new HashSet<>();
int depth = 0;
while (cursor != null) {
if (cursor.getOwner() == null || !cursor.getOwner().getId().equals(user.getId())) {
throw new ResponseStatusException(
HttpStatus.BAD_REQUEST, "Folder hierarchy is corrupted; contact support");
}
if (forbidId != null && cursor.getId().equals(forbidId)) {
throw new ResponseStatusException(
HttpStatus.BAD_REQUEST,
"Cannot move a folder inside one of its descendants");
}
if (!seen.add(cursor.getId())) {
// broken graph (cycle in stored data)
throw new ResponseStatusException(
HttpStatus.BAD_REQUEST, "Folder hierarchy is corrupted; contact support");
}
depth += 1;
// candidateParent is at depth 1 from the new child's perspective. After the walk,
// `depth` equals the number of ancestors including candidateParent, which is the
// depth at which the new child would live. Reject before exceeding the cap.
if (depth >= MAX_FOLDER_DEPTH) {
throw new ResponseStatusException(
HttpStatus.BAD_REQUEST,
"Folder nesting limit reached (max " + MAX_FOLDER_DEPTH + " levels)");
}
cursor = cursor.getParent();
}
}
private User requireAuthenticatedUser() {
Authentication authentication = SecurityContextHolder.getContext().getAuthentication();
if (authentication == null
|| !authentication.isAuthenticated()
|| !(authentication.getPrincipal() instanceof User user)) {
throw new ResponseStatusException(HttpStatus.UNAUTHORIZED, "Authentication required");
}
return user;
}
}
@@ -1,71 +0,0 @@
package stirling.software.proprietary.cluster;
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.lang.reflect.Field;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import org.junit.jupiter.api.Test;
/**
* Runtime license gate contract. Verifies cluster mode is gated by the existing {@code
* runningProOrHigher} bean and reports a clear error when the license is missing.
*
* <p>The gate uses reflection-friendly field injection (one optional bean) so the test wires it
* directly without bringing up a full Spring context.
*/
class ClusterLicenseGateTest {
private void injectRunningProOrHigher(ClusterLicenseGate gate, Boolean value) throws Exception {
Field f = ClusterLicenseGate.class.getDeclaredField("runningProOrHigher");
f.setAccessible(true);
f.set(gate, value);
}
private void invokeVerify(ClusterLicenseGate gate) throws Throwable {
Method m = ClusterLicenseGate.class.getDeclaredMethod("verifyLicense");
m.setAccessible(true);
try {
m.invoke(gate);
} catch (InvocationTargetException e) {
throw e.getCause();
}
}
@Test
void serverOrEnterpriseLicense_allowsClusterMode() throws Throwable {
ClusterLicenseGate gate = new ClusterLicenseGate();
injectRunningProOrHigher(gate, Boolean.TRUE);
assertDoesNotThrow(() -> invokeVerify(gate));
}
@Test
void normalLicense_refusesClusterMode_withActionableMessage() throws Exception {
ClusterLicenseGate gate = new ClusterLicenseGate();
injectRunningProOrHigher(gate, Boolean.FALSE);
IllegalStateException ex =
assertThrows(IllegalStateException.class, () -> invokeVerify(gate));
String msg = ex.getMessage();
// The error message must tell the operator exactly what to do.
assertTrue(msg.contains("SERVER"), "message must mention SERVER license tier: " + msg);
assertTrue(msg.contains("ENTERPRISE"), "message must mention ENTERPRISE tier: " + msg);
assertTrue(
msg.contains("stirling.premium.key") || msg.contains("license key"),
"message must explain how to set the license: " + msg);
assertTrue(
msg.contains("cluster.enabled=false"),
"message must offer the opt-out (disable cluster): " + msg);
}
@Test
void saasFlavor_bypassesGate_whenRunningProOrHigherBeanAbsent() throws Throwable {
// In saas builds the runningProOrHigher bean is @Profile("security & !saas") so absent.
// The gate's @Autowired(required=false) leaves the field null. Must not throw.
ClusterLicenseGate gate = new ClusterLicenseGate();
// field stays null (default)
assertDoesNotThrow(() -> invokeVerify(gate));
}
}
@@ -1,132 +0,0 @@
package stirling.software.proprietary.cluster;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import io.micrometer.core.instrument.Gauge;
import io.micrometer.core.instrument.simple.SimpleMeterRegistry;
import stirling.software.common.model.ApplicationProperties;
/**
* Contract test for cluster metrics. Asserts every named metric is registered and the recorder
* methods write to them, so dashboards do not silently lose a metric to a rename.
*/
class ClusterMetricsTest {
private SimpleMeterRegistry registry;
private ClusterMetrics metrics;
private static final String NODE = "test-node";
@BeforeEach
void setUp() {
registry = new SimpleMeterRegistry();
ApplicationProperties props = new ApplicationProperties();
props.getCluster().getNode().setId(NODE);
metrics = new ClusterMetrics(registry, props);
}
@Test
void registersAllRequiredMeters() {
assertNotNull(registry.find("stirling_cluster_sticky_miss_total").counter());
assertNotNull(registry.find("stirling_cluster_ratelimit_rejected_total").counter());
assertNotNull(registry.find("stirling_cluster_backplane_latency_seconds").timer());
assertNotNull(registry.find("stirling_cluster_job_wait_seconds").timer());
Gauge inflight = registry.find("stirling_cluster_jobs_inflight").tag("node", NODE).gauge();
assertNotNull(inflight, "jobs_inflight gauge with node tag must be registered eagerly");
}
@Test
void registersKnownLaneGaugesEagerly() {
// Lanes (FAST, SLOW, AI) are a fixed enum, so all three gauges must exist at construction
// - dashboards must never have a missing series for a known lane.
for (String lane : new String[] {"FAST", "SLOW", "AI"}) {
Gauge g = registry.find("stirling_cluster_queue_depth").tag("lane", lane).gauge();
assertNotNull(g, "lane gauge must be eagerly registered for " + lane);
assertEquals(0.0, g.value(), "lane gauge default value must be 0 for " + lane);
}
assertEquals(
3,
registry.find("stirling_cluster_queue_depth").gauges().size(),
"exactly the three known lane gauges should be registered at boot");
}
@Test
void recordStickyMissIncrementsCounter() {
metrics.recordStickyMiss();
metrics.recordStickyMiss();
assertEquals(2.0, registry.find("stirling_cluster_sticky_miss_total").counter().count());
}
@Test
void recordRateLimitRejectIncrementsCounter() {
metrics.recordRateLimitReject();
assertEquals(
1.0, registry.find("stirling_cluster_ratelimit_rejected_total").counter().count());
}
@Test
void incrementAndDecrementInflightUpdatesGauge() {
metrics.incrementInflight();
metrics.incrementInflight();
metrics.incrementInflight();
metrics.decrementInflight();
Gauge gauge = registry.find("stirling_cluster_jobs_inflight").tag("node", NODE).gauge();
assertEquals(2.0, gauge.value(), "expected 2 inflight after 3 inc / 1 dec");
}
@Test
void setQueueDepthUpdatesEagerlyRegisteredLaneGauge() {
// The three known-lane gauges (FAST, SLOW, AI) are registered eagerly at construction
// (see registersKnownLaneGaugesEagerly); setQueueDepth only updates the holder value.
metrics.setQueueDepth("FAST", 4);
metrics.setQueueDepth("SLOW", 7);
Gauge fast = registry.find("stirling_cluster_queue_depth").tag("lane", "FAST").gauge();
Gauge slow = registry.find("stirling_cluster_queue_depth").tag("lane", "SLOW").gauge();
assertEquals(4.0, fast.value());
assertEquals(7.0, slow.value());
}
@Test
void setQueueDepthForUnknownLane_lazyRegistersFallbackGauge() {
// Defensive: if a caller passes an unrecognised lane, we still register so we don't lose
// the signal. This is a fallback, not the supported path.
metrics.setQueueDepth("custom-lane", 5);
Gauge g = registry.find("stirling_cluster_queue_depth").tag("lane", "custom-lane").gauge();
assertNotNull(g);
assertEquals(5.0, g.value());
}
@Test
void setQueueDepthIsIdempotentAcrossCalls() {
metrics.setQueueDepth("FAST", 1);
metrics.setQueueDepth("FAST", 2);
metrics.setQueueDepth("FAST", 9);
// Only one gauge per lane, not three.
assertEquals(
1,
registry.find("stirling_cluster_queue_depth").tag("lane", "FAST").gauges().size());
assertEquals(
9.0,
registry.find("stirling_cluster_queue_depth").tag("lane", "FAST").gauge().value());
}
@Test
void backplaneLatencyTimerAcceptsRecordings() {
metrics.backplaneLatency().record(java.time.Duration.ofMillis(7));
metrics.backplaneLatency().record(java.time.Duration.ofMillis(11));
assertEquals(
2L, registry.find("stirling_cluster_backplane_latency_seconds").timer().count());
}
@Test
void jobWaitTimerAcceptsRecordings() {
metrics.jobWaitSeconds().record(java.time.Duration.ofMillis(50));
assertEquals(1L, registry.find("stirling_cluster_job_wait_seconds").timer().count());
}
}
@@ -1,126 +0,0 @@
package stirling.software.proprietary.cluster;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.ArgumentMatchers.any;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.never;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import java.time.Duration;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.mockito.ArgumentCaptor;
import org.springframework.test.util.ReflectionTestUtils;
import stirling.software.common.cluster.ClusterNode;
import stirling.software.common.cluster.InstanceRegistry;
import stirling.software.common.model.ApplicationProperties;
/** Verifies the bootstrap registers / heartbeats / deregisters as expected. */
class ClusterNodeBootstrapTest {
private InstanceRegistry registry;
private ApplicationProperties props;
private ClusterNodeBootstrap bootstrap;
@BeforeEach
void setUp() {
registry = mock(InstanceRegistry.class);
props = new ApplicationProperties();
props.getCluster().setEnabled(true);
props.getCluster().getNode().setId("node-test-1");
props.getCluster().getNode().setRole("worker");
// Pin heartbeat to 10s so TTL math is stable across PR2 default changes (TTL = 3x = 30s).
props.getCluster().getNode().setHeartbeatIntervalMs(10_000L);
bootstrap = new ClusterNodeBootstrap(props, registry);
ReflectionTestUtils.setField(bootstrap, "serverPort", 8080);
}
@Test
void registerOnStartupCallsRegistryWithResolvedNodeId() {
bootstrap.registerOnStartup();
ArgumentCaptor<ClusterNode> nodeCaptor = ArgumentCaptor.forClass(ClusterNode.class);
ArgumentCaptor<Duration> ttlCaptor = ArgumentCaptor.forClass(Duration.class);
verify(registry, times(1)).register(nodeCaptor.capture(), ttlCaptor.capture());
ClusterNode captured = nodeCaptor.getValue();
assertEquals("node-test-1", captured.nodeId());
assertTrue(captured.internalAddress().startsWith("http://"));
assertTrue(captured.internalAddress().endsWith(":8080"));
assertEquals("WORKER", captured.role());
assertEquals(30L, ttlCaptor.getValue().toSeconds());
}
@Test
void registerHonoursExplicitInternalAddress() {
props.getCluster().getNode().setInternalAddress("app-1:8080");
bootstrap.registerOnStartup();
ArgumentCaptor<ClusterNode> nodeCaptor = ArgumentCaptor.forClass(ClusterNode.class);
verify(registry).register(nodeCaptor.capture(), any());
assertEquals("http://app-1:8080", nodeCaptor.getValue().internalAddress());
}
@Test
void registerUsesHttpsSchemeWhenConfigured() {
// SE3: nodes that terminate TLS themselves need https:// in the registry so peers can reach
// them. The default (http) is correct for the common LB-terminates-TLS topology.
props.getCluster().getNode().setInternalAddress("app-1:8443");
props.getCluster().getNode().setScheme("https");
ClusterNodeBootstrap httpsBootstrap = new ClusterNodeBootstrap(props, registry);
ReflectionTestUtils.setField(httpsBootstrap, "serverPort", 8443);
httpsBootstrap.registerOnStartup();
ArgumentCaptor<ClusterNode> nodeCaptor = ArgumentCaptor.forClass(ClusterNode.class);
verify(registry).register(nodeCaptor.capture(), any());
assertEquals("https://app-1:8443", nodeCaptor.getValue().internalAddress());
}
@Test
void heartbeatAfterStartup_callsRegister_forSelfHealing() {
// Heartbeat re-invokes register() (idempotent) so a wiped backplane re-populates
// every field, not just lastHeartbeat. Expect 2 register() calls: startup + heartbeat.
bootstrap.start();
bootstrap.registerOnStartup();
bootstrap.heartbeat();
verify(registry, times(2))
.register(
any(ClusterNode.class),
org.mockito.ArgumentMatchers.eq(Duration.ofSeconds(30)));
}
@Test
void smartLifecycleStop_deregisters() {
bootstrap.start();
bootstrap.registerOnStartup();
bootstrap.stop();
verify(registry, times(1)).deregister("node-test-1");
}
@Test
void smartLifecycleStop_beforeStartup_isNoop() {
bootstrap.stop();
verify(registry, never()).deregister(any());
}
@Test
void heartbeatAfterStop_doesNotReRegister() {
// Heartbeat-after-stop race: SmartLifecycle.stop() deregisters, but the @Scheduled
// tick keeps firing during a slow drain. Without a guard, the next tick would
// re-register the dead node and the entry would resurface in the registry until TTL
// expiry. Rolling deploys with slow shutdown = draining nodes keep re-announcing
// themselves indefinitely.
bootstrap.start();
bootstrap.registerOnStartup();
// 1 register from startup.
verify(registry, times(1)).register(any(ClusterNode.class), any(Duration.class));
bootstrap.stop();
verify(registry, times(1)).deregister("node-test-1");
// Critical: next scheduled tick after stop must NOT re-register.
bootstrap.heartbeat();
// Still exactly 1 register call (the startup one); no second register from heartbeat.
verify(registry, times(1)).register(any(ClusterNode.class), any(Duration.class));
}
}
@@ -1,110 +0,0 @@
package stirling.software.proprietary.cluster;
import static org.assertj.core.api.Assertions.assertThat;
import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.runner.ApplicationContextRunner;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.serializer.GenericJackson2JsonRedisSerializer;
import org.springframework.data.redis.serializer.RedisSerializer;
import org.springframework.mock.web.MockHttpServletRequest;
import org.springframework.mock.web.MockHttpServletResponse;
import org.springframework.session.web.http.CookieSerializer;
import org.springframework.session.web.http.DefaultCookieSerializer;
import jakarta.servlet.http.Cookie;
/**
* Security regression test for {@link ClusterSessionConfiguration}.
*
* <p>The bean name {@code springSessionDefaultRedisSerializer} is the exact override hook Spring
* Session inspects. If it is absent, Spring Session falls back to JDK serialization on session read
* / write, which is a deserialization-gadget RCE surface for anyone with Valkey write access.
* Asserting both the bean's existence and its concrete type guards against accidental removal
* during future refactors.
*/
class ClusterSessionConfigurationTest {
@Configuration
static class StubConnectionFactoryConfig {
@Bean
LettuceConnectionFactory lettuceConnectionFactory() {
// Stub - satisfies @ConditionalOnBean without opening a real connection.
return new LettuceConnectionFactory();
}
}
private final ApplicationContextRunner runner =
new ApplicationContextRunner()
.withUserConfiguration(
StubConnectionFactoryConfig.class, ClusterSessionConfiguration.class);
@Test
void clusterDisabled_configurationIsInert_noSerializerBean() {
runner.run(
context ->
assertThat(context)
.hasNotFailed()
.doesNotHaveBean("springSessionDefaultRedisSerializer"));
}
@Test
@SuppressWarnings(
"removal") // see ClusterSessionConfiguration#springSessionDefaultRedisSerializer
void clusterEnabled_withLettuceFactory_wiresJsonSerializerUnderExpectedBeanName() {
runner.withPropertyValues("cluster.enabled=true")
.run(
context -> {
assertThat(context).hasNotFailed();
assertThat(context.containsBean("springSessionDefaultRedisSerializer"))
.as("Spring Session looks up this exact bean name")
.isTrue();
RedisSerializer<?> serializer =
context.getBean(
"springSessionDefaultRedisSerializer",
RedisSerializer.class);
assertThat(serializer)
.as(
"must be JSON serializer; JDK serialization is a"
+ " deserialization-gadget RCE surface")
.isInstanceOf(GenericJackson2JsonRedisSerializer.class);
});
}
@Test
void clusterEnabled_sessionCookie_isSecureHttpOnlyAndSameSiteLax() {
runner.withPropertyValues("cluster.enabled=true")
.run(
context -> {
assertThat(context).hasNotFailed();
assertThat(context.containsBean("cookieSerializer")).isTrue();
CookieSerializer serializer =
context.getBean("cookieSerializer", CookieSerializer.class);
assertThat(serializer)
.as("must be the Spring Session DefaultCookieSerializer")
.isInstanceOf(DefaultCookieSerializer.class);
MockHttpServletRequest request = new MockHttpServletRequest();
request.setSecure(true);
MockHttpServletResponse response = new MockHttpServletResponse();
serializer.writeCookieValue(
new CookieSerializer.CookieValue(
request, response, "sess-value"));
Cookie cookie = response.getCookie("SESSION");
assertThat(cookie).as("SESSION cookie must be written").isNotNull();
assertThat(cookie.getSecure())
.as("Secure flag MUST be set for HTTPS deployments")
.isTrue();
assertThat(cookie.isHttpOnly())
.as("HttpOnly MUST be set to block JS access")
.isTrue();
// SameSite is not a Cookie API field; check the raw Set-Cookie header.
assertThat(response.getHeader("Set-Cookie"))
.as("SameSite=Lax MUST be present to mitigate CSRF")
.contains("SameSite=Lax");
});
}
}
@@ -1,147 +0,0 @@
package stirling.software.proprietary.cluster;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.time.Duration;
import java.time.Instant;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;
import stirling.software.common.cluster.ClusterBackplane;
import stirling.software.common.cluster.JobStore;
import stirling.software.common.cluster.JobStoreEntry;
import stirling.software.common.cluster.KeyValueCache;
import stirling.software.common.cluster.RateLimitStore;
import stirling.software.common.cluster.RateLimitStore.RateLimitDecision;
import stirling.software.common.cluster.inprocess.InProcessJobStore;
import stirling.software.common.cluster.inprocess.InProcessKeyValueCache;
import stirling.software.common.cluster.inprocess.InProcessRateLimitStore;
/**
* Multi-node CONTRACT validation in a single JVM. Shares the cluster-visible state (JobStore,
* RateLimitStore, KeyValueCache) across two "nodes" - exactly the partition Valkey creates in
* production - and asserts cross-node visibility / global counters / cache propagation.
*
* <p><b>Scope note:</b> this test uses the in-process backplane implementations ({@link
* InProcessJobStore}, {@link InProcessKeyValueCache}, {@link InProcessRateLimitStore}), not the
* Valkey impls. It verifies the CONTRACT every {@code ClusterBackplane} flavor must honor (shared
* map semantics, monotonic counters, evict propagation) and is fast / Docker-free so it runs on
* every PR. The Valkey impls share the same contract by construction (single shared Valkey keyspace
* = single shared {@code ConcurrentHashMap} from the consumer's POV), so a regression here would
* also break the Valkey path.
*
* <p>For Valkey-specific verification (real Lettuce client, MULTI/EXEC atomicity, TTL expiry, WATCH
* race semantics on {@code delete}) see {@code LiveValkeyIntegrationTest}, which spins up a real
* Valkey via Testcontainers.
*
* <p>Result downloads are handled by sticky-session affinity at the load balancer + a {@code 410
* Gone} response on the rare miss (verified in {@code JobControllerOwnershipTest}).
*/
class MultiNodeClusterScenarioTest {
private JobStore sharedJobStore;
private RateLimitStore sharedRateLimit;
private KeyValueCache sharedCache;
private ClusterBackplane backplaneA;
private ClusterBackplane backplaneB;
@BeforeEach
void setUp() {
sharedJobStore = new InProcessJobStore();
sharedRateLimit = new InProcessRateLimitStore();
sharedCache = new InProcessKeyValueCache();
backplaneA = constBackplane("node-A", "valkey");
backplaneB = constBackplane("node-B", "valkey");
}
@Test
@DisplayName("async job created on node-A is readable from node-B via shared JobStore")
void jobStatusVisibleCrossNode() {
JobStoreEntry entry =
new JobStoreEntry(
"job-1",
JobStoreEntry.JobState.RUNNING,
"node-A",
Instant.now(),
null,
null,
List.of("file-1"),
Map.of());
sharedJobStore.put(entry, Duration.ofMinutes(30));
Optional<JobStoreEntry> seenOnB = sharedJobStore.get("job-1");
assertTrue(seenOnB.isPresent(), "node-B must see node-A's job in shared JobStore");
assertEquals("node-A", seenOnB.get().owningNodeId());
assertEquals(JobStoreEntry.JobState.RUNNING, seenOnB.get().state());
}
@Test
@DisplayName("global rate limit - capacity counted once across both nodes")
void rateLimitGlobalAcrossNodes() {
long capacity = 4L;
RateLimitDecision a1 =
sharedRateLimit.tryConsume("user:bob", capacity, Duration.ofMinutes(1));
RateLimitDecision b1 =
sharedRateLimit.tryConsume("user:bob", capacity, Duration.ofMinutes(1));
RateLimitDecision a2 =
sharedRateLimit.tryConsume("user:bob", capacity, Duration.ofMinutes(1));
RateLimitDecision b2 =
sharedRateLimit.tryConsume("user:bob", capacity, Duration.ofMinutes(1));
RateLimitDecision a3 =
sharedRateLimit.tryConsume("user:bob", capacity, Duration.ofMinutes(1));
assertTrue(a1.allowed());
assertTrue(b1.allowed());
assertTrue(a2.allowed());
assertTrue(b2.allowed());
assertFalse(a3.allowed(), "5th request across both nodes must be rejected (limit=4)");
}
@Test
@DisplayName("KeyValueCache populated on A is observed on B; evict on A propagates")
void apiKeyCacheVisibleCrossNode() {
sharedCache.put("apikey", "hash-bob", "bob", Duration.ofSeconds(60));
assertEquals("bob", sharedCache.get("apikey", "hash-bob").orElse(null));
sharedCache.evict("apikey", "hash-bob");
assertFalse(sharedCache.get("apikey", "hash-bob").isPresent());
}
@Test
@DisplayName("backplaneType reports 'valkey' on every node; localNodeId is distinct")
void backplaneType() {
assertEquals("valkey", backplaneA.backplaneType());
assertEquals("valkey", backplaneB.backplaneType());
assertEquals("node-A", backplaneA.localNodeId());
assertEquals("node-B", backplaneB.localNodeId());
assertNotEquals(backplaneA.localNodeId(), backplaneB.localNodeId());
assertNotNull(backplaneA.localNodeId());
}
private ClusterBackplane constBackplane(String nodeId, String type) {
return new ClusterBackplane() {
@Override
public boolean isHealthy() {
return true;
}
@Override
public String backplaneType() {
return type;
}
@Override
public String localNodeId() {
return nodeId;
}
};
}
}
@@ -0,0 +1,147 @@
package stirling.software.proprietary.cluster.s3;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatCode;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import java.net.URI;
import org.junit.jupiter.api.Test;
import software.amazon.awssdk.core.checksums.RequestChecksumCalculation;
import software.amazon.awssdk.core.checksums.ResponseChecksumValidation;
class S3ClientsTest {
@Test
void validateEndpointHost_publicAwsHost_passes() {
assertThatCode(
() ->
S3Clients.validateEndpointHost(
URI.create("https://s3.us-east-1.amazonaws.com"), false))
.doesNotThrowAnyException();
}
@Test
void validateEndpointHost_metadataServiceIp_rejected() {
assertThatThrownBy(
() ->
S3Clients.validateEndpointHost(
URI.create("http://169.254.169.254/"), false))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("allow-private-endpoints");
}
@Test
void validateEndpointHost_loopback_rejected() {
assertThatThrownBy(
() ->
S3Clients.validateEndpointHost(
URI.create("http://127.0.0.1:9000/"), false))
.isInstanceOf(IllegalStateException.class);
}
@Test
void validateEndpointHost_rfc1918Private_rejected() {
assertThatThrownBy(
() ->
S3Clients.validateEndpointHost(
URI.create("http://10.0.0.5:9000/"), false))
.isInstanceOf(IllegalStateException.class);
}
@Test
void validateEndpointHost_allowPrivateOptIn_bypassesCheck() {
assertThatCode(
() ->
S3Clients.validateEndpointHost(
URI.create("http://169.254.169.254/"), true))
.doesNotThrowAnyException();
assertThatCode(
() ->
S3Clients.validateEndpointHost(
URI.create("http://127.0.0.1:9000/"), true))
.doesNotThrowAnyException();
}
@Test
void validateEndpointHost_missingHost_rejected() {
assertThatThrownBy(
() ->
S3Clients.validateEndpointHost(
URI.create("file:///etc/passwd"), false))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("must include a host");
}
@Test
void validateEndpointHost_errorMessageNamesTheFlag() {
assertThat(
catchMessage(
() ->
S3Clients.validateEndpointHost(
URI.create("http://192.168.1.10:9000/"), false)))
.contains("storage.s3.allow-private-endpoints");
}
// ----- requestChecksumCalculation parsing -----
@Test
void parseRequestChecksum_nullOrBlank_defaultsToWhenSupported() {
assertThat(S3Clients.parseRequestChecksum(null))
.isEqualTo(RequestChecksumCalculation.WHEN_SUPPORTED);
assertThat(S3Clients.parseRequestChecksum(""))
.isEqualTo(RequestChecksumCalculation.WHEN_SUPPORTED);
assertThat(S3Clients.parseRequestChecksum(" "))
.isEqualTo(RequestChecksumCalculation.WHEN_SUPPORTED);
}
@Test
void parseRequestChecksum_caseInsensitive_andTrimmed() {
assertThat(S3Clients.parseRequestChecksum("when_required"))
.isEqualTo(RequestChecksumCalculation.WHEN_REQUIRED);
assertThat(S3Clients.parseRequestChecksum(" WHEN_REQUIRED "))
.isEqualTo(RequestChecksumCalculation.WHEN_REQUIRED);
assertThat(S3Clients.parseRequestChecksum("When_Supported"))
.isEqualTo(RequestChecksumCalculation.WHEN_SUPPORTED);
}
@Test
void parseRequestChecksum_unknownValue_fallsBackToDefault() {
assertThat(S3Clients.parseRequestChecksum("yes-please"))
.isEqualTo(RequestChecksumCalculation.WHEN_SUPPORTED);
assertThat(S3Clients.parseRequestChecksum("disabled-completely"))
.isEqualTo(RequestChecksumCalculation.WHEN_SUPPORTED);
}
// ----- responseChecksumValidation parsing -----
@Test
void parseResponseChecksum_nullOrBlank_defaultsToWhenSupported() {
assertThat(S3Clients.parseResponseChecksum(null))
.isEqualTo(ResponseChecksumValidation.WHEN_SUPPORTED);
assertThat(S3Clients.parseResponseChecksum(""))
.isEqualTo(ResponseChecksumValidation.WHEN_SUPPORTED);
}
@Test
void parseResponseChecksum_explicitWhenRequired_returnedAsEnum() {
assertThat(S3Clients.parseResponseChecksum("WHEN_REQUIRED"))
.isEqualTo(ResponseChecksumValidation.WHEN_REQUIRED);
}
@Test
void parseResponseChecksum_unknownValue_fallsBackToDefault() {
assertThat(S3Clients.parseResponseChecksum("nope"))
.isEqualTo(ResponseChecksumValidation.WHEN_SUPPORTED);
}
private static String catchMessage(Runnable r) {
try {
r.run();
return "";
} catch (RuntimeException e) {
return e.getMessage() == null ? "" : e.getMessage();
}
}
}
@@ -0,0 +1,253 @@
package stirling.software.proprietary.cluster.s3;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.stream.Stream;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.testcontainers.containers.MinIOContainer;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import stirling.software.common.cluster.FileStore;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.S3Configuration;
import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
import software.amazon.awssdk.services.s3.model.HeadObjectRequest;
import software.amazon.awssdk.services.s3.model.NoSuchKeyException;
@Testcontainers(disabledWithoutDocker = true)
class S3FileStoreTest {
private static final String BUCKET = "stirling-test-filestore";
private static final String ACCESS_KEY = "minioadmin";
private static final String SECRET_KEY = "minioadmin";
@Container
static MinIOContainer minio =
new MinIOContainer("minio/minio:latest")
.withUserName(ACCESS_KEY)
.withPassword(SECRET_KEY);
private static S3Client s3Client;
private static S3FileStore store;
@BeforeAll
static void setUp() {
URI endpoint = URI.create(minio.getS3URL());
AwsBasicCredentials creds = AwsBasicCredentials.create(ACCESS_KEY, SECRET_KEY);
S3Configuration s3Config = S3Configuration.builder().pathStyleAccessEnabled(true).build();
s3Client =
S3Client.builder()
.endpointOverride(endpoint)
.httpClient(UrlConnectionHttpClient.create())
.region(Region.US_EAST_1)
.credentialsProvider(StaticCredentialsProvider.create(creds))
.serviceConfiguration(s3Config)
.build();
s3Client.createBucket(CreateBucketRequest.builder().bucket(BUCKET).build());
store = new S3FileStore(s3Client, BUCKET, "transient/", false);
}
@AfterAll
static void tearDown() {
if (store != null) {
store.close();
}
if (s3Client != null) {
s3Client.close();
}
}
@Test
void blankBucket_constructorRejects() {
assertThatThrownBy(() -> new S3FileStore(s3Client, ""))
.isInstanceOf(IllegalArgumentException.class);
assertThatThrownBy(() -> new S3FileStore(s3Client, null))
.isInstanceOf(IllegalArgumentException.class);
}
@Test
void store_thenRetrieve_roundTripsContent() throws IOException {
byte[] payload = "hello cluster s3".getBytes(StandardCharsets.UTF_8);
FileStore.Stored stored = store.store(new ByteArrayInputStream(payload), "foo.txt");
assertThat(stored.fileId()).isNotBlank();
assertThat(stored.size()).isEqualTo(payload.length);
assertThat(store.exists(stored.fileId())).isTrue();
assertThat(store.size(stored.fileId())).isEqualTo(payload.length);
assertThat(store.retrieveBytes(stored.fileId())).isEqualTo(payload);
try (InputStream in = store.retrieve(stored.fileId())) {
assertThat(in.readAllBytes()).isEqualTo(payload);
}
}
@Test
void store_keysUseConfiguredPrefix() throws IOException {
byte[] payload = "prefixed".getBytes(StandardCharsets.UTF_8);
FileStore.Stored stored = store.store(new ByteArrayInputStream(payload), "p.txt");
String prefixed = store.resolveKey(stored.fileId());
assertThat(prefixed).startsWith("transient/");
s3Client.headObject(HeadObjectRequest.builder().bucket(BUCKET).key(prefixed).build());
assertThatThrownBy(
() ->
s3Client.headObject(
HeadObjectRequest.builder()
.bucket(BUCKET)
.key(stored.fileId())
.build()))
.isInstanceOfAny(
NoSuchKeyException.class,
software.amazon.awssdk.services.s3.model.S3Exception.class);
}
@Test
void emptyPrefix_writesAtBucketRoot() throws IOException {
S3FileStore rootStore = new S3FileStore(s3Client, BUCKET, "", false);
byte[] payload = "no-prefix".getBytes(StandardCharsets.UTF_8);
FileStore.Stored stored = rootStore.store(new ByteArrayInputStream(payload), "r.txt");
assertThat(rootStore.resolveKey(stored.fileId())).isEqualTo(stored.fileId());
assertThat(rootStore.retrieveBytes(stored.fileId())).isEqualTo(payload);
assertThat(rootStore.delete(stored.fileId())).isTrue();
}
@Test
void delete_removesObject_andReturnsTrue() throws IOException {
FileStore.Stored stored =
store.store(new ByteArrayInputStream(new byte[] {1, 2, 3}), "d.bin");
assertThat(store.delete(stored.fileId())).isTrue();
assertThat(store.exists(stored.fileId())).isFalse();
assertThatThrownBy(() -> store.retrieveBytes(stored.fileId()))
.isInstanceOf(IOException.class);
}
@Test
void delete_unknownKey_isIdempotentReturnsTrue() {
// S3 DeleteObject is idempotent (returns 204 whether or not the object existed).
// The store reflects S3's behaviour rather than racing a HEAD before each DELETE.
assertThat(store.delete("00000000-0000-0000-0000-000000000000")).isTrue();
}
@Test
void retrieve_missingKey_throwsIOException() {
assertThatThrownBy(() -> store.retrieveBytes("does-not-exist"))
.isInstanceOf(IOException.class);
assertThatThrownBy(() -> store.retrieve("does-not-exist")).isInstanceOf(IOException.class);
assertThatThrownBy(() -> store.size("does-not-exist")).isInstanceOf(IOException.class);
}
@Test
void exists_returnsFalseForBlankOrTraversalIds() {
assertThat(store.exists(null)).isFalse();
assertThat(store.exists("")).isFalse();
assertThat(store.exists("..")).isFalse();
assertThat(store.exists("a/b")).isFalse();
assertThat(store.exists("a\\b")).isFalse();
}
@Test
void delete_traversalId_returnsFalseWithoutCall() {
assertThat(store.delete("../etc/passwd")).isFalse();
assertThat(store.delete("foo/bar")).isFalse();
}
@Test
void store_largePayload_streamsViaTempFileWithoutBufferingInMemory() throws IOException {
long payloadSize = 16L * 1024 * 1024;
Path tempDir = Path.of(System.getProperty("java.io.tmpdir"));
long uploadTempsBefore = countS3UploadTemps(tempDir);
FileStore.Stored stored;
try (InputStream large = new RepeatingInputStream((byte) 0x42, payloadSize)) {
stored = store.store(large, "big.bin");
}
assertThat(stored.size()).isEqualTo(payloadSize);
assertThat(store.size(stored.fileId())).isEqualTo(payloadSize);
assertThat(countS3UploadTemps(tempDir)).isEqualTo(uploadTempsBefore);
store.delete(stored.fileId());
}
@Test
void store_uploadFailure_stillDeletesTempFile() {
Path tempDir = Path.of(System.getProperty("java.io.tmpdir"));
long uploadTempsBefore = countS3UploadTemps(tempDir);
// Non-existent bucket causes putObject to fail after the temp file is written, exercising
// the failure-path cleanup in the finally block.
S3FileStore brokenStore =
new S3FileStore(s3Client, "bucket-that-does-not-exist", "transient/", false);
assertThatThrownBy(
() ->
brokenStore.store(
new ByteArrayInputStream(
"payload".getBytes(StandardCharsets.UTF_8)),
"x.bin"))
.isInstanceOf(IOException.class);
assertThat(countS3UploadTemps(tempDir)).isEqualTo(uploadTempsBefore);
}
private static long countS3UploadTemps(Path tempDir) {
try (Stream<Path> entries = Files.list(tempDir)) {
return entries.filter(p -> p.getFileName().toString().startsWith("s3-upload-")).count();
} catch (IOException e) {
return 0L;
}
}
/** Generates {@code length} bytes of a single value without buffering them in memory. */
private static final class RepeatingInputStream extends InputStream {
private final byte value;
private long remaining;
RepeatingInputStream(byte value, long length) {
this.value = value;
this.remaining = length;
}
@Override
public int read() {
if (remaining <= 0) {
return -1;
}
remaining--;
return value & 0xFF;
}
@Override
public int read(byte[] b, int off, int len) {
if (remaining <= 0) {
return -1;
}
int toWrite = (int) Math.min(len, remaining);
for (int i = 0; i < toWrite; i++) {
b[off + i] = value;
}
remaining -= toWrite;
return toWrite;
}
}
}
@@ -0,0 +1,779 @@
package stirling.software.proprietary.cluster.s3;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.UUID;
import org.junit.jupiter.api.AfterAll;
import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.condition.EnabledIfEnvironmentVariable;
import org.springframework.mock.web.MockMultipartFile;
import stirling.software.common.cluster.FileStore;
import stirling.software.common.model.ApplicationProperties;
import stirling.software.proprietary.security.model.User;
import stirling.software.proprietary.storage.provider.S3StorageProvider;
import stirling.software.proprietary.storage.provider.StoredObject;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.services.s3.model.NoSuchBucketException;
import software.amazon.awssdk.services.s3.model.NoSuchKeyException;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;
import software.amazon.awssdk.services.s3.presigner.model.PresignedGetObjectRequest;
/**
* Comprehensive live-vendor test against a real S3-compatible endpoint specified via {@code
* S3_SMOKE_*} env vars. Skipped automatically when {@code S3_SMOKE_ENDPOINT} is not set, so CI is
* not affected. Covers:
*
* <ul>
* <li>{@code S3StorageProvider} CRUD: store / load / delete / presigned URL
* <li>{@code S3FileStore} CRUD (cluster artifact path)
* <li>Folder semantics simulated via key prefixes (matches production usage)
* <li>Negative paths: wrong secret, missing bucket, missing key, traversal IDs
* <li>Edge cases: zero-byte, unicode filename, multi-megabyte streaming
* <li>Configuration guards: SSRF endpoint rejection, bucket validation
* </ul>
*
* Every uploaded key is tracked and removed in {@link #cleanUp} so re-running against the same
* bucket leaves no residue.
*/
@EnabledIfEnvironmentVariable(named = "S3_SMOKE_ENDPOINT", matches = ".+")
class S3VendorComprehensiveTest {
private static final String PREFIX = "stirling-comprehensive/" + UUID.randomUUID() + "/";
private static ApplicationProperties.Storage.S3 cfg;
private static S3Clients.Bundle bundle;
private static S3StorageProvider provider;
private static String bucket;
private static String vendorLabel;
private static User owner;
private static final List<String> keysToCleanup =
Collections.synchronizedList(new ArrayList<>());
@BeforeAll
static void setUp() {
cfg = configFromEnv();
bucket = cfg.getBucket();
vendorLabel = System.getenv().getOrDefault("S3_SMOKE_LABEL", "external");
bundle = S3Clients.build(cfg, "comprehensive[" + vendorLabel + "]");
provider = new S3StorageProvider(bundle.client(), bundle.presigner(), bucket);
owner = new User();
owner.setId(7L);
owner.setUsername("comprehensive-tester");
}
@AfterAll
static void cleanUp() {
if (bundle != null) {
for (String key : keysToCleanup) {
try {
bundle.client().deleteObject(d -> d.bucket(bucket).key(key));
} catch (Exception e) {
// Best-effort cleanup; ignore.
}
}
try {
provider.close();
} catch (Exception ignored) {
}
bundle.close();
}
}
private static String track(String key) {
keysToCleanup.add(key);
return key;
}
private static ApplicationProperties.Storage.S3 configFromEnv() {
ApplicationProperties.Storage.S3 c = new ApplicationProperties.Storage.S3();
c.setEndpoint(System.getenv("S3_SMOKE_ENDPOINT"));
c.setBucket(requireEnv("S3_SMOKE_BUCKET"));
c.setRegion(System.getenv().getOrDefault("S3_SMOKE_REGION", "us-east-1"));
c.setAccessKey(requireEnv("S3_SMOKE_KEY"));
c.setSecretKey(requireEnv("S3_SMOKE_SECRET"));
c.setPathStyleAccess(
Boolean.parseBoolean(System.getenv().getOrDefault("S3_SMOKE_PATHSTYLE", "false")));
c.setAllowPrivateEndpoints(false);
return c;
}
private static String requireEnv(String name) {
String value = System.getenv(name);
if (value == null || value.isBlank()) {
throw new IllegalStateException(name + " env var must be set");
}
return value;
}
// ==========================================================================================
// FILE CRUD via S3StorageProvider (user-uploaded files)
// ==========================================================================================
@Test
void provider_store_thenLoad_matchesBytes() throws IOException {
byte[] payload = ("provider-roundtrip-" + vendorLabel).getBytes(StandardCharsets.UTF_8);
MockMultipartFile file = new MockMultipartFile("file", "doc.txt", "text/plain", payload);
StoredObject obj = provider.store(owner, file);
track(obj.getStorageKey());
assertThat(obj.getStorageKey()).isNotBlank();
assertThat(obj.getSizeBytes()).isEqualTo(payload.length);
assertThat(provider.load(obj.getStorageKey()).getInputStream().readAllBytes())
.isEqualTo(payload);
}
@Test
void provider_delete_removesObject() throws IOException {
byte[] payload = "delete-me".getBytes(StandardCharsets.UTF_8);
MockMultipartFile file = new MockMultipartFile("file", "x.txt", "text/plain", payload);
StoredObject obj = provider.store(owner, file);
track(obj.getStorageKey());
provider.delete(obj.getStorageKey());
assertThatThrownBy(() -> provider.load(obj.getStorageKey()))
.isInstanceOf(IOException.class);
}
@Test
void provider_load_missingKey_throws() {
assertThatThrownBy(() -> provider.load(PREFIX + "does-not-exist"))
.isInstanceOf(IOException.class);
}
@Test
void provider_presignedDownload_returnsBytesOverHttp() throws Exception {
byte[] payload = "presign me".getBytes(StandardCharsets.UTF_8);
MockMultipartFile file = new MockMultipartFile("file", "p.txt", "text/plain", payload);
StoredObject obj = provider.store(owner, file);
track(obj.getStorageKey());
java.util.Optional<java.net.URI> url =
provider.signedDownloadUrl(obj.getStorageKey(), Duration.ofMinutes(5));
assertThat(url).isPresent();
HttpResponse<byte[]> resp =
HttpClient.newHttpClient()
.send(
HttpRequest.newBuilder(url.get()).GET().build(),
HttpResponse.BodyHandlers.ofByteArray());
assertThat(resp.statusCode()).isEqualTo(200);
assertThat(resp.body()).isEqualTo(payload);
}
@Test
void provider_store_zeroBytes_isAccepted() throws IOException {
MockMultipartFile empty =
new MockMultipartFile("file", "empty.txt", "text/plain", new byte[0]);
StoredObject obj = provider.store(owner, empty);
track(obj.getStorageKey());
assertThat(obj.getSizeBytes()).isZero();
assertThat(provider.load(obj.getStorageKey()).getInputStream().readAllBytes())
.isEqualTo(new byte[0]);
}
@Test
void provider_store_unicodeFilename_yieldsOpaqueAsciiKey_andPreservesNameForDisplay()
throws IOException {
// Regression: pre-fix, the storage key embedded the filename verbatim, which Supabase
// rejected with 400 Invalid key. Post-fix, the key is {ownerId}/{uuid} (ASCII-only)
// and the original unicode name lives on StoredObject.originalFilename.
String unicodeName = "résumé-日本語-é.pdf";
byte[] payload = "u".getBytes(StandardCharsets.UTF_8);
MockMultipartFile file =
new MockMultipartFile("file", unicodeName, "application/pdf", payload);
StoredObject obj = provider.store(owner, file);
track(obj.getStorageKey());
assertThat(obj.getStorageKey()).matches("[0-9]+/[0-9a-fA-F-]+");
assertThat(obj.getStorageKey())
.isEqualTo(
new String(
obj.getStorageKey().getBytes(StandardCharsets.US_ASCII),
StandardCharsets.US_ASCII));
assertThat(obj.getOriginalFilename()).isEqualTo(unicodeName);
assertThat(provider.load(obj.getStorageKey()).getInputStream().readAllBytes())
.isEqualTo(payload);
}
// ==========================================================================================
// Concurrency, overwrite, TTL expiry (added after initial run surfaced the unicode bug)
// ==========================================================================================
@Test
void provider_concurrent10Uploads_allSucceedWithDistinctKeys() throws Exception {
int n = 10;
java.util.concurrent.ExecutorService pool =
java.util.concurrent.Executors.newFixedThreadPool(n);
try {
List<java.util.concurrent.Future<StoredObject>> futures = new ArrayList<>();
for (int i = 0; i < n; i++) {
final int idx = i;
futures.add(
pool.submit(
() -> {
byte[] payload =
("concurrent-" + idx).getBytes(StandardCharsets.UTF_8);
MockMultipartFile f =
new MockMultipartFile(
"file",
"c-" + idx + ".txt",
"text/plain",
payload);
StoredObject obj = provider.store(owner, f);
track(obj.getStorageKey());
return obj;
}));
}
java.util.Set<String> keys = new java.util.HashSet<>();
for (java.util.concurrent.Future<StoredObject> fut : futures) {
StoredObject obj = fut.get(30, java.util.concurrent.TimeUnit.SECONDS);
assertThat(keys.add(obj.getStorageKey()))
.as("distinct key for each parallel upload")
.isTrue();
assertThat(provider.load(obj.getStorageKey()).getInputStream().readAllBytes())
.isNotEmpty();
}
} finally {
pool.shutdownNow();
}
}
@Test
void sameKey_overwrite_returnsLatestPayload() {
String key = PREFIX + "overwrite-" + UUID.randomUUID() + ".txt";
track(key);
byte[] first = "FIRST".getBytes(StandardCharsets.UTF_8);
byte[] second = "SECOND".getBytes(StandardCharsets.UTF_8);
bundle.client().putObject(p -> p.bucket(bucket).key(key), RequestBody.fromBytes(first));
bundle.client().putObject(p -> p.bucket(bucket).key(key), RequestBody.fromBytes(second));
assertThat(getRaw(key)).isEqualTo(second);
}
@Test
void presignedDownload_afterTtlExpiry_returns403() throws Exception {
String key = PREFIX + "presign-expiry-" + UUID.randomUUID() + ".txt";
byte[] payload = "presign expiry".getBytes(StandardCharsets.UTF_8);
track(putRaw(key, "presign expiry"));
// 2-second TTL, then wait long enough that any vendor clock skew tolerance is also past.
PresignedGetObjectRequest presigned =
bundle.presigner()
.presignGetObject(
GetObjectPresignRequest.builder()
.signatureDuration(Duration.ofSeconds(2))
.getObjectRequest(g -> g.bucket(bucket).key(key))
.build());
// Confirm it works while valid - rules out unrelated failures.
HttpResponse<byte[]> ok =
HttpClient.newHttpClient()
.send(
HttpRequest.newBuilder(presigned.url().toURI()).GET().build(),
HttpResponse.BodyHandlers.ofByteArray());
assertThat(ok.statusCode()).isEqualTo(200);
assertThat(ok.body()).isEqualTo(payload);
Thread.sleep(5_000);
HttpResponse<byte[]> expired =
HttpClient.newHttpClient()
.send(
HttpRequest.newBuilder(presigned.url().toURI()).GET().build(),
HttpResponse.BodyHandlers.ofByteArray());
assertThat(expired.statusCode())
.as("presigned URL must be rejected after TTL expires")
.isIn(400, 403);
}
@Test
void provider_store_4MBPayload_streams() throws IOException {
byte[] payload = new byte[4 * 1024 * 1024];
java.util.Arrays.fill(payload, (byte) 0x42);
MockMultipartFile file =
new MockMultipartFile("file", "big.bin", "application/octet-stream", payload);
StoredObject obj = provider.store(owner, file);
track(obj.getStorageKey());
assertThat(obj.getSizeBytes()).isEqualTo(payload.length);
assertThat(provider.load(obj.getStorageKey()).getInputStream().readAllBytes())
.isEqualTo(payload);
}
// ==========================================================================================
// FILE CRUD via S3FileStore (cluster artifact path)
// ==========================================================================================
@Test
void fileStore_storeAndRetrieve_roundTrip() throws IOException {
S3FileStore store = new S3FileStore(bundle.client(), bucket, PREFIX + "fs/", false);
byte[] payload = "filestore round trip".getBytes(StandardCharsets.UTF_8);
FileStore.Stored stored = store.store(new ByteArrayInputStream(payload), "rt.txt");
track(store.resolveKey(stored.fileId()));
assertThat(store.size(stored.fileId())).isEqualTo(payload.length);
assertThat(store.retrieveBytes(stored.fileId())).isEqualTo(payload);
assertThat(store.exists(stored.fileId())).isTrue();
}
@Test
void fileStore_delete_returnsTrue_andExistsFalseAfter() throws IOException {
S3FileStore store = new S3FileStore(bundle.client(), bucket, PREFIX + "fs/", false);
FileStore.Stored stored = store.store(new ByteArrayInputStream("x".getBytes()), "del.txt");
assertThat(store.delete(stored.fileId())).isTrue();
assertThat(store.exists(stored.fileId())).isFalse();
}
@Test
void fileStore_retrieveBytes_missingKey_throws() {
S3FileStore store = new S3FileStore(bundle.client(), bucket, PREFIX + "fs/", false);
assertThatThrownBy(() -> store.retrieveBytes("does-not-exist"))
.isInstanceOf(IOException.class);
}
@Test
void fileStore_rejectsTraversalId() {
S3FileStore store = new S3FileStore(bundle.client(), bucket, PREFIX + "fs/", false);
assertThat(store.exists("..")).isFalse();
assertThat(store.delete("../etc/passwd")).isFalse();
assertThat(store.exists("a/b")).isFalse();
assertThat(store.exists("a\\b")).isFalse();
}
// ==========================================================================================
// Folder semantics simulated via key prefixes
// ==========================================================================================
@Test
void folderPrefix_isolatesObjects_andDeleteByPrefixDoesNotTouchRoot() throws IOException {
// Two "folders" + a root object - all reuse the test PREFIX so cleanup catches them.
String folderA = PREFIX + "folder-A/";
String folderB = PREFIX + "folder-B/";
String rootObj = PREFIX + "root-" + UUID.randomUUID() + ".txt";
track(putRaw(folderA + "file-1.txt", "in-A"));
track(putRaw(folderA + "file-2.txt", "in-A2"));
track(putRaw(folderB + "file-1.txt", "in-B"));
track(putRaw(rootObj, "at-root"));
// "Delete folder A": delete every key under folderA prefix
deleteAllUnderPrefix(folderA);
// Verify A is empty, B and root untouched
assertThat(headOrNull(folderA + "file-1.txt")).isNull();
assertThat(headOrNull(folderB + "file-1.txt")).isNotNull();
assertThat(headOrNull(rootObj)).isNotNull();
}
@Test
void moveBetweenFolders_viaCopyAndDelete_preservesContent() throws Exception {
String oldKey = PREFIX + "move-old/" + UUID.randomUUID() + ".txt";
String newKey = PREFIX + "move-new/" + UUID.randomUUID() + ".txt";
byte[] payload = "moveable".getBytes(StandardCharsets.UTF_8);
track(oldKey);
track(newKey);
bundle.client()
.putObject(p -> p.bucket(bucket).key(oldKey), RequestBody.fromBytes(payload));
// Simulate move: server-side copy + delete original.
bundle.client()
.copyObject(
c ->
c.sourceBucket(bucket)
.sourceKey(oldKey)
.destinationBucket(bucket)
.destinationKey(newKey));
bundle.client().deleteObject(d -> d.bucket(bucket).key(oldKey));
assertThat(headOrNull(oldKey)).isNull();
assertThat(getRaw(newKey)).isEqualTo(payload);
}
// ==========================================================================================
// Negative: wrong settings / wrong creds
// ==========================================================================================
@Test
void wrongSecret_throwsOnFirstOperation() {
ApplicationProperties.Storage.S3 bad = configFromEnv();
bad.setSecretKey("definitely-not-the-real-secret-" + UUID.randomUUID());
try (S3Clients.Bundle badBundle = S3Clients.build(bad, "wrong-secret")) {
assertThatThrownBy(() -> badBundle.client().headBucket(h -> h.bucket(bucket)))
.isInstanceOf(S3Exception.class)
.satisfies(e -> assertThat(((S3Exception) e).statusCode()).isIn(401, 403, 400));
}
}
@Test
void nonExistentBucket_throwsOnHeadOrPut() {
String fakeBucket = "stirling-no-such-bucket-" + UUID.randomUUID();
assertThatThrownBy(() -> bundle.client().headBucket(h -> h.bucket(fakeBucket)))
.isInstanceOfAny(NoSuchBucketException.class, S3Exception.class);
}
@Test
void blankBucket_atBuildTime_throwsIllegalState() {
ApplicationProperties.Storage.S3 bad = configFromEnv();
bad.setBucket("");
assertThatThrownBy(() -> S3Clients.build(bad, "blank-bucket"))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("bucket");
}
@Test
void invalidEndpointUri_atBuildTime_throwsIllegalState() {
ApplicationProperties.Storage.S3 bad = configFromEnv();
bad.setEndpoint("not a valid uri ::::");
assertThatThrownBy(() -> S3Clients.build(bad, "bad-uri"))
.isInstanceOf(IllegalStateException.class);
}
@Test
void privateEndpoint_withoutOptIn_atBuildTime_throwsIllegalState() {
ApplicationProperties.Storage.S3 bad = configFromEnv();
bad.setEndpoint("http://127.0.0.1:9000");
bad.setAllowPrivateEndpoints(false);
assertThatThrownBy(() -> S3Clients.build(bad, "loopback"))
.isInstanceOf(IllegalStateException.class)
.hasMessageContaining("private");
}
@Test
void getMissingKey_returnsNoSuchKey() {
String missing = PREFIX + "missing-" + UUID.randomUUID();
assertThatThrownBy(() -> bundle.client().getObject(g -> g.bucket(bucket).key(missing)))
.isInstanceOfAny(NoSuchKeyException.class, S3Exception.class);
}
// ==========================================================================================
// Bundle lifecycle
// ==========================================================================================
@Test
void bundleClose_isIdempotent() {
ApplicationProperties.Storage.S3 c = configFromEnv();
S3Clients.Bundle b = S3Clients.build(c, "lifecycle");
b.close();
b.close(); // should not throw
}
// ==========================================================================================
// Internal helpers (using the bundle directly for prefix/folder simulation)
// ==========================================================================================
private String putRaw(String key, String body) {
bundle.client()
.putObject(
p -> p.bucket(bucket).key(key),
RequestBody.fromBytes(body.getBytes(StandardCharsets.UTF_8)));
return key;
}
private byte[] getRaw(String key) {
return bundle.client().getObjectAsBytes(g -> g.bucket(bucket).key(key)).asByteArray();
}
private Object headOrNull(String key) {
try {
return bundle.client().headObject(h -> h.bucket(bucket).key(key));
} catch (Exception e) {
return null;
}
}
private byte[] tryGetBytes(String key) {
try {
return bundle.client().getObjectAsBytes(g -> g.bucket(bucket).key(key)).asByteArray();
} catch (Exception e) {
return null;
}
}
private void deleteAllUnderPrefix(String prefix) {
var listing = bundle.client().listObjectsV2(l -> l.bucket(bucket).prefix(prefix));
for (var obj : listing.contents()) {
bundle.client().deleteObject(d -> d.bucket(bucket).key(obj.key()));
}
}
// ==========================================================================================
// Key edge cases: leading/trailing/double slash, length, URL-special chars
// ==========================================================================================
@Test
void key_trailingSlash_storesAsZeroByteFolderMarker() {
String key = PREFIX + "folder-marker-" + UUID.randomUUID() + "/";
track(key);
// S3 spec: trailing slash is legal and creates a 0-byte "folder marker" object.
// Some vendors normalize it away; capture either behavior.
bundle.client()
.putObject(p -> p.bucket(bucket).key(key), RequestBody.fromBytes(new byte[0]));
Object head = headOrNull(key);
// Either: vendor accepts the marker (head is non-null) or normalizes to bare key.
assertThat(head != null || headOrNull(key.substring(0, key.length() - 1)) != null)
.as("vendor should either accept trailing-slash marker or normalize to bare key")
.isTrue();
}
@Test
void key_doubleSlash_normalizedOrStoredVerbatim() {
String key = PREFIX + "double//slash-" + UUID.randomUUID() + ".txt";
track(key);
bundle.client()
.putObject(
p -> p.bucket(bucket).key(key),
RequestBody.fromBytes("ds".getBytes(StandardCharsets.UTF_8)));
// Either GET-with-the-exact-key works, or vendor normalized -> single-slash form works.
String alt = key.replace("//", "/");
track(alt);
byte[] viaExact = tryGetBytes(key);
byte[] viaNormalized = tryGetBytes(alt);
assertThat(viaExact != null || viaNormalized != null)
.as("either exact double-slash key or normalized single-slash form must return")
.isTrue();
}
@Test
void key_200Chars_isStoredAndRetrievable() {
// Stirling production keys are ~45 chars ({ownerId}/{uuid}). 200 chars exceeds that by
// ~5x but stays inside every vendor's documented limit. The S3 spec max is 1024 bytes
// but some vendors (Supabase) impose stricter caps (~250-byte total path including
// bucket prefix - 1000 chars fails with KeyTooLongError).
StringBuilder sb = new StringBuilder(PREFIX + "long/");
while (sb.length() < 200) {
sb.append("abcdefghij");
}
String longKey = sb.substring(0, 200);
track(longKey);
byte[] payload = "long-key".getBytes(StandardCharsets.UTF_8);
bundle.client()
.putObject(p -> p.bucket(bucket).key(longKey), RequestBody.fromBytes(payload));
assertThat(getRaw(longKey)).isEqualTo(payload);
}
@Test
void key_safeSpecialChars_areSignedAndRetrievableViaSdk() {
// Restrict to chars every S3-compatible vendor accepts: dot, dash, underscore.
// Stirling's production key format ({ownerId}/{uuid}) is even narrower; this test
// confirms the SDK SigV4 signer copes with slightly more exotic ASCII-safe keys.
// Note: Supabase rejects keys containing space / + / ? / & / # ("400 Invalid key"),
// see documentsVendorKeyRestrictions_tolerantTest for that documentation.
String key =
PREFIX
+ "safe-special/"
+ UUID.randomUUID()
+ "_segment.with-dots.and_underscores.txt";
track(key);
bundle.client()
.putObject(
p -> p.bucket(bucket).key(key),
RequestBody.fromBytes("safe".getBytes(StandardCharsets.UTF_8)));
assertThat(getRaw(key)).isEqualTo("safe".getBytes(StandardCharsets.UTF_8));
}
@Test
void documentsVendorKeyRestrictions_tolerantTest() {
// Documents - rather than enforces - which key characters cause vendor rejection.
// Stirling production code is safe because S3StorageProvider always emits an
// ASCII-safe UUID-only key. If you ever change that, this test becomes a canary.
// AWS S3 and MinIO accept all of these; Supabase rejects all of them with 400.
String[] suspiciousKeys = {
PREFIX + "with space.txt",
PREFIX + "with+plus.txt",
PREFIX + "with#hash.txt",
PREFIX + "with?question.txt",
PREFIX + "with&amp.txt",
};
int accepted = 0;
int rejected = 0;
for (String k : suspiciousKeys) {
track(k);
try {
bundle.client()
.putObject(
p -> p.bucket(bucket).key(k),
RequestBody.fromBytes("x".getBytes(StandardCharsets.UTF_8)));
accepted++;
} catch (S3Exception e) {
assertThat(e.statusCode())
.as("vendor rejection must be a clean 4xx, not a signature mismatch")
.isBetween(400, 499);
rejected++;
}
}
assertThat(accepted + rejected).isEqualTo(suspiciousKeys.length);
}
// ==========================================================================================
// Presigned-URL: TTL bounds + Content-Disposition behavior (Stirling uses this for shares)
// ==========================================================================================
@Test
void presignedGet_ttlExceeding7Days_isRejectedAtSigningTime() {
String key = PREFIX + "ttl-overflow-" + UUID.randomUUID() + ".txt";
track(putRaw(key, "x"));
// SigV4 caps presigned URL TTL at 7 days. SDK should refuse to sign anything larger.
assertThatThrownBy(
() ->
bundle.presigner()
.presignGetObject(
GetObjectPresignRequest.builder()
.signatureDuration(Duration.ofDays(8))
.getObjectRequest(
g -> g.bucket(bucket).key(key))
.build()))
.isInstanceOfAny(IllegalArgumentException.class, RuntimeException.class);
}
@Test
void provider_signedDownloadUrl_attachmentDisposition_endsWithAttachmentHeader()
throws Exception {
byte[] payload = "attach me".getBytes(StandardCharsets.UTF_8);
MockMultipartFile file =
new MockMultipartFile("file", "report.pdf", "application/pdf", payload);
StoredObject obj = provider.store(owner, file);
track(obj.getStorageKey());
java.util.Optional<java.net.URI> url =
provider.signedDownloadUrl(
obj.getStorageKey(), Duration.ofMinutes(2), false, "report.pdf");
assertThat(url).isPresent();
HttpResponse<byte[]> resp =
HttpClient.newHttpClient()
.send(
HttpRequest.newBuilder(url.get()).GET().build(),
HttpResponse.BodyHandlers.ofByteArray());
assertThat(resp.statusCode()).isEqualTo(200);
// Supabase + AWS both honor response-content-disposition query param.
assertThat(resp.headers().firstValue("content-disposition").orElse(""))
.as("vendor must honor response-content-disposition override in presigned URL")
.startsWith("attachment");
}
@Test
void provider_signedDownloadUrl_inlineDisposition_endsWithInlineHeader() throws Exception {
byte[] payload = "inline".getBytes(StandardCharsets.UTF_8);
MockMultipartFile file =
new MockMultipartFile("file", "preview.pdf", "application/pdf", payload);
StoredObject obj = provider.store(owner, file);
track(obj.getStorageKey());
java.util.Optional<java.net.URI> url =
provider.signedDownloadUrl(
obj.getStorageKey(), Duration.ofMinutes(2), true, "preview.pdf");
assertThat(url).isPresent();
HttpResponse<byte[]> resp =
HttpClient.newHttpClient()
.send(
HttpRequest.newBuilder(url.get()).GET().build(),
HttpResponse.BodyHandlers.ofByteArray());
assertThat(resp.statusCode()).isEqualTo(200);
assertThat(resp.headers().firstValue("content-disposition").orElse(""))
.as("inline=true must set 'inline' disposition")
.startsWith("inline");
}
// ==========================================================================================
// List pagination + HEAD missing semantics
// ==========================================================================================
@Test
void listObjectsV2_paginationWithMaxKeys_returnsContinuationToken() {
// Stage 3 objects under a unique sub-prefix.
String prefix = PREFIX + "page-" + UUID.randomUUID() + "/";
for (int i = 0; i < 3; i++) {
track(putRaw(prefix + "obj-" + i, "p" + i));
}
var first = bundle.client().listObjectsV2(l -> l.bucket(bucket).prefix(prefix).maxKeys(1));
assertThat(first.contents()).hasSize(1);
assertThat(first.isTruncated()).isTrue();
assertThat(first.nextContinuationToken()).isNotBlank();
var second =
bundle.client()
.listObjectsV2(
l ->
l.bucket(bucket)
.prefix(prefix)
.maxKeys(2)
.continuationToken(first.nextContinuationToken()));
assertThat(second.contents()).hasSize(2);
assertThat(second.isTruncated()).isFalse();
}
@Test
void headObject_missingKey_throwsNoSuchKeyOr404() {
String missing = PREFIX + "head-missing-" + UUID.randomUUID();
assertThatThrownBy(() -> bundle.client().headObject(h -> h.bucket(bucket).key(missing)))
.isInstanceOf(S3Exception.class)
.satisfies(e -> assertThat(((S3Exception) e).statusCode()).isEqualTo(404));
}
/**
* Presigned-URL test scaffolding for parity with the smoke test (covers the SDK presign path).
*/
@Test
void presignGetObject_independentOfProvider_returnsBytes() throws Exception {
String key = PREFIX + "presign-direct-" + UUID.randomUUID() + ".txt";
byte[] payload = "direct presign".getBytes(StandardCharsets.UTF_8);
track(putRaw(key, "direct presign"));
PresignedGetObjectRequest presigned =
bundle.presigner()
.presignGetObject(
GetObjectPresignRequest.builder()
.signatureDuration(Duration.ofMinutes(2))
.getObjectRequest(g -> g.bucket(bucket).key(key))
.build());
HttpResponse<byte[]> resp =
HttpClient.newHttpClient()
.send(
HttpRequest.newBuilder(presigned.url().toURI()).GET().build(),
HttpResponse.BodyHandlers.ofByteArray());
assertThat(resp.statusCode()).isEqualTo(200);
assertThat(resp.body()).isEqualTo(payload);
}
}

Some files were not shown because too many files have changed in this diff Show More