The saas PaygChargeInterceptor classifies a request as AI / AUTOMATION /
API / BYPASSED using a fixed precedence: X-Stirling-Automation header >
@RequiresFeature(AUTOMATION) > @RequiresFeature(AI_SUPPORT) > API-key
auth > BYPASSED (manual UI).
Plug the AI surface and the automation sub-step path:
AI controllers (saas → easy import):
- AiCreateController, AiCreateInternalController, AiProxyController each
get class-level @RequiresFeature(AI_SUPPORT). The interceptor already
falls back to the bean type via AnnotationUtils.findAnnotation when no
method-level annotation is present.
Automation sub-step header (InternalApiClient in common):
- Every caller of InternalApiClient.post() is a parent automation flow
(PipelineProcessor, AiWorkflowService, PolicyExecutor) running a child
tool via loopback HTTP. Tag every dispatch with
X-Stirling-Automation: true so the child step bills as AUTOMATION
regardless of its own annotation — i.e. an AI-OCR step inside a
policy run is correctly billed AUTOMATION rather than AI.
Deviation (documented in test javadoc): PipelineController lives in
core and PolicyController lives in proprietary. Neither module can
import @RequiresFeature from saas without a forbidden upward dependency
(both build under STIRLING_FLAVOR=core/proprietary where saas is
absent). Promoting RequiresFeature + FeatureGate to common is a
non-trivial refactor (touches 18 files across multiple C-bucket
commits) and out of scope for C4. The X-Stirling-Automation header
covers sub-step dispatch via InternalApiClient; direct user-facing
calls to /api/v1/pipeline/handleData therefore bill as BYPASSED (WEB)
or API (api-key auth) rather than AUTOMATION. Annotating those
controllers is left to a future refactor that promotes the cap types
to common.
Tests:
- RequiresFeatureAnnotationRolloutTest pins the AI controller
classifications so a refactor can't silently regress to BYPASSED.
- InternalApiClientTest.postTagsRequestAsAutomation captures the
HttpHeaders submitted to RestTemplate and asserts the marker.
Push billable usage to Stripe via Supabase meter-payg-units edge function after the
ledger DEBIT durably commits. Closes the "we tracked it" → "Stripe knows" loop.
PaygMeterReportingService POSTs {team_id, stripe_customer_id, units, idempotency_key,
metadata.category} to a config-driven endpoint. Both payg.meter.endpoint and
payg.meter.service-role-token default empty → no-op in local dev / tests. Never
throws; non-2xx and exceptions bump payg.meter.errors counter and log WARN.
JobChargeService.close() delegates to JobService.close() and registers an
afterCommit synchronization that fires the meter event for paid teams. Skips
when: shadow row is REFUNDED, category is BYPASSED, team has no stripeCustomerId
(free tier — ledger entry suffices), or no synchronization is active (defensive).
Idempotency key: "process:<jobId>:close" — deterministic across pod retries and
the reconciliation backfill so Stripe never bills twice for one close.
Tests:
- PaygMeterReportingServiceTest: happy path body+headers, 5xx, ConnectException,
RuntimeException, blank/null endpoint, zero units, blank service-role-token.
- JobChargeServiceTest extended: subscribed posts; free-tier skips; refunded skips;
bypassed skips; no shadow row skips; no team-ext row skips; meter exception is
swallowed inside afterCommit; close() without an active transaction skips the
meter post but still closes the job.
Free-tier detection uses stripeCustomerId == null as the established pre-#6532
stand-in (see PaygWalletController for the same pattern). Tightens to
paygSubscriptionId != null when PR #6532's payg_team_extensions column lands.
GET /api/v1/payg/wallet returns the snapshot the FE useWallet hook
consumes — derived from EntitlementService (spend / cap / period),
PaygTeamExtensions (subscription state), and the wallet_category_summary
view (per-category breakdown). Leader callers get the team roster +
per-member sub-caps; member callers see an empty roster.
PATCH /api/v1/payg/cap updates wallet_policy.cap_units (no Stripe call —
cap is application-layer, enforced via the entitlement guard) and
invalidates the team's snapshot cache so the next read reflects the
change immediately. Leader-only — authorised inside the method since the
team is derived from the caller, not a path variable.
Adds findPrimaryMembership(userId) to TeamMembershipRepository so the
wallet endpoint can resolve the caller's team + role in a single query.
Tests: 11-case pure-Mockito slice covering free vs subscribed, leader vs
member, no-cap, anonymous → 401, teamless → empty free snapshot, plus
the cap update endpoint's leader-only enforcement and cache invalidation.
Known dependency on PR #6532: stripeSubscriptionId stays null until the
payg_subscription_id column lands; "subscribed" derived from
stripe_customer_id presence as a stand-in. billableLimit fallback of 500
units mirrors EntitlementService.DEFAULT_FREE_TIER_UNITS until
pricing_policy.free_tier_units_per_cycle ships.
Build the missing pieces from PR-C1's original plan:
- EntitlementService: Caffeine-cached (30s TTL, 10k entries) per-team snapshot of
spend vs cap, with invalidate(teamId) for explicit eviction after subscription
changes / cap edits. Reads from wallet_policy when present, falls back to a
500-unit free-tier default until PR #6532 lands payg_subscription_id +
pricing_policy.free_tier_units_per_cycle.
- EntitlementGuard: HandlerInterceptor on @AutoJobPostMapping routes only.
Anonymous + billable gate (AUTOMATION / AI_SUPPORT) -> 401 SIGNUP_REQUIRED.
Authenticated + missing gate -> 402 FEATURE_DEGRADED with state, cap, spend,
periodEnd, missingGates. Fails open on any internal error.
- MINIMAL semantics fix: CapEvaluator.gatesFor(MINIMAL) now returns
{OFFSITE_PROCESSING, CLIENT_SIDE}. DEGRADED state still blocks AUTOMATION +
AI_SUPPORT, but manual server-side tools stay available -- which is the
v2.1 PAYG framing (manual tools are free).
- RequiresFeature: extend @Target to TYPE so class-level annotation is legal
(the charge interceptor was already checking bean type).
- Register guard in PaygWebMvcConfig at order 1100, after the charge
interceptor (1000). 402 short-circuits handler but charge interceptor's
afterCompletion still cleans up temp files.
Tests: 50 new test cases across EntitlementServiceTest (12), EntitlementGuardTest
(13), updated CapEvaluatorTest (25, MINIMAL assertions). All 17 payg test suites
green.
Categorise every billable tool call before the charge pipeline runs. Manual
UI tools (BYPASSED) now short-circuit ahead of multipart materialisation and
openProcess — no temp files, no shadow row, no DB writes. API-key, AI, and
automation calls thread their category through ChargeContext onto the shadow
row alongside the originating JobSource so analytics survive job pruning.
Precedence (interceptor): X-Stirling-Automation header or
@RequiresFeature(AUTOMATION) -> AUTOMATION; @RequiresFeature(AI_SUPPORT) -> AI;
ApiKeyAuthenticationToken -> API; else BYPASSED.
- PaygChargeInterceptor: determineCategory() resolves category from header +
@RequiresFeature + auth principal. Bypass fast-path runs before doPreHandle.
New Micrometer counter payg.filter.bypassed counts skipped manual calls.
- ChargeContext: now carries BillingCategory as a non-null record component;
any context that reaches openProcess is API / AI / AUTOMATION.
- JobChargeService.recordShadowRow: copies billingCategory + jobSource onto
the shadow row so post-prune analytics joins stand alone.
- Tests: extended PaygChargeInterceptorTest with bypass + category resolution
(header / annotation / API-key / AI-with-header precedence); existing
billable-path tests switched from JWT to API-key auth since plain JWT now
short-circuits as BYPASSED. JobChargeServiceTest verifies the shadow row
carries billingCategory + jobSource end-to-end.
Stripe stays on a single flat-priced meter — category is metadata only.
V16 adds wallet_ledger.billing_category, payg_shadow_charge.billing_category +
job_source, and pricing_policy_stripe_price.stripe_product_id (operator
populates manually). All adds are nullable so pre-V16 rows stay NULL — the
interceptor stamps the category for new rows.
PAYG stays on a single flat-priced Stripe meter forever; this axis is purely
for in-app breakdowns and analytics. The wallet_category_summary view
pre-groups debits by team / month / category for the dashboard widget.
BillingCategory enum lists BYPASSED first as the default sentinel; no
downstream relies on ordinal(). JPA mapping uses STRING so renames stay
backward-compatible.
First chunk of Bucket 5 (cap evaluator + entitlement guard + wallet
controller + cap-setting endpoints). Pure compute layer only — no DB, no
caches, no Spring. The entitlement service (next commit) supplies the
inputs from wallet_policy + payg_shadow_charge sums; this just does the
math.
Ships:
- @RequiresFeature method-level annotation declaring required FeatureGate
values. Per the design doc the annotation is NOT required on every
@AutoJobPostMapping — the guard's default rule is "no annotation +
AutoJob → OFFSITE_PROCESSING". Only AUTOMATION (pipeline) and AI_SUPPORT
(AI proxies) need the explicit annotation.
- CapEvaluator with three static helpers:
- evaluate(spend, cap, warn%, degrade%, degradedFeatureSet)
→ Evaluation(state, featureSet, enabledGates). Single integer compare
on the hot path; defensive against null/zero/misconfigured inputs.
- combineTeamAndMember(teamEval, memberEval) → effective eval,
stricter state, intersected gates. Handles the per-member sub-cap
case from §3.6.
- allEnabled(required, enabled) → predicate the guard calls per request.
- 22 unit tests covering: null/zero cap, warn boundary, degrade boundary,
over-threshold, misconfigured thresholds, member-team combination,
state ordering, gate intersection, mapping FeatureSet → gates,
guard predicate edge cases.
Next commits (same PR):
- EntitlementService — snapshot computation with 30s Caffeine cache
- EntitlementGuard — HandlerInterceptor reading @RequiresFeature
- PaygWalletController — GET /api/v1/payg/wallet (PaygSnapshot DTO matching FE)
- CapAdminController — admin direct cap-setting endpoints
Tracked in notes/PAYG_DESIGN.md §3.6 + §3.7 + §7.6 (PR-C1) and
notes/payg-launch-checklist.html Bucket 5.
# Description of Changes
Add a backend for running any multi-step PDF operations. This is
designed to be used for the upcoming Policies feature, along with
anything else that will require automated running of PDF operations,
like the Automate tool or Processing Folders.
The implementation is not complete. I've tried to get all the
infrastructure in there so that we can add in whichever triggers we need
in the future (like cron triggers or watching folders on disk) but
currently it just supports manual triggering of the policy.
The basis of this work was the operation running from the Stirling
Engine, which this PR removes in favour of this new system. The only
currently accessible frontend way to test this work is to ask the AI
chat to execute multiple operations on a PDF, but I've also extensively
tested with direct API calls to make sure that the policies work and
persist properly.
### To test
- Ask the agent to “list all the things you can do and put them in a
markdown table”. I know we’re explicitly asking it for markdown, but I
don’t want to update the system prompt to ask it to make tables when
necessary because it’ll probably turn everything into a table, not sure
though, we can test in future.
- Notice how the loading is different
- Notice how the user chat is in a bubble but the agent chat is flat
(super standard design practice in AI tools, and looks much better when
the agent outputs mardown, expecially tables and needs room to do so)
- Ask it to do something different, then close the chat, and see that
the agent is marked as running and has a green outline and a green dot.
- Play around with resizing the chat to make it bigger/smaller
Open to any and all criticisms on any of the design choices, and of
course the usual, code etc.
Resizing
<img width="1572" height="812" alt="Screenshot 2026-06-01 at 2 47 53 PM"
src="https://github.com/user-attachments/assets/ec0ac1d0-01da-4025-bf7e-eea4eb544181"
/>
Loading (cool animation not visible through screenshot obviously)
<img width="559" height="141" alt="Screenshot 2026-06-01 at 2 53 41 PM"
src="https://github.com/user-attachments/assets/99f0b1f5-1719-4d78-8947-21b142293052"
/>
Removed bubbles for agent chat (maybe controversial, let me know) and
markdown now renders properly again
<img width="654" height="1060" alt="Screenshot 2026-06-01 at 2 55 01 PM"
src="https://github.com/user-attachments/assets/445f0889-a632-4751-9a16-f80ae388c632"
/>
## What this PR does
Wires the **B-1 shadow charging engine** into real HTTP request flow.
After this lands, flipping an internal team to ``PAYG_SHADOW`` via SQL
begins populating ``payg_shadow_charge`` automatically — with **zero
impact** on the legacy credit deduction path.
**This is the load-bearing PR for shadow mode.** Without it, B-1's
engine sits idle — nothing in the codebase calls
``JobChargeService.openProcess()`` from a real HTTP request.
Stacks on top of #6477 (PR B-1).
## Components
| Class | Role |
|---|---|
| ``PaygResponseBodyWrapperFilter`` | Servlet filter, installs tee'ing
response wrapper. Defers wrapper close to ``AsyncListener`` for
``DeferredResult`` / ``CompletableFuture`` controllers so the lifetime
spans the async window. |
| ``PaygResponseBodyWrapper`` | ``HttpServletResponseWrapper`` —
in-memory ``ByteArrayOutputStream`` up to 10 MiB; spills to ``TempFile``
above. ``materialisedPath()`` always returns a uniform ``Path``
interface. |
| ``PaygChargeInterceptor`` | ``AsyncHandlerInterceptor`` mirroring
``UnifiedCreditInterceptor`` shape. ``preHandle`` gates on
``@AutoJobPostMapping``, materialises multipart inputs, calls
``JobChargeService.openProcess``. ``afterCompletion`` branches on HTTP
status. |
| ``PaygOutputExtractor`` | Pulls PDFs out of the response body. Direct
``application/pdf`` returns body verbatim; ``application/zip`` iterates
entries and keeps each ``.pdf`` entry whose first bytes match the
``%PDF-`` magic. |
| ``PaygWebMvcConfig`` | Registers filter at end of Spring filter chain
(after security); interceptor after ``UnifiedCreditInterceptor``. |
| ``PaygFilterProperties`` | ``payg.filter.enabled`` master switch +
in-memory threshold + optional max-bytes ceiling. |
## Status branching in afterCompletion
| HTTP status | Action |
|---|---|
| **2xx** | Append OK step; extract PDFs from response;
``JobService.recordOutput`` per PDF |
| **4xx** | Append FAILED step with ``errorCode``. No refund — customer
paid for the attempt. No OUTPUT recording. |
| **5xx + OPENED** (first-step) |
``JobChargeService.markFirstStepFailed`` → shadow row flipped to
``REFUNDED``, process CLOSED. Refund counter incremented. |
| **5xx + JOINED** (mid-chain) | ``JobChargeService.decrementStepCount``
— step slot returned without resetting ``lastStepAt`` (workflow window
stays active for retry). |
## New ``JobChargeService`` methods
- **``markFirstStepFailed(jobId, reason)``** — flips shadow row to
``REFUNDED`` with ``refundedAt`` + ``refundReason``, closes the process.
Idempotent. Mimics the eventual Stripe
``meter_event_adjustment(cancel)`` flow that real-mode will invoke at
the same callsite. **Refund implies close** so a same-input retry can't
lineage-join into a refunded chain for free work.
- **``decrementStepCount(jobId)``** — defensive floor at 1; never drives
count negative.
## Schema
- Backend: ``V13__payg_shadow_charge_status.sql`` adds ``status``
(``CHARGED`` | ``REFUNDED``) + ``refunded_at`` + ``refund_reason``.
``DEFAULT 'CHARGED'`` so existing B-1 rows stay correct without
backfill.
- Supabase: matching migration in
[Stirling-PDF-SaaS#payg-shadow-charge-status](https://github.com/Stirling-Tools/Stirling-PDF-SaaS/tree/payg-shadow-charge-status)
## Fail-open semantics in shadow
Any ``RuntimeException`` in ``preHandle`` / ``afterCompletion`` is
logged at WARN, increments ``payg.filter.errors``, and lets the
customer's tool call proceed unbilled. This **reverses to fail-closed**
when ``wallet_policy.engine = PAYG`` (real charging) — that reversal
lives inside ``JobChargeService`` and ships with the cap evaluator PR
(PR-C1 in PAYG_DESIGN.md).
## Observability
Micrometer metrics:
- ``payg.filter.errors`` Counter — internal failures (preHandle +
afterCompletion). Alert source.
- ``payg.filter.calls`` Counter, tagged ``disposition`` (``OPENED`` |
``JOINED`` | ``SHORT_CIRCUIT``)
- ``payg.filter.refunds`` Counter — first-step 5xx refunds
- ``payg.filter.duration`` Timer — preHandle + afterCompletion
wall-clock per request
## Test coverage (38 tests across 4 classes)
- **PaygResponseBodyWrapperTest** (12 tests) — in-memory, spill,
threshold crossing mid-chunk, writer vs outputStream exclusivity,
``resetBuffer`` with and without spill, close idempotency, single-byte
writes across threshold.
- **PaygOutputExtractorTest** (7 tests) — direct PDF, parametrised
content type, ZIP with mixed entries + magic-byte gate, corrupt ZIP
fail-open, empty ZIP.
- **PaygChargeInterceptorTest** (13 tests) — all preHandle
short-circuits, OPENED disposition stash, fail-open on chargeService
exception, 2xx recordOutputs path, 5xx OPENED → markFirstStepFailed, 5xx
JOINED → decrementStepCount, 4xx FAILED step append, max-bytes ceiling
skip, PIPELINE header detection.
- **JobChargeServiceTest extended** (+6 tests) — markFirstStepFailed
happy path, idempotency, missing-shadow-row case, long-reason trim;
decrementStepCount happy path, floor-at-1 defence, missing-job no-op.
## What's NOT in this PR (deliberate)
- **No SpringBootTest layer.** The saas module doesn't have bootstrap
test infrastructure (Supabase JWT config + H2 schema harness).
Integration confidence comes from the manual staging deploy + SQL-flip
of an internal team. Bootstrap-test infra is a focused follow-up if
needed.
- **No saas-mode Behave / docker-compose.** Per design §17 — deferred.
Existing ``testing/cucumber/`` infrastructure doesn't yet have a
saas-profile compose target; that's its own PR when warranted.
- **No CreditService wire-in** (per design §13 decision). Per-row
comparison data moves to the reconciliation report PR (PR-S2).
``legacy_credits_charged`` + ``diff_pct`` columns stay at 0 in shadow
rows.
- **No reconciliation report endpoint.** Direct SQL queries against
``payg_shadow_charge`` cover the data-access need until patterns emerge.
## Rollback levers
| Symptom | Lever |
|---|---|
| Some / all tool calls breaking due to filter |
``payg.filter.enabled=false`` + restart (~20s) |
| Shadow rows look wrong for a specific team | ``UPDATE wallet_policy
SET engine = 'LEGACY' WHERE team_id = ?`` |
| Mass shadow weirdness | ``UPDATE wallet_policy SET engine = 'LEGACY'``
|
| Memory exhaustion from response tee | Lower
``payg.filter.response.in-memory-threshold-bytes`` |
## Test plan
- [ ] CI green (build + tests)
- [ ] Aikido / Snyk / SonarCloud clean
- [ ] Manual: deploy to staging
- [ ] Manual: flip one internal team via ``UPDATE wallet_policy SET
engine = 'PAYG_SHADOW' WHERE team_id = ?``
- [ ] Manual: hit ``/api/v1/security/add-password`` with that team's
JWT; verify a ``payg_shadow_charge`` row appears with
``status='CHARGED'``
- [ ] Manual: trigger a 503 (e.g. via temporary backend kill
mid-request); verify the resulting row is ``status='REFUNDED'`` + the
process is ``CLOSED``
- [ ] Manual: hit ``/api/v1/general/split`` with a multi-page PDF;
verify one OUTPUT signature per inner PDF appears in
``job_artifact_hash``
- [ ] Manual: chain ``add-password`` → ``compress`` on the output;
verify the second call JOINS the first process (no new shadow row) and
the inner output OUTPUT signature is what drove the lineage join
## Stacks on / references
- Stacks on: #6477 (B-1 — shadow charging engine)
- Schema mirror: Stirling-PDF-SaaS#payg-shadow-charge-status branch
- Design doc: ``notes/PAYG_FILTER_DESIGN.md`` (all 19 decisions DECIDED)
> 📌 **Stacked on
[#6464](https://github.com/Stirling-Tools/Stirling-PDF/pull/6464)**
(lineage primitives, still in review). #6469 has merged so its commits
are no longer in this PR's diff. Once #6464 merges, a final rebase
collapses the lineage-primitives commits out of this diff too — leaving
only the B-1 work.
## What this is
Process tracking + shadow charging engine. Bundles PR-I7 service half
with the non-filter piece of PR-I7a so the pieces ship together — none
of them is useful in isolation.
**Review focus:** the new files in:
- \`app/saas/src/main/java/stirling/software/saas/payg/job/\`
(\`JobService\`, \`JobContext\`, \`JoinOrOpenResult\`,
\`StaleJobCloser\`)
- \`app/saas/src/main/java/stirling/software/saas/payg/charge/\`
(\`JobChargeService\`, \`ChargeContext\`, \`ChargeOutcome\`,
\`JobInput\`)
-
\`app/saas/src/main/java/stirling/software/saas/payg/lineage/LineagePruneScheduler.java\`
- their tests
The 8 files inherited from #6464 (lineage primitives) are unchanged from
there — they ride along in this diff until #6464 lands.
The remaining work for shadow-in-staging is the ingress/egress filter
that wires controllers into this engine — that's PR B-2.
## Scope
### \`JobService\` — persistence + lineage policy
- **\`joinOrOpen\`** — the multi-input "any-match-joins, newest wins"
rule. Hash every input via the lineage detector; if any matches an open
process in the workflow window, attach to the one with the freshest
\`lastStepAt\`. Step-limit overflow on the matched job spawns a fresh
process; the new job's input signatures are still recorded so
\`mostRecentMatchWins\` routes future calls forward.
- **\`recordOutput\`** — post-tool-success path. Records OUTPUT
signatures so the next call that takes this file as input
lineage-matches into the same process.
- **\`appendStep\`** — audit-trail step row written after a tool
completes.
- **\`close\`** — idempotent; safe to call from multiple paths
(explicit, FE on-unload, scheduler). Returns the same row on re-close,
no state mutation.
- **\`findStale\` / \`closeStale\`** — workflow-window-based stale
closure used by the scheduler.
### \`JobChargeService\` — the orchestrator (shadow variant)
\`openProcess\` resolves the effective policy via
\`PricingPolicyService\` (now in main via #6469), derives the step-limit
for the current \`JobSource\` (with a defensive fallback if the policy
is missing an entry), delegates to \`JobService.joinOrOpen\`, and on
OPENED runs the \`DocumentClassifier\` + writes a \`payg_shadow_charge\`
row. Applies the policy-level \`minChargeUnits\` floor per design § 3.4.
Shadow variant only — never debits the ledger, never posts a Stripe
meter event. The real-charging follow-up reuses the same orchestration
and swaps the side-effect. \`legacyCreditsCharged\` on the shadow row
stays \`0\` until the legacy \`CreditService\` is wired in (PR B-2),
where the comparison becomes meaningful.
### Schedulers (both plain \`@Scheduled\`)
- **\`StaleJobCloser\`** — fixed-rate 60 s. Closes \`OPEN\` jobs idle
past the workflow window. API users never have to call close explicitly
— this is the safety net.
- **\`LineagePruneScheduler\`** — hourly cron, retention 1 h. Deletes
\`job_artifact_hash\` rows older than the retention window.
- **No \`@SchedulerLock\` / no \`shedlock\` table** — consistent with
the 5 existing unguarded \`@Scheduled\` tasks in \`:saas\`
(\`CreditResetScheduler\` and friends, none of which are guarded today).
Cluster-correctness across all 7 saas schedulers is tracked in design §
9 as a separate focused cleanup. Underlying operations are idempotent —
duplicate firings on multi-pod would be wasted DB load, not data
corruption.
### Records (call-shape glue for PR B-2's filter)
- \`JobContext\` / \`JoinOrOpenResult\` — input/output for
\`JobService\`.
- \`ChargeContext\` / \`ChargeOutcome\` — input/output for
\`JobChargeService\`.
- \`JobInput\` — paired \`(MultipartFile, materialised Path)\` so the
upcoming ingress filter can pass both views without re-materialising.
## Tests
**26 new, all green.**
- 14 × \`JobServiceTest\` — no-match → opens new, single-match → joins
existing, multi-input any-match-joins, multi-match newest-wins (older
job never even looked up), step-limit hit spawns fresh job (and original
\`stepCount\` is NOT mutated), empty inputs reject, stale-signature
handling, recordOutput delegation, close idempotency, closeStale,
appendStep persistence.
- 7 × \`JobChargeServiceTest\` — JOINED skips classifier + shadow write
entirely, OPENED writes shadow row + classifies (single + multi file
paths), \`minChargeUnits\` floor applied, step-limit resolved
per-\`JobSource\` from policy, missing source entry falls back to
conservative default of 10.
- 2 × \`StaleJobCloserTest\`, 3 × \`LineagePruneSchedulerTest\` —
scheduler-wiring smoke + constructor-validation tests.
\`ENABLE_SAAS=true ./gradlew :saas:test\` — BUILD SUCCESSFUL.
## What's not in this PR (lands in PR B-2)
- **Tool ingress/egress servlet filter.** The highest-risk piece —
materialises the request body into a \`JobInput\`, calls
\`JobChargeService.openProcess\` from every \`@AutoJobPostMapping\`,
records OUTPUT after success. Edge cases to validate: multipart parts,
async controllers, streaming responses, errored 5xx paths, very large
files. Design decisions for the filter are being worked through in
\`notes/PAYG_FILTER_DESIGN.md\` before any code is written.
- **Wire shadow path into legacy \`CreditService\`.** Every legacy debit
writes a comparison row carrying both the PAYG would-be units and the
legacy actual credits, populating \`diffPct\`.
## Design doc
\`notes/PAYG_DESIGN.md\` — PR-I7 + PR-I7a status updated to reflect this
bundle. § 9 carries the cluster-correctness deferral note alongside the
existing LISTEN/NOTIFY trade-off note. § 7.5.1 readiness summary shows
the path now needs just **1 more PR** (the filter half + CreditService
wire-in, both bundled into PR B-2).
# Description of Changes
Currently, it's not possible to develop the backend on Mac without
manually signing the JPDFium binaries yourself since macOS will reject
running the unsigned binaries. [We've now updated JPDFium to sign the
Mac binaries in
v1.0.2](https://github.com/Stirling-Tools/JPDFium/releases/tag/v1.0.2),
so update to use that version.
# Description of Changes
Change Stirling Engine to support deleting documents automatically. This
happens both on user logout and after an amount of time specified by the
Java when ingesting a document (allowing for personal documents to have
short lifetimes but org documents to be left in the db with no expiry
date). Also sets up an [ACL
policy](https://en.wikipedia.org/wiki/Access-control_list) for the
documents so the database knows which users have access to which
documents. This is not fully implemented in the Java, so currently all
docs are treated as having a single owner, the uploader, but
theoretically when we need to support org storage, we shouldn't need to
change the db schema.
## What this is
Three orthogonal interfaces — each with one production impl — for
detecting whether an incoming tool call should join an existing process
via content-hash lineage. Groundwork for PR-I7a: nothing in this PR
calls the detector yet; the ingress/egress filter that wires it into
every controller lands separately.
Built to be modular along three axes. Swapping any of them should not
require changes elsewhere:
| Axis | Interface | V1 impl | Plausible future impl |
|---|---|---|---|
| Hash algorithm | `LineageSignatureExtractor` |
`ByteHashSignatureExtractor` (SHA-256) | `PdfMetadataSignatureExtractor`
(PDF `/ID`, content-stream hash) |
| Storage backend | `JobLineageStore` | `JpaJobLineageStore` |
`RedisJobLineageStore` |
| Matching policy | `HashLineageDetector` | `DefaultHashLineageDetector`
| strategy-driven variant (any-match-joins for multi-input — lives in
JobService) |
## Interfaces
### `LineageSignatureExtractor` — what counts as a fingerprint
```java
public interface LineageSignatureExtractor {
Set<LineageSignature> extract(Path file) throws IOException;
String name();
}
```
File-based (not stream-based) so a future PDF-aware extractor can open
the same file via jpdfium / PDFBox and pull `/ID[0]` or a content-stream
hash. Multiple extractors compose at the detector layer — Spring
auto-wires all `LineageSignatureExtractor` beans, the detector unions
their results.
Production impl: **`ByteHashSignatureExtractor`** — SHA-256 over the
file via 64 KiB-buffered `DigestInputStream`. Hardware-accelerated by
the JVM (Intel SHA-NI, ARM SHA).
### `JobLineageStore` — where signatures live
```java
public interface JobLineageStore {
void record(UUID jobId, Set<LineageSignature> signatures, ArtifactKind kind);
Optional<LineageMatch> findOpenJobForSignatures(Long userId, Set<LineageSignature> candidates, Duration window);
int pruneOlderThan(Instant cutoff);
}
```
Knows nothing about storage technology. Production impl
**`JpaJobLineageStore`** runs a single joined query against
`job_artifact_hash` ⋈ `processing_job` — status + window filtering
happen at the database. The query is bounded by `Limit.of(1)` on the hot
path so a job set sharing a popular signature doesn't materialise
unwanted rows. A future `RedisJobLineageStore` (or write-through hybrid)
is a drop-in.
### `HashLineageDetector` — the high-level API
```java
public interface HashLineageDetector {
Optional<LineageMatch> detect(Long userId, Path inputFile) throws IOException;
void record(UUID jobId, Path file, ArtifactKind kind) throws IOException;
}
```
**`DefaultHashLineageDetector`** delegates extraction to every
registered `LineageSignatureExtractor`, storage to the configured
`JobLineageStore`, and reads `payg.lineage.workflow-window` (default
`PT5M`) from config. When a single extractor throws (e.g. a future
PDF-aware extractor against a malformed PDF), the other extractors still
contribute — failures don't block the byte-hash from landing.
## Profile gating
All three `@Component` beans (`JpaJobLineageStore`,
`ByteHashSignatureExtractor`, `DefaultHashLineageDetector`) are
`@Profile("saas")` — consistent with every other `:saas` bean. Without
this guard the JPA store would fail to wire against its profile-gated
repository in non-saas profiles that pull `:saas` onto the classpath.
## Tests
Run entirely in-memory; no database required.
- **`LineageSignatureTest`** — storage-key encoding round-trips, rejects
malformed `"type:value"` keys.
- **`ByteHashSignatureExtractorTest`** — identical bytes → identical
sigs; empty file hashes to the well-known SHA-256-of-empty constant; 10
MiB file streams without OOM.
- **`DefaultHashLineageDetectorTest`** — same-user / within-window /
status=OPEN filtering, multi-signature matching (one extractor sees
`pdf-id` and matches even when bytes differ), most-recent-job-wins,
record+detect round-trip, extractor-throwing-doesn't-break-others.
**`InMemoryJobLineageStore`** (in test sources) implements the same
`JobLineageStore` interface as the JPA impl, plus a `registerJob` hook
for tests to model job state. Same contract — proves the abstraction is
portable. When the Redis impl lands it gets the same contract tests.
## What's not in this PR (deliberate)
- The tool ingress/egress filter that wires the detector into every
controller — separate, focused review.
- `JobChargeService.openProcess()` — uses the detector, part of the
charging machinery, separate PR.
- Prune scheduler that calls `pruneOlderThan` — small follow-up
alongside the `shedlock` foundational table.
- PDF-aware extractor (`PdfMetadataSignatureExtractor`) — to be added
when we measure how often byte-hash-only misses real workflows.
- Multi-input "any-match-joins" lineage policy — that's a `JobService`
decision (PR-I7), not a primitive.
## Self-review pass applied
An independent code-review on this PR caught:
- **HIGH:** Missing `@Profile("saas")` on the three `@Component` beans →
fixed.
- **MEDIUM:** `pruneOlderThan` missing `@Transactional` (its
`@Modifying` query would have thrown
`InvalidDataAccessApiUsageException`) → fixed.
- **MEDIUM:** `findOpenJobsForSignatures` fetching the whole match set
just to `get(0)` → now takes `Limit`, JPA store passes `Limit.of(1)` on
the hot path.
- **LOW:** `InMemoryJobLineageStore` used both `synchronized` methods
and `ConcurrentHashMap` → dropped the redundant `ConcurrentHashMap`.
Deferred: project-wide UTC unification (`LocalDateTime.now()`
system-zone is the established convention; flipping one file mid-stack
caused a real test failure — proper fix needs its own audit).
## Rollback
Straight `git revert`. No callers yet; deleting these classes wouldn't
break anything.
---
## Checklist
- [x] Tests pass: `ENABLE_SAAS=true ./gradlew :saas:test`
- [x] No new warnings
- [x] Self-review performed (HIGH + MEDIUM findings addressed)
## What this is
PR-I1 service half from `notes/PAYG_DESIGN.md`. Built on top of the data
model from #6460 — answers "what pricing policy applies to this team
right now?" with a fast cache and an admin write surface.
## Scope
| Piece | Where |
|---|---|
| `PricingPolicyService` — `getEffectivePolicy(teamId)` with 30s
Caffeine cache + admin write paths |
`app/saas/.../payg/policy/PricingPolicyService.java` |
| `PolicyChangedEvent` — published after admin writes for in-process
cache invalidation | `app/saas/.../payg/policy/PolicyChangedEvent.java`
|
| Admin REST — list / get / create / set-default / set team override /
get effective |
`app/saas/.../payg/policy/admin/PricingPolicyAdminController.java` +
DTOs |
| `PricingPolicyRepository.clearDefaultFlag()` — atomic clear for
set-default | repository update |
| `SaasJpaConfigScanTest` — drift guard against the JPA scan paths going
stale (carried over from the #6460 review concern) | new test |
| V12 default-policy seed (`v1-initial`, 25 pages/unit, 5 MiB/unit,
per-`JobSource` step limits) | `V12__seed_default_payg_policy.sql` |
## Lookup precedence
1. `PaygTeamExtensions.pricingPolicyId` set → return that policy
2. Else return the `pricing_policy` row with `is_default = TRUE`
3. Override row points at a deleted policy → log warn, fall back to
default (safety net for racing deletes)
4. No default → `IllegalStateException` (V12 seed guarantees one exists)
## Cache behaviour
- 30s `expireAfterWrite` Caffeine, max 10k entries, keyed by `teamId`.
- **Single correctness model: the TTL.** Cross-instance propagation is
at-most-30-seconds. The writer instance sees its own change immediately
via the `PolicyChangedEvent` after-commit publish. Other instances pick
it up on the next TTL expiry.
- Admin reads use `getEffectivePolicyUncached` so admins always see
their own write straight back.
**Why no LISTEN/NOTIFY runner.** An earlier cut of this PR included a
Postgres `LISTEN policy_changed` runner so cross-instance propagation
was instant. Dropped — admin policy changes are events-per-week and the
30s TTL is already the correctness floor; the listener was ~250 lines of
nontrivial code (raw JDBC outside HikariCP, daemon thread, reconnect
loop, lock-protected connection lifecycle) for a use case that isn't on
the hot path. Trade-off is documented in `notes/PAYG_DESIGN.md` §9 with
three concrete triggers that would justify reintroducing it (aggressive
cap enforcement, Redis landing for other reasons, real-time admin UI).
## Writes — transactional, fire `PolicyChangedEvent` after commit
- `create(draft)` — rejects pre-set `policy_id` or `is_default=true`
(promotion must go through `setDefault` so the partial unique idx is
freed first).
- `setDefault(id)` — atomically clears the existing default via
`clearDefaultFlag()` then flips the new row. Idempotent: silent no-op if
the row is already default.
- `setTeamOverride(teamId, policyId | null)` — validates the policy
exists before save; `null` clears the override.
`publishOnCommit` uses `TransactionSynchronizationManager.afterCommit`
so listeners never see pre-commit state. Outside a transaction (test
paths) falls through to immediate publish.
## Admin REST surface — `/api/v1/admin/payg/...`
All endpoints `@PreAuthorize("hasRole('ADMIN')")`:
- `GET /policies` — list all
- `GET /policies/{id}` — read one
- `POST /policies` — create new (non-default)
- `POST /policies/{id}/set-default` — atomic promote
- `PUT /teams/{teamId}/policy-override` — set or clear per-team override
- `GET /teams/{teamId}/effective-policy` — cache-bypassing live read
Validation errors → 400, unknown rows → 404.
## Counterpart Supabase PR
[`Stirling-PDF-SaaS#298`](https://github.com/Stirling-Tools/Stirling-PDF-SaaS/pull/298)
— seeds the same V1 default policy on the Supabase side via
`20260528000002_payg_seed_default_policy.sql`.
## Tests
- 17 × `PricingPolicyServiceTest` — lookup precedence, cache hit/miss,
invalidation on event, mutation paths publishing event, error cases.
- 14 × `PricingPolicyAdminControllerTest` — every endpoint's happy path
+ error mapping, DTO defensive-copy invariant.
- 2 × `SaasJpaConfigScanTest` — reflection-based guard that
`payg.repository` is in `@EnableJpaRepositories` and `payg` is in
`@EntityScan`. Without this, new sub-packages can silently fail to wire
at runtime — same class of bug that the #6460 review caught.
Full `:saas:test` BUILD SUCCESSFUL.
## Design doc
`notes/PAYG_DESIGN.md` §7.4 PR-I1 — completes the service half (the
schema half landed in #6460). §9 carries the 30s-TTL trade-off note.
# Description of Changes
Two layers — the `DocumentClassifier` utility plus the full data model
for the new billing engine. Nothing wires the entities into application
behaviour yet; services and controllers land in follow-up PRs.
**Companion PR:**
[Stirling-PDF-SaaS#296](https://github.com/Stirling-Tools/Stirling-PDF-SaaS/pull/296)
— Supabase migration for the v3 dev branch, schema-equivalent to the
Flyway migration in this PR.
## 1. DocumentClassifier (under `payg.docs`)
`DocumentClassifier` computes the doc-unit cost of an uploaded file (or
multi-file input) under a `PricingPolicy`. PDFs read page count via
`stirling.software.jpdfium.PdfDocument`; non-PDFs are bytes-only.
Formula: `max(ceil(pages / docPagesPerUnit), ceil(bytes /
docBytesPerUnit))` clamped to `[1, fileUnitCap]`. Multi-file is the sum
of raw per-file units capped at `fileUnitCap × file_count`.
Two floors, by design: the classifier returns `docUnits` with an
absolute `1` floor for non-empty input; the policy-level
`minChargeUnits` is intentionally applied later, at process-open time in
`JobChargeService`, per design § 3.4 (`unitsForProcess =
max(policy.min_charge_units, docUnits)`). Documented in the interface +
impl javadoc.
Upload bytes are materialised through
`TempFileManager.createManagedTempFile` so jpdfium gets a `Path`; the
temp file auto-deletes on close.
Twelve tests, all in-memory fixtures generated with PDFBox at test time
— no committed binary blobs.
## 2. PAYG data model (under `payg.*`)
JPA entities, repositories, and a Flyway migration covering the full
schema in §6 of the design.
**Enums** (`payg.model`):
`JobSource`, `ProcessType`, `JobStatus`, `JobStepStatus`,
`ArtifactKind`, `LedgerEntryType`, `LedgerBucket`, `ReferenceType`,
`EntitlementState`, `FeatureSet`, `FeatureGate`, `WalletEngine`,
`CapPeriod`, `AutoGroupStrategy`.
**Entities + repositories:**
| Entity | Table | Notes |
|---|---|---|
| `PricingPolicy` | `pricing_policy` | Promoted from a record.
`stepLimits` is `Map<JobSource, Integer>` persisted via normalised child
table `pricing_policy_step_limit`. `stripePriceIds` is `Set<String>`
persisted via `pricing_policy_stripe_price` — currency comes from
`stripe.prices` via Sync Engine, not stored locally. |
| `ProcessingJob` | `processing_job` | UUID PK. Tracks lineage window
via `step_count` and `last_step_at`. |
| `ProcessingJobStep` | `processing_job_step` | Per-tool-call audit. |
| `JobArtifactHash` | `job_artifact_hash` | Composite key `(job_id,
content_hash, kind)`. `content_hash VARCHAR(128)` so multiple signature
schemes coexist as `"type:value"` storage keys. Lineage detector queries
this. |
| `WalletLedgerEntry` | `wallet_ledger` | Append-only, signed
`amount_units`. Two unique indexes kill double-posting. |
| `WalletPolicy` | `wallet_policy` | Per-team engine + cap + degradation
rules + lineage strategy. No `@Version` — admin-only writes (documented
in javadoc). |
| `WalletEntitlementSnapshot` | `wallet_entitlement_snapshot` |
Composite key `(team_id, user_id)`; `user_id = 0` is the team-wide
sentinel. No `@Version` — full-row recompute via
`EntitlementService.recompute` (documented in javadoc). |
| `PaygShadowCharge` | `payg_shadow_charge` | Per-job diff while in
`PAYG_SHADOW` engine mode. |
| `PaygTeamExtensions` | `payg_team_extensions` | Sidecar 1:1 with
`teams` carrying `pricing_policy_id` (per-team override) +
`stripe_customer_id`. Sidecar pattern (mirrors `saas_team_extensions`)
so OSS Hibernate ddl-auto never sees PAYG columns on `teams`. |
**Column adds:**
- `team_memberships.cap_units` (optional per-member sub-cap)
**Width split (intentional, documented in V11):** per-row deltas
(`wallet_ledger.amount_units`, `processing_job.charged_units`) are
`INTEGER` because no single charge realistically approaches 2B units.
Cap and period-rollup columns (`team_memberships.cap_units`,
`wallet_policy.cap_units`,
`wallet_entitlement_snapshot.period_spend_units / period_cap_units`) are
`BIGINT` because they accumulate across a billing period and admins may
legitimately set headroom-cap values into the millions.
**JPA wiring:** `SaasJpaConfig` was updated to include
`stirling.software.saas.payg.repository` in
`@EnableJpaRepositories.basePackages` and `stirling.software.saas.payg`
in `@EntityScan` (covers `payg.policy` / `payg.job` / `payg.wallet` /
`payg.entitlement` / `payg.shadow` recursively). New
`SaasJpaConfigScanTest` reads the annotations reflectively and asserts
every expected package is wired — catches the next time someone adds a
new sub-package without updating the scan paths.
**Migration:** `V11__saas_payg_model.sql` (purely additive).
Schema-equivalent to the Supabase migration in the companion PR —
including the `VARCHAR(128) content_hash` width that's needed for the
multi-signature-scheme storage encoding the lineage layer uses.
## 3. Smoke tests
`PaygEntitiesSmokeTest` exercises each entity via the no-arg ctor JPA
requires, plus getter/setter round-trips and composite-key equality —
catches Lombok/annotation regressions without needing a database.
Real-DB integration coverage lands alongside the services that consume
each entity.
## Why this is safe to land now
- All schema changes are additive — no existing rows modified, no
columns dropped.
- The entities are not yet referenced from any production code path;
they exist for the next PRs to build on.
- The v3 Supabase dev branch picks up the schema via the companion PR;
the main repo's Flyway migration applies the same shape when an instance
boots against a freshly-migrated v3 database.
## Open decisions made
- **Step-limits keyed by `JobSource`** rather than by `ProcessType`.
Captures the "self-hosted gets a different knob" framing in earlier
feedback. Trivially overridable per pricing policy version.
- **Step limits + Stripe price IDs normalised into child tables** rather
than JSONB on `pricing_policy` (per Connor's review on #296). Typed
columns, queryable directly, no JSON parsing.
- **Currency dropped from `pricing_policy_stripe_price`** — it lives on
`stripe.prices.currency` and is resolved via Sync Engine. App is
currency-blind.
## Rollback
Straight `git revert` on this PR. The Supabase migration in #296 is
additive and can be left in place safely — the running app ignores
tables it doesn't reference.
---
## Checklist
- [x] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [x] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
- [x] I have performed a self-review of my own code
- [x] My changes generate no new warnings
- [x] I have run `task check` (via `./gradlew :saas:test` with
`ENABLE_SAAS=true`) — passes
# Description of Changes
Two narrowly-scoped hardening changes to the credits engine.
## 1. CreditService — move Stripe meter call to `afterCommit`
The Stripe metered-usage call sits inside the surrounding
`@Transactional`, holding the `user_credits` row lock for the duration
of an HTTP round-trip to Supabase. Under load this starves concurrent
debits; a transient Stripe blip rolls back a (correct) free-credit
consumption and forces the caller to retry.
The Stripe call now runs in a `TransactionSynchronization.afterCommit`
hook — DB commits first, Stripe fires immediately after. If Stripe fails
after commit, we log + increment a new `credits.stripe_report.failures`
counter; the idempotency key is stable, so a manual replay recovers
without double-charging.
Applied to both `consumeCreditBySupabaseId` and
`consumeCreditWithWaterfall`.
**Dead-code removed:**
- Unreachable UUID fallback for MDC `requestId` — `CorrelationIdFilter`
already guarantees the key on every request.
- The `"Unable to report usage to Stripe"` `RuntimeException` and its
catch block — the afterCommit refactor eliminates the throw path.
- `StripeRollbackOnFailureTest` — pinned the rollback-on-Stripe-fail
behaviour this refactor replaces.
## 2. `@AutoJobPostMapping` — build-time lint for `resourceWeight`
`UnifiedCreditInterceptor` multiplies `resourceWeight` into the per-call
charge. An endpoint that falls through to the annotation default
produces a charge derived from a value nobody chose.
- Annotation default flipped from `1` to `Integer.MIN_VALUE` (sentinel).
Both runtime readers (`UnifiedCreditInterceptor`, `AutoJobAspect`)
already clamp into `[1, 100]` so behaviour is unchanged.
- New `AutoJobPostMappingWeightTest` scans the classpath and fails the
build if any method leaves the sentinel.
- Initial run caught 11 endpoints relying on the default. Explicit
weights now declared, chosen by comparing to peer endpoints:
- `EditTextController` — LARGE
- `EmailController#sendEmailWithAttachment` — SMALL
- `ConvertPDFToMarkdown` — MEDIUM
- `AttachmentController` (extract/list/rename/delete) — SMALL × 4
- `ConvertImgPDFController` (cbr/cbz ↔ pdf) — MEDIUM × 2, LARGE × 2
## Tests
- `StripeUsageIdempotencyKeyTest` — pins the `(supabaseId, overage,
requestId)` idempotency key shape so Stripe always dedupes a retry.
- `StripeAfterCommitOrderingTest` — pins that `afterCommit` fires after
commit and NOT on rollback.
- `AutoJobPostMappingWeightTest` — the lint itself, plus a self-check
that the classpath scan finds at least 10 `@AutoJobPostMapping` methods
(guards against the lint passing vacuously).
Build verified: `ENABLE_SAAS=true ./gradlew :stirling-pdf:test
:saas:test`.
---
## Checklist
### General
- [x] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [x] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable) — no translation changes
- [x] I have performed a self-review of my own code
- [x] My changes generate no new warnings
### Documentation
- [ ] I have updated relevant docs on [Stirling-PDF's doc
repo](https://github.com/Stirling-Tools/Stirling-Tools.github.io/blob/main/docs/)
— internal-billing change, no public docs impact
- [ ] I have read the section [Add New Translation
Tags](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md#add-new-translation-tags)
— N/A
### Translations (if applicable)
- [ ] Not applicable
### UI Changes (if applicable)
- [ ] Not applicable
### Testing (if applicable)
- [x] I have run `task check` (via `./gradlew :stirling-pdf:test
:saas:test` with `ENABLE_SAAS=true`) — passes
- [x] I have tested my changes locally
# Description of Changes
Various fixes and improvements I made while testing the SaaS code:
- Changes the new `.env.saas` file to live in `app/` and match the
semantics of the other `.env` files
- Adds top-level `task dev:saas` command to spawn SaaS frontend &
backend
- Deletes dead SaaS code and improves some overriding logic
- Fixes refreshing issue when coming back to the tab
- Fix the Compare tool's selection logic
- Make Compare handle error cases properly
- Fixes the location of the "Dismiss All Errors" button (was rendering
on top of the top-bar with a transparent background previously so it
looked rubbish)
- Fixes file selection in PDF Editor
# Description of Changes
## What & why
Customers using ADFS (or any generic OIDC provider that doesn't emit
`email`) hit `Attribute value for 'email' cannot be null` during OAuth2
login with no visibility into what claims the provider actually sent.
The only available remedy was guessing at
`security.oauth2.useAsUsername` until something worked.
This PR adds a new opt-in `security.oauth2.debugLogging` flag (default
`false`). When enabled, `CustomOAuth2UserService` logs:
- All ID token claims (sorted, with values)
- All UserInfo endpoint claims (if any)
- The merged attribute key set Spring exposes to `getAttribute()`
- The value the configured `useAsUsername` actually resolved to
- A **`Hint:`** line listing the claim keys present in the token that
map to a valid `UsernameAttribute` enum value — i.e. exactly what the
operator could put in `useAsUsername` to make login work
Logged at `INFO` on the success path and `ERROR` on failure (inside the
existing `catch (IllegalArgumentException)` block that throws
`OAuth2AuthenticationException`). The block is wrapped with a `[OAUTH2
DEBUG] ... [/OAUTH2 DEBUG]` banner and ends with a PII warning so
operators don't leave it on in production.
Default off → zero observable change for anyone not actively
troubleshooting.
## Files changed
| File | Why |
|---|---|
| `app/common/.../ApplicationProperties.java` | New `debugLogging` field
on the `OAUTH2` config class with javadoc warning about PII |
| `app/core/src/main/resources/settings.yml.template` | Documents
`oauth2.debugLogging` so it appears on next startup |
| `app/proprietary/.../security/service/CustomOAuth2UserService.java` |
Emits the claim dump + suggestion hint when the flag is on |
|
`app/proprietary/.../security/service/CustomOAuth2UserServiceDebugLoggingTest.java`
(new) | Unit test: mocks the OIDC delegate, asserts off-path is silent
and on-path emits the dump with the right Hint contents |
## End-to-end verification
Ran the bundled `testing/compose/docker-compose-keycloak-oauth.yml`
Keycloak realm, configured `security.oauth2.useAsUsername: mail`
(Keycloak emits `email`, not `mail`) and `provider: demarest` (matches
the original customer bug report). Triggered the OAuth flow at
`http://localhost:8080/oauth2/authorization/demarest` and confirmed:
- The ERROR-level dump fires with the full 19-claim ID token decoded
- `-- Value at 'mail' : <NULL — this is why login fails>` correctly
identifies the missing claim
- `-- Hint:` correctly suggests `[email, family_name, given_name,
preferred_username]` (the four keys present that map to valid
`UsernameAttribute` values)
- Auth still fails with the original `OAuth2AuthenticationException` —
no change to control flow, just added diagnostic logging
Unit test (`CustomOAuth2UserServiceDebugLoggingTest`) covers both
branches.
## Reviewer notes
- **No new public APIs.** The flag is config-only; no servlet endpoints
exposed.
- **PII is logged when the flag is on.** This is the whole point —
operators need to see the claims to fix their config — but it's gated,
defaults off, and the dump self-documents with a `WARNING: ... Set
security.oauth2.debugLogging=false once troubleshooting is complete.`
footer.
- **Why log everything, not just sub/email?** Because the operator
doesn't know in advance which claim they actually want. ADFS uses `upn`
in some configs and `preferred_username` in others; Azure AD uses `oid`;
the customer here had neither. Dumping the full set is the only way to
make the diagnostic self-service.
- **Out of scope for this PR (follow-ups):**
- The `UsernameAttribute` enum doesn't include `upn` / `unique_name`
(common ADFS claims). If the customer's token only has `upn`, the Hint
will be empty even though the operator can see `upn` in the dump. Worth
a separate PR to extend the enum.
- The known-provider validator in `Provider.java` (rejects e.g.
`useAsUsername: mail` for `provider: keycloak` at startup) bypasses our
diagnostic for those provider names. ADFS customers using `provider:
<name>` fall into the `default` branch so are not affected — but it's a
sharp edge worth documenting.
---
## Checklist
### General
- [x] I have read the [Contribution
Guidelines](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/CONTRIBUTING.md)
- [x] I have read the [Stirling-PDF Developer
Guide](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/DeveloperGuide.md)
(if applicable)
- [ ] I have read the [How to add new languages to
Stirling-PDF](https://github.com/Stirling-Tools/Stirling-PDF/blob/main/devGuide/HowToAddNewLanguage.md)
(if applicable) — N/A, backend-only change
- [x] I have performed a self-review of my own code
- [x] My changes generate no new warnings
### Documentation
- [ ] Doc-repo update (if functionality has heavily changed) —
diagnostic flag is self-documenting via the `settings.yml.template`
comment and the in-log warning; happy to add a doc-repo entry if
reviewers want one
- [ ] Translation tags — N/A
### UI Changes (if applicable)
- [ ] N/A — backend-only
### Testing (if applicable)
- [x] Unit test added (`CustomOAuth2UserServiceDebugLoggingTest`)
covering on/off paths and Hint correctness
- [x] End-to-end verified locally against bundled Keycloak compose with
intentionally misconfigured `useAsUsername`
- [x] Full `:proprietary:test` suite passes
`app/common/src/main/java/stirling/software/common/util/RequestUriUtils.java:175`
section comment said "Health Endoints" → "Health Endpoints".
Comment-only.
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
# Description of Changes
Give Edit Agent access to descriptions of the request from the Java API.
This opens the door to us better documenting our Java APIs to give the
stirling engine better knowledge of what the various tools are and how
to use them.
Also improves the tool selection sub-agent to get the tool parameters
and descriptions so it can more intelligently decide which operations
should be used to fulfil the user's request. Also provides it more
encouragement to string together multiple operations if necessary.
# Description of Changes
Adds storage in the database for full document content alongside the RAG
content (and changes the service to `DocumentService` instead of
`RagService`). Then adds a generic capability that should be usable by
any agent (currently just used by the Question Agent) which allows the
agent to pull out the full contents of the doc, chunks it into various
sections that will fit in the context window, and then processes them in
parallel to create an intermediate result, and then processes the
intermediate result into a final answer. It will re-chunk as many times
as necessary to get the content small enough for the actual answer to be
analysed (I've tested on PDFs ~3500 pages long, which is well above the
context limit and requires maybe 3 rounds of compression to get an
answer).
The new full doc analysis stuff is heavier than the RAG lookup so both
remain. The agents should use RAG for targeted info and the chunked
reasoner for info that requires reading the full doc.
Auto-generated by stirlingbot[bot]
This PR updates the backend license report based on dependency changes.
---------
Signed-off-by: stirlingbot[bot] <stirlingbot[bot]@users.noreply.github.com>
Co-authored-by: stirlingbot[bot] <195170888+stirlingbot[bot]@users.noreply.github.com>
Co-authored-by: Anthony Stirling <77850077+Frooodle@users.noreply.github.com>
## Description
Consolidates Playwright running under cohesive Task namespaces, isolates
Playwright state from the developer's local working tree, and swaps CI's
frontend webserver from `vite` dev to `vite preview` against a pre-built
`dist/`.
### `e2e:*` namespace
Renames `.taskfiles/testing.yml` to `.taskfiles/e2e.yml` and
consolidates everything Playwright-related under one `e2e:` namespace:
- `e2e:stubbed` / `e2e:live` / `e2e:enterprise` / `e2e:cross-browser`:
project-specific runners
- `e2e:check` (no-Docker subset) and `e2e:check:all` (full)
- `e2e:oauth:up` / `:down`, `e2e:saml:up` / `:down`: symmetric lifecycle
for the keycloak compose stacks
- `e2e:install`: Playwright browser install
- `docker:test`: full Docker integration suite
The redundant `frontend:test:e2e:*` project shortcuts are removed. CI
workflows (`e2e-stubbed.yml`, `e2e-live.yml`, `build-enterprise.yml`,
`nightly.yml`) are updated to call the new task names.
### Isolated Playwright state
New `STIRLING_BASE_PATH` (and `-Dstirling.base-path=`) override in
`InstallationPathConfig` redirects the entire state tree (configs,
backups, customFiles, pipeline, logs) at startup. `task e2e:live` points
it at `.test-state/playwright/` (purged on every invocation) so the
suite never touches the developer's local DB, settings.yml or backups.
`task e2e:live` auto-spawns gradle, waits for `/api/v1/info/status` to
come up, runs Playwright, then tears down the whole backend process
tree.
### CI runs Playwright against `vite preview`
Builds the frontend up-front with `VITE_BUILD_FOR_PREVIEW=1` (forces
absolute base so deep SPA routes resolve `/assets/...`) and the
playwright `webServer` now uses `vite preview --port 5173 --strictPort`
in CI. Avoids the per-page on-demand transform cost that was blowing the
30s navigation timeout under `--workers=3` on
`all-tool-pages-load.spec.ts`. Local dev keeps `vite` dev for HMR.
### OAuth/SAML compose helpers
`start-oauth-test.sh` and `start-saml-test.sh` gain a `--license-key
<KEY>` (`-k`) flag so CI and scripted runs can skip the interactive
license prompt. `start-oauth-test.sh` also moves from `for arg in "$@"`
to a `while`-with-`shift` arg loop to support multi-arg flags
consistently with the SAML script.
### Backend gradlew unification
Drops the per-platform `cmd /c gradlew.bat` branches from `backend.yml`
and routes every gradle invocation through `bash gradlew`. Works
uniformly on Linux/macOS and Windows-with-Git-Bash.
### Compare.tsx flake fix (re-land of
[#6316](https://github.com/Stirling-Tools/Stirling-PDF/pull/6316))
Piggybacks Anthony's never-merged fix from #6316. Without it,
`e2e:stubbed` continues to flake under `--workers=3` on
`compare.spec.ts`'s second-upload case via a React "Maximum update depth
exceeded" infinite loop in the Compare auto-fill effect. CI traces from
recent failed runs match exactly; 10 local runs of `compare.spec.ts`
with `CI=1 --workers=3` pass cleanly with the fix applied.
---------
Co-authored-by: James Brunton <jbrunton96@gmail.com>
Connor Yoh