feat(autoscan): Sonarr/Radarr webhook intake without arr API keys (#353)

* docs(autoscan): add arr webhook intake spec and implementation plan

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(autoscan): add webhook intake schema migration

Adds delivery_mode to autoscan_sources, the autoscan_webhook_endpoints
table, and delivery_mode/provider_event_type on autoscan_events.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(autoscan): add built-in arr-webhook source identity

Host-discovered scan-source entry so webhook-mode sources need no
plugin installation; composite lister appends it to plugin discovery.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(autoscan): persist delivery mode, webhook endpoints, event metadata

Sources carry delivery_mode; autoscan_webhook_endpoints CRUD with
SHA-256 token lookup and AAD-bound encrypted redisplay; events record
delivery_mode/provider_event_type; CreateEvent gains SkipRunningCheck
so webhook deliveries are never dropped by the poll exclusion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(autoscan): share the consume path and add webhook IngestChanges

Extracts consumeSourceChanges from PollOnce (marker semantics
preserved, existing poll tests unchanged); PollOnce skips webhook
sources; IngestChanges feeds deliveries through the shared pipeline
without markers and without the running-event exclusion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(autoscan): add Sonarr/Radarr webhook payload parser

Host-side arrwebhook package: provider inference, import/rename/delete
path extraction with vanished-path-friendly previous paths, subtree
fallback, exact-path dedupe, and no-op unknown events. Fixture-backed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(autoscan): add public webhook delivery route and admin endpoint management

Public POST /api/v1/autoscan/webhooks/{token} with per-IP rate
limiting, 256KiB body cap, 202-for-noop semantics, and token/body kept
out of logs; admin create/rotate/delete endpoint routes; source
responses carry delivery mode + webhook status/URL; create/update
validate delivery mode against source identity.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(web): add webhook delivery mode to Autoscan admin UI

Webhook sources get a generate/copy/rotate webhook URL section,
provider selector, delivery status, and a connection-free Add-source
flow; activity rows badge webhook deliveries with the arr event type.
Path rewrites stay editable in both modes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(api): redact secret path params from request and activity logs

The request logger and activity-log middleware recorded raw URLs, so
bearer credentials in secret path segments (autoscan webhook {token},
webhook-sync {secret}) were persisted to app logs and activity_log.
Redact the secret segment via the chi route params in both sinks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(autoscan): make webhook delivery reliable

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Quick
2026-07-09 14:13:31 -04:00
committed by GitHub
co-authored by Claude Fable 5
parent 8b9149a789
commit d68e70bb47
45 changed files with 4863 additions and 304 deletions
+1
View File
@@ -2052,6 +2052,7 @@ func main() {
intervalMs = int64(settings.DefaultPollIntervalSeconds) * 1000
}
taskMgr.Register(tasks.NewAutoscanPollTask(autoscanSvc, intervalMs))
taskMgr.Register(tasks.NewAutoscanWebhookRetryTask(autoscanSvc))
}
reconcileProviderIDRepo := catalog.NewProviderIDRepository(deps.DB)
reconcileEpisodeRepo := catalog.NewEpisodeRepository(deps.DB)
@@ -0,0 +1,376 @@
# Implementation Plan: Autoscan Sonarr/Radarr Webhook Intake
**Spec:** `docs/superpowers/specs/2026-07-09-autoscan-arr-webhook-intake-design.md`
(Option A, decisions locked — read the spec's "Resolved review decisions" before
starting.)
**Date:** 2026-07-09
Commands assume the repository root is the cwd.
## Overview
Add a webhook delivery mode to Autoscan so Sonarr/Radarr can POST import,
upgrade, rename, and delete notifications directly to Silo without an arr API
key. Webhook sources bind to a host-discovered built-in identity
(`silo.autoscan.arr-webhook`), never the installed ARR plugin. The host parses
payloads and feeds the existing rewrite → resolve → suppress → enqueue → event
pipeline.
Non-negotiable design points from the review:
1. **Built-in source identity** — no plugin install required for webhook mode.
2. **Webhook deliveries are never single-flight-dropped.** Only poll cycles may
be skipped when a running event exists (their marker window is re-read).
3. Event rows record `delivery_mode` and `provider_event_type` from day one.
4. Unknown arr event types → `202` no-op. Secret lookup uses plain SHA-256.
Webhook URL stays redisplayable on admin endpoints.
Suggested commit sequence is one commit per phase below, all on one branch, PR
subject `feat(autoscan): add Sonarr/Radarr webhook intake`.
## Phase 1 — Migration
Create with `make migrate-create NAME=autoscan_webhook_intake` (single
timestamped Goose migration; do not hand-number, do not create paired
up/down files).
Up:
```sql
ALTER TABLE autoscan_sources
ADD COLUMN delivery_mode text NOT NULL DEFAULT 'poll'
CONSTRAINT autoscan_sources_delivery_mode_check
CHECK (delivery_mode = ANY (ARRAY['poll'::text, 'webhook'::text]));
CREATE TABLE autoscan_webhook_endpoints (
source_id uuid PRIMARY KEY
REFERENCES autoscan_sources(id) ON DELETE CASCADE,
secret_hash text UNIQUE NOT NULL,
secret_ref text NOT NULL,
secret_suffix text NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
rotated_at timestamptz,
last_received_at timestamptz,
last_error_at timestamptz,
last_error_message text NOT NULL DEFAULT ''
);
ALTER TABLE autoscan_events
ADD COLUMN delivery_mode text NOT NULL DEFAULT 'poll',
ADD COLUMN provider_event_type text NOT NULL DEFAULT '';
```
Down reverses all three (drop table, drop columns). Verify with
`make migrate-status` / `make migrate-up` against a scratch DB, and check
rollback.
## Phase 2 — Built-in source identity
**`internal/autoscan/discovery.go`**
- Add exported constants:
- `BuiltinArrWebhookPluginID = "silo.autoscan.arr-webhook"`
- `BuiltinArrWebhookCapabilityID = "arr-webhook"`
- display name `"Sonarr/Radarr Webhook"`.
- Add a composite lister so the built-in identity lives beside the discovery
types it extends (not in wiring code):
```go
// WithBuiltinSources wraps a ScanSourceLister and appends host-built-in
// source identities that need no plugin installation.
func WithBuiltinSources(inner ScanSourceLister, builtins ...DiscoveredSource) ScanSourceLister
```
It must tolerate `inner == nil` (return only builtins) because
`Service.lister` is already allowed to be nil.
**`internal/api/autoscan_wiring.go`**
- In `BuildAutoscanService`, wrap the existing lister:
```go
autoscan.WithBuiltinSources(
PluginScanSourceLister{installationStore},
autoscan.BuiltinArrWebhookSource(), // helper returning the DiscoveredSource
)
```
Tests (`internal/autoscan/discovery_test.go`): builtin appears with and without
an inner lister; plugin entries pass through unchanged.
## Phase 3 — Repository and types
**`internal/autoscan/types.go`**
- `Source`: add `DeliveryMode string` (values `"poll"`/`"webhook"`; constants
`DeliveryModePoll`, `DeliveryModeWebhook`).
- `Event`: add `DeliveryMode string`, `ProviderEventType string`.
- `EventCreate`: add `DeliveryMode string`, `ProviderEventType string`,
`SkipRunningCheck bool`.
- New:
```go
type WebhookEndpoint struct {
SourceID string
SecretSuffix string
CreatedAt time.Time
RotatedAt *time.Time
LastReceivedAt *time.Time
LastErrorAt *time.Time
LastErrorMessage string
}
```
The plaintext token and hash never live on the struct that flows to
handlers/responses; creation/rotation return the token separately.
**`internal/autoscan/repository.go`**
- Thread `delivery_mode` through `CreateSource`, `UpdateSource`,
`ListSources`, `ListEnabledSources`, `GetSource` (default `'poll'` when
empty on write).
- `CreateEvent`: write the two new columns. When `in.SkipRunningCheck` is
true, use the simple no-transaction insert path (the same shape as the
existing `sourceID == ""` branch, but with the source id set) — no advisory
lock, no running-row check. Poll-mode behavior is unchanged.
- `FinishEvent` / `ListEvents` / `ListRunningEvents` / scan-event joins:
select and populate the new event fields.
- Webhook endpoint methods (token = 32 bytes from `crypto/rand`,
`base64.RawURLEncoding`; hash = hex SHA-256 of the token; `secret_ref` =
`r.cipher.Encrypt(token, secret.RowAAD("autoscan_webhook_endpoints", "secret_ref", sourceID))`;
suffix = last 6 chars of the token):
```go
// CreateWebhookEndpoint creates (or returns the existing) endpoint; token is
// "" when the endpoint already existed.
CreateWebhookEndpoint(ctx, sourceID string) (WebhookEndpoint, token string, err error)
RotateWebhookEndpoint(ctx, sourceID string) (WebhookEndpoint, token string, err error)
DeleteWebhookEndpoint(ctx, sourceID string) error
GetWebhookEndpoint(ctx, sourceID string) (WebhookEndpoint, error)
// RevealWebhookToken decrypts secret_ref for admin redisplay.
RevealWebhookToken(ctx, sourceID string) (string, error)
// ResolveWebhookToken maps a raw delivery token to its source. Constant
// shape: hash the token, look up by secret_hash, join autoscan_sources.
// Returns ErrNotFound for unknown hashes.
ResolveWebhookToken(ctx, token string) (Source, WebhookEndpoint, error)
TouchWebhookReceived(ctx, sourceID string) error
RecordWebhookError(ctx, sourceID, msg string) error // bounded, sanitized by caller
```
DB-backed tests follow the existing `SILO_TEST_DATABASE_URL` pattern used by
the rest of `internal/autoscan`. Cover: create/rotate invalidates the old
token, resolve-by-token round-trip, cascade delete with the source, reveal
decrypts to the original token.
## Phase 4 — Service: shared consume path + `IngestChanges`
**`internal/autoscan/service.go`**
- Extract everything in `PollOnce` after `PollChanges` returns into:
```go
type consumeOptions struct {
EventID int64
Marker string // poll only; "" for webhook
NextMarker string // poll only
AdvanceMarker bool // false for webhook
}
func (s *Service) consumeSourceChanges(ctx context.Context, src Source, changes []Change, opts consumeOptions) error
```
It owns: `rewriteChanges` → `resolveAndClaim` → target-cap collapse →
`enqueueScanTargets` (+ `releaseClaims` on failure) → the
transient-error / unresolved / success status decision → `finishEvent`.
Marker advancement stays conditional on `opts.AdvanceMarker`; the hold-marker
comments and semantics in `PollOnce` must survive the refactor intact.
- `PollOnce`: skip sources with `src.DeliveryMode == DeliveryModeWebhook`
(before the interval check), then delegate to `consumeSourceChanges`.
Existing poll tests in `service_test.go` must pass unmodified except for
construction changes.
- Add:
```go
type ChangeIngest struct {
SourceID string
ProviderEventType string
Changes []Change
ReceivedAt time.Time
}
type IngestResult struct {
Enqueued int
Suppressed int
Unresolved bool
}
func (s *Service) IngestChanges(ctx context.Context, in ChangeIngest) (IngestResult, error)
```
Behavior:
- Load the source; reject non-webhook sources.
- `CreateEvent` with `DeliveryMode: "webhook"`, `ProviderEventType`,
`SkipRunningCheck: true` — deliveries are never dropped because another
event is running (spec: "Security and reliability").
- Delegate to `consumeSourceChanges` with `AdvanceMarker: false` and empty
markers.
- Transient resolve failure → finish event as error and return an error (the
handler maps it; a duplicate later delivery is safe). All paths outside
Silo folders → finish as `unresolved`, return success.
Service tests: webhook ingest enqueues the same targets as an equivalent poll;
concurrent ingests for one source both complete (no `ErrPollAlreadyRunning`);
suppressor claims dedupe overlapping targets; `PollOnce` skips webhook sources.
## Phase 5 — Parser: `internal/autoscan/arrwebhook`
New package, no dependencies on api/handlers.
```go
type ParsedWebhook struct {
Provider string // "sonarr" | "radarr"
EventType string
Test bool
Changes []autoscan.Change
}
func Parse(provider string, body []byte) (ParsedWebhook, error)
```
- `provider` is `sonarr`, `radarr`, or `auto`; `auto` infers from top-level
`series` vs `movie` keys. Inference failure on a non-test event returns a
typed error the handler maps to 400 (sanitized message, no payload echo).
- Event handling per the spec's "Payload parsing" section: Test → no changes;
Download/Import/Upgrade/DownloadComplete → imported file paths; Rename → new
**and** previous paths; EpisodeFileDelete/MovieFileDelete → deleted path.
All paths emit `ChangeScopeFile` (`resolveChange` already falls back through
`ResolveVanishedPath` for vanished files). Series/movie folder paths are a
subtree fallback (`ChangeScopeSubtree`) only when no file path exists.
- Unknown event types return `ParsedWebhook` with no changes and a marker the
handler can distinguish (e.g. `EventType` set, `Changes` empty, no error) —
they must not be an error.
- Dedupe exact paths before returning.
Fixtures in `internal/autoscan/arrwebhook/testdata/`: Sonarr Download, Rename,
EpisodeFileDelete, Test; Radarr Download, Rename, MovieFileDelete, Test; plus
an unknown-event and a malformed body. Build them from the Servarr custom-
scripts docs linked in the spec.
## Phase 6 — HTTP: public delivery route + admin endpoints
**New file `internal/api/handlers/autoscan_webhook.go`**
Public handler `HandleWebhookDelivery` (`POST /api/v1/autoscan/webhooks/{token}`):
1. `http.MaxBytesReader` at 256 KiB → 413 on overflow.
2. Hash token, `ResolveWebhookToken` → 404 on miss (identical body for
deleted/never-existed; do not distinguish).
3. Global autoscan disabled or source disabled → 202, no enqueue, still
`TouchWebhookReceived`.
4. `arrwebhook.Parse` with `source_config.webhook_provider` (default `auto`)
→ 400 on parse error; 202 no-op for Test and unknown event types.
5. `IngestChanges`; on success `TouchWebhookReceived` and return `202`. On
transient ingest failure, `RecordWebhookError` (bounded message) and return
`500` so arr retries — a duplicate later delivery is safe. Log only source
id, provider, event type, and path count. **Never log the token, URL, or
body.**
Admin handlers (same file, wired through `AutoscanHandler` deps):
- `POST /admin/autoscan/sources/{id}/webhook` → create-if-missing, return
URL + status.
- `POST /admin/autoscan/sources/{id}/webhook/rotate` → rotate, return new URL.
- `DELETE /admin/autoscan/sources/{id}/webhook` → delete.
- URL construction: `deps.PublicURL + "/api/v1/autoscan/webhooks/" + token`
when `PublicURL` is set; otherwise return the relative path and let the UI
prepend `window.location.origin`. Response field is `webhook_url` either way.
**`internal/api/handlers/autoscan.go`**
- `sourceResponse`: add `delivery_mode`, `webhook_configured`, `webhook_url`,
`webhook_secret_suffix`, `webhook_last_received_at`, `webhook_last_error_at`,
`webhook_last_error_message` (fetch endpoint rows for listed sources; a
single batched query, not N+1).
- `HandleCreateSource` / `HandleUpdateSource` validation:
- `delivery_mode` defaults to `poll`; must be `poll` or `webhook`.
- `webhook` is valid only for the built-in identity; `poll` is invalid for
the built-in identity.
- Webhook mode requires no connection; a bound connection is allowed
(rewrite suggestions only).
- `source_config.webhook_provider` ∈ {`sonarr`, `radarr`, `auto`} when set.
- The installed-capability check passes for the built-in identity
automatically because the composite lister supplies it — verify with a
handler test rather than special-casing `scanSourceInstalled`.
- Event list responses: add `delivery_mode` and `provider_event_type`.
**`internal/api/router.go`**
- Admin routes: add the three webhook-management routes next to the existing
`/autoscan/sources` block.
- Public route: register `r.Post("/autoscan/webhooks/{token}", ...)` alongside
the other public routes (near the auth/discord-callback blocks), gated on
`autoscanHandler != nil`.
- Rate limiting: reuse `internal/ratelimit` — mirror the
`Middleware.AuthEndpointHandler(endpoint)` per-IP pattern with a
`"autoscan_webhook"` endpoint entry (config default on the generous side,
e.g. 60/min burst 30; arr bursts are legitimate). Per-token limiting beyond
that is unnecessary for v1 given the suppressor already collapses duplicate
work; note this in the PR.
Handler tests (`internal/api/handlers`): status-code matrix from the spec's
Testing section — 202 (ok / test / disabled / unknown event), 400, 404, 405
(chi handles), 413, 429; rotation invalidates the old token; no token/body in
logs (assert via a captured slog handler).
## Phase 7 — Frontend
- `web/src/api/types.ts`: extend `AutoscanSource` with the new response
fields; add webhook endpoint response type.
- `web/src/hooks/queries/useAutoscan.ts` (+ `keys.ts`): mutations for webhook
create/rotate/delete; invalidate the sources query on success.
- `web/src/pages/admin/autoscan/SourcesPanel.tsx`:
- Delivery mode segmented control (`Poll` / `Webhook`) in the source
create/edit form. Only offered when the picked capability is the built-in
webhook identity (which forces webhook mode) or a plugin capability
(which forces poll mode) — in practice the mode follows the picker choice;
render the control as the mode indicator rather than a free toggle.
- Webhook mode panel: provider select (`Auto`/`Sonarr`/`Radarr` →
`source_config.webhook_provider`), webhook URL display with copy button
(prepend `window.location.origin` when the API returns a relative path),
rotate action with confirm, last-received / last-error status line.
- Connection picker hidden in webhook mode (or shown as explicitly optional
for rewrite suggestions); poll-interval input hidden.
- Path-rewrite editor stays visible in both modes.
- `web/src/pages/admin/autoscan/ActivityPanel.tsx`: show delivery mode and
provider event type on event rows.
- Follow existing panel patterns; 2-space/double-quote Prettier config.
## Phase 8 — Verification
```bash
GOWORK=off go test ./internal/autoscan/... ./internal/api/handlers
GOWORK=off go test ./...
cd web && pnpm run lint && pnpm run format:check && pnpm run build
make lint
make verify-local-paths
```
Manual smoke (dev): create a webhook source, paste the URL into a Sonarr
Connect → Webhook (On Import + On Rename + On File Delete), use Sonarr's
"Test" button (expect 202 and `last_received_at` updating), import a file,
confirm a webhook event row and a file-scope scan appear in the Activity
panel.
## Risks / notes
- The `PollOnce` refactor is the riskiest step: marker-advancement semantics
are subtle and documented inline — preserve the decision comments and rely
on the existing `service_test.go` suite as the regression harness before
adding webhook paths.
- `secret_ref` uses `internal/secret.Cipher` with row-bound AAD; never rename
the table/column in SQL later without a `db_loader`-style fallback
(encryption is AAD-bound).
- Keep `/api/v1` additive: every API change above is a new field, new
endpoint, or new route; nothing existing is renamed or repurposed.
- Client repos: none required for v1 — this is an admin-web-only surface; the
public endpoint is consumed by arr, not by Silo clients.
@@ -0,0 +1,661 @@
# Autoscan Sonarr/Radarr Webhook Intake
**Status:** Reviewed 2026-07-09 — decisions locked (see "Resolved review decisions")
**Date:** 2026-07-09
**Area:** `internal/autoscan`, `internal/api`, `web/src/pages/AdminAutoscan.tsx`, `web/src/pages/admin/autoscan/`, `migrations/sql/`
Commands assume the repository root is the cwd.
## Summary
Silo's current Sonarr/Radarr Autoscan integration is pull-based. The installed
`silo.autoscan.arr` plugin implements `scan_source.v1`; the host polls it on an
interval, passes a resolved `{base_url, api_key}` connection, the plugin reads
arr `/api/v3/history`, and the host applies path rewrites, resolves scan
targets, suppresses duplicates, enqueues scans, and records poll events.
That design works, but it means a Sonarr/Radarr Autoscan source needs an arr API
key. This spec adds a **webhook delivery mode** so Sonarr/Radarr can POST import,
upgrade, rename, and delete notifications directly to Silo. Webhook mode should
not require an arr connection or API key. The host still owns all security,
path-rewrite, scan-resolution, scan-queue, suppression, and event-bookkeeping
behavior.
The recommended v1 is **host-native webhook intake attached to Autoscan sources**,
using a **host-discovered built-in source identity** (`silo.autoscan.arr-webhook`)
so webhook sources require neither an arr API key nor an installed plugin.
No SDK change is required for v1. The existing pull plugin remains available for
polling mode and for operators who prefer API-based history polling.
## Current state
- `autoscan_sources` binds a source to `plugin_id`, `capability_id`, an optional
`connection_id`, `enabled`, optional `poll_interval_seconds`, `path_rewrites`,
`source_config`, label, marker, last run time, and last error.
- `autoscan_connections` can store own credentials or a soft link to a Requests
integration. Keys are encrypted by `internal/secret`.
- `autoscan.Service.PollOnce` loads enabled sources, optionally resolves their
connection, calls the plugin `PollChanges`, applies rewrites, resolves scan
targets, suppresses duplicate work, enqueues scans, advances the marker on
success, and writes `autoscan_events`.
- `scan_source.v1` is pull only. The SDK has no "plugin received a webhook;
submit these paths to the host" RPC.
- Plugin HTTP routes exist and can be public, but they route by installation id
and are not currently connected to Autoscan enqueueing.
- The current ARR plugin has no config. It reads connection data only from
`PollChangesRequest.connection` and returns raw arr-side paths.
## Goals
- Let operators configure Sonarr/Radarr webhooks without storing an arr API key
in Silo.
- Keep polling mode intact and compatible with existing Autoscan sources.
- Reuse the existing host path rewrite, resolution, suppression, enqueue, and
event-history behavior for both poll and webhook deliveries.
- Give each webhook source a stable public callback URL with a rotatable secret.
- Support Sonarr/Radarr import, upgrade, rename, file-delete, and test events.
- Keep `/api/v1` additive: new fields and endpoints only.
- Avoid leaking webhook secrets or raw payloads in logs, events, or frontend
responses beyond admin-only setup surfaces.
## Non-goals
- No automatic creation of Sonarr/Radarr webhook connections inside arr. That
would require the arr API key this feature is trying to avoid.
- No SDK change for v1.
- No plugin-to-host push RPC for v1.
- No broad webhook framework for every future provider. This spec only defines
Sonarr/Radarr webhook intake.
- No webhook signing requirement from Sonarr/Radarr beyond the generated Silo
URL secret. If a future arr version supports request signing, add it
additively.
- No replacement of the poll plugin. Polling remains useful for fallback,
history recovery, and operators who prefer not to expose a public callback.
## Approach comparison
### Option A: Host-native webhook mode on Autoscan sources (recommended)
Silo adds a public secret-bearing endpoint such as
`POST /api/v1/autoscan/webhooks/{token}`. The endpoint resolves the token to an
Autoscan source, parses the Sonarr/Radarr payload in the host, turns it into
`autoscan.Change` values, and calls a new host-side `IngestChanges` path that
shares the same rewrite/resolve/suppress/enqueue/event logic as `PollOnce`.
**Pros**
- No SDK release or plugin release needed for v1.
- No arr API key needed for webhook-only sources.
- Stable callback URL is source-owned, not plugin-installation-id-owned.
- Host has direct access to scan queue, resolver, suppressor, DB, event logs,
secret encryption, and rate limiting.
- Existing Autoscan activity UI and source rows remain the operator surface.
- The security boundary is simpler: one host-owned bearer secret, one host-owned
parser, one host-owned enqueue path.
**Cons**
- Sonarr/Radarr payload parsing becomes host code, so provider-specific logic is
no longer purely in plugins.
- Future non-arr webhook providers would need host work unless a later generic
push contract is added.
A variant of Option A would reuse the `silo.autoscan.arr` plugin identity for
webhook sources. That was considered and rejected: `HandleCreateSource` hard
rejects any source whose `(plugin_id, capability_id)` is not a
currently-installed capability, so plugin-bound identity would force operators
to install a plugin the webhook path never calls — quietly defeating the
feature's own pitch. The built-in identity below avoids that at near-zero cost
(see "Recommended v1").
### Option B: SDK push RPC plus plugin HTTP route
Add an SDK/runtime-host RPC such as `SubmitScanSourceChanges`, update
`silo-server` to implement it, update the ARR plugin to declare `http_routes.v1`
and receive public webhook requests, then have the plugin parse payloads and
push changes to the host.
**Pros**
- Provider-specific webhook parsing lives in the provider plugin.
- The host can stay more generic once the push RPC exists.
- Future push-style providers can reuse the same plugin-to-host contract.
**Cons**
- Requires a coordinated SDK release, server update, plugin update, and catalog
update before the feature exists.
- Plugin public URLs include installation id, so reinstalling or replacing the
plugin can change callback URLs unless the host adds another stable alias.
- The plugin still needs durable webhook secret/config state or host-provided
secret state.
- More moving parts in the most security-sensitive path.
- The feature cannot ship as a narrow server/API/UI change.
### Option C: Separate `arr-webhook` plugin with internal buffer
Create a new plugin that exposes a public webhook route, stores received paths
in an internal buffer, and returns them when the host later polls
`scan_source.v1`.
**Pros**
- No SDK push RPC required.
- Provider-specific parsing stays outside the host.
- Reuses the existing pull contract.
**Cons**
- Still has poll latency even though arr sent a real-time webhook.
- The plugin needs durable queue/buffer storage, idempotency, and cleanup. The
current plugin model is not designed around plugin-owned durable event queues.
- Reinstalling the plugin risks losing buffered events or changing callback URLs.
- More operationally fragile than letting the host enqueue immediately.
- Creates a second ARR Autoscan plugin with overlapping purpose.
### Option D: Keep polling only
Improve docs or setup around current polling, but still require an arr API key.
**Pros**
- No implementation work.
- Pulling `/history` remains robust for recovery after Silo downtime.
**Cons**
- Does not satisfy the request.
- Still requires API keys.
- Adds latency compared to direct arr webhook delivery.
## Recommended v1
Implement Option A with a built-in source identity.
Add webhook delivery as a mode on Autoscan source rows. Existing poll sources
remain `delivery_mode = "poll"`. Webhook-only sources use
`delivery_mode = "webhook"` and do not require `connection_id`.
Webhook sources bind to a **host-discovered built-in source identity**,
`silo.autoscan.arr-webhook`, rather than the installed ARR plugin's capability.
`ScanSourceLister` is a narrow interface with a single adapter
(`PluginScanSourceLister` in `internal/api/autoscan_wiring.go`); wrap it in a
composite lister that appends one synthetic `DiscoveredSource` entry for the
built-in identity. Because the Add-source picker, create-time validation,
labels, and events all consume only `(plugin_id, capability_id, display_name)`,
they work unchanged. The provider is never invoked for webhook sources —
`PollOnce` skips them — so no `PollChanges` implementation exists for the
built-in identity, and none is needed.
For v1, `delivery_mode = "webhook"` is accepted only for the built-in
`silo.autoscan.arr-webhook` identity, and `delivery_mode = "poll"` is rejected
for it (it has no plugin to poll). Existing ARR plugin sources stay poll-mode.
## Data model
Add a Goose migration.
### `autoscan_sources`
Add:
| column | type / default | purpose |
|---|---|---|
| `delivery_mode` | text NOT NULL DEFAULT `'poll'` | `poll` or `webhook`; future `hybrid` can be added later |
Check constraint:
```sql
delivery_mode = ANY (ARRAY['poll'::text, 'webhook'::text])
```
Rules:
- Poll mode keeps today's behavior.
- Webhook mode skips `PollChanges` and does not require a connection.
- Webhook mode is valid only for the built-in `silo.autoscan.arr-webhook`
identity; poll mode is invalid for it.
- Existing rows default to poll.
- `source_config.webhook_provider` may be `sonarr`, `radarr`, or `auto`.
Use `auto` by default and infer from payload shape when possible.
### `autoscan_webhook_endpoints`
One active webhook endpoint per source for v1.
| column | type / default | purpose |
|---|---|---|
| `source_id` | uuid PK REFERENCES `autoscan_sources(id)` ON DELETE CASCADE | webhook owner |
| `secret_hash` | text UNIQUE NOT NULL | lookup hash of the bearer URL token |
| `secret_ref` | text NOT NULL | encrypted token, admin-display only |
| `secret_suffix` | text NOT NULL | last 6-8 chars for UI/debug |
| `created_at` | timestamptz NOT NULL DEFAULT now() | creation time |
| `rotated_at` | timestamptz | last token rotation |
| `last_received_at` | timestamptz | latest valid delivery |
| `last_error_at` | timestamptz | latest parse/ingest error |
| `last_error_message` | text NOT NULL DEFAULT '' | bounded sanitized error |
Secret handling:
- Generate at least 32 random bytes and encode URL-safe.
- Store a plain SHA-256 hash for lookup. For a 32-byte random token this is
standard and sufficient (preimage resistance is what matters); an HMAC-keyed
variant adds key-management complexity for no practical gain here.
- Store encrypted `secret_ref` using `internal/secret.Cipher` and AAD bound to
`autoscan_webhook_endpoints.source_id`.
- Never log the token or full URL.
### `autoscan_webhook_deliveries`
Persist each actionable parsed delivery before returning `202`. Rows store the
source, provider event type, normalized changes, original receive time, retry
attempt count/backoff, and a bounded lease owner/time. The request attempts the
delivery immediately; transient resolver/database/queue failures release it for
an internal retry task. Workers claim due rows with `FOR UPDATE SKIP LOCKED`, so
multiple API nodes can drain the queue without double-owning a delivery. A
successful consume deletes the row; deleting the source cascades its pending
deliveries.
## API
All admin endpoints are additive under `/api/v1/admin/autoscan`.
### Source response additions
Add fields to `AutoscanSource` responses:
```json
{
"delivery_mode": "poll",
"webhook_configured": true,
"webhook_url": "https://example/api/v1/autoscan/webhooks/...",
"webhook_secret_suffix": "abc123",
"webhook_last_received_at": "2026-07-09T12:34:56Z",
"webhook_last_error_at": null,
"webhook_last_error_message": ""
}
```
`webhook_url` is admin-only and stays redisplayable in admin source responses
(the encrypted `secret_ref` exists for exactly this). The token's blast radius
is small — the worst an attacker can do with it is trigger scans of paths that
already resolve inside Silo libraries — so one-time display would buy little
security while costing real operator pain (re-pasting into arr after a config
wipe would force a rotation).
### Source write additions
Extend create/update payloads:
```json
{
"delivery_mode": "webhook",
"source_config": {
"webhook_provider": "sonarr"
}
}
```
Validation:
- `delivery_mode = "poll"`: current validation stays. If enabled, a connection
should be bound for ARR poll sources.
- `delivery_mode = "webhook"`: only valid with the built-in
`silo.autoscan.arr-webhook` identity. No connection is required. A bound
connection is allowed only for rewrite suggestions.
- The installed-capability check treats the built-in identity as always
available (it comes from the composite lister, not plugin installations).
- `source_config.webhook_provider`: optional; when present must be `sonarr`,
`radarr`, or `auto`.
### Webhook endpoint management
Add:
- `POST /admin/autoscan/sources/{id}/webhook`
- Creates a webhook endpoint if missing and returns URL/status.
- `POST /admin/autoscan/sources/{id}/webhook/rotate`
- Replaces the token, invalidates the old URL, returns the new URL.
- `DELETE /admin/autoscan/sources/{id}/webhook`
- Deletes the endpoint and makes future deliveries return not found.
### Public delivery endpoint
Add:
```http
POST /api/v1/autoscan/webhooks/{token}
Content-Type: application/json
```
Responses:
- `202 Accepted`: valid token; event accepted, ignored, or enqueued. Use 202 for
valid test events and valid disabled sources so arr setup tests do not fail
noisily.
- `400 Bad Request`: valid token but malformed/unsupported payload.
- `404 Not Found`: token not found. Do not reveal whether a source exists.
- `405 Method Not Allowed`: non-POST.
- `413 Payload Too Large`: body exceeds the configured cap.
- `429 Too Many Requests`: per-token or per-IP rate limit exceeded.
- `500 Internal Server Error`: the delivery could not be durably persisted.
Sonarr/Radarr do not replay failed notification events. Silo must therefore
persist actionable deliveries before `202` and retry ingest failures internally;
HTTP error responses are not a delivery retry mechanism.
The body should be small; cap at 256 KiB unless real payload fixtures prove that
is too low.
## Host service design
Factor the existing `PollOnce` consume path so polling and webhooks share it.
Proposed shape:
```go
type ChangeIngest struct {
SourceID string
DeliveryMode string // "poll" or "webhook"
ProviderEventType string
Changes []autoscan.Change
ReceivedAt time.Time
}
func (s *Service) IngestChanges(ctx context.Context, input ChangeIngest) (IngestResult, error)
```
Internally, introduce an unexported helper that both `PollOnce` and
`IngestChanges` call after changes are known:
```go
func (s *Service) consumeSourceChanges(ctx context.Context, src Source, changes []Change, opts consumeOptions) (consumeResult, error)
```
It should reuse:
- `rewriteChanges`
- `resolveAndClaim`
- `collapseTargetsToLibraryScans`
- `enqueueScanTargets`
- `releaseClaims`
- event creation/finish logic
Differences from polling:
- No marker is advanced for webhook deliveries.
- Webhook events use marker fields as empty/null.
- Webhook event creation skips the running-event single-flight exclusion (see
"Security and reliability") — concurrent deliveries each ingest; the
suppressor claim is the duplicate guard.
- `changes_returned`, `changes_resolved`, `targets_claimed`,
`scans_created`, `scans_reused`, and `scans_suppressed` should be recorded
exactly like poll events.
- If any resolve attempt fails transiently, return an error and record the
event as error. Already-enqueued targets are allowed; a duplicate later
delivery is safe.
- If all paths are outside Silo folders, finish as `unresolved`, same as poll.
`PollOnce` changes:
- Skip sources with `delivery_mode = "webhook"`.
- Continue current behavior for `delivery_mode = "poll"`.
- If a future `hybrid` mode is added, poll only when a connection is bound.
## Payload parsing
Implement parsing in a small ARR-specific host package, for example
`internal/autoscan/arrwebhook`.
Input:
```go
func Parse(provider string, body []byte) (ParsedWebhook, error)
```
Output:
```go
type ParsedWebhook struct {
Provider string
EventType string
Test bool
Changes []autoscan.Change
}
```
Provider inference:
- If `source_config.webhook_provider` is `sonarr` or `radarr`, use it.
- If `auto`, infer from top-level `series` vs `movie`, or from top-level field
prefixes if supporting flattened payloads.
- If inference fails on a non-test event, return a 400 with a sanitized message.
Events:
- `Test`: no-op success. Update `last_received_at`; do not enqueue.
- `Download`, `Import`, `Upgrade`, `DownloadComplete`: enqueue imported file
paths as `ChangeScopeFile`.
- `Rename`: enqueue new and previous file paths as `ChangeScopeFile`. Previous
paths may no longer exist; `resolveChange` already falls back through
`ResolveVanishedPath` for file changes.
- `EpisodeFileDelete`, `MovieFileDelete`: enqueue deleted file path as
`ChangeScopeFile` when present.
- Unknown event types: valid token, unsupported event. Return 202 no-op and
record a low-noise status rather than making arr mark the webhook unhealthy.
Path extraction should support the common webhook JSON shapes and should be
covered by fixtures:
- Sonarr:
- `episodeFile.path`
- `episodeFiles[].path`
- `episodeFile.previousPath`, `episodeFiles[].previousPath`
- `series.path` as a subtree fallback only when no file path exists
- Radarr:
- `movieFile.path`
- `movieFiles[].path`
- `movieFile.previousPath`, `movieFiles[].previousPath`
- `movie.path` or `movie.folderPath` as a subtree fallback only when no file
path exists
The parser should dedupe exact paths before returning changes.
## Security and reliability
- Public endpoint is unauthenticated but token-bearing.
- Token lookup should be constant-shape: invalid tokens return 404 without
distinguishing disabled/deleted sources.
- Valid token but disabled source/global Autoscan disabled should return 202
and perform no enqueue. This prevents arr retries or setup failures while
respecting Silo's disabled state.
- Apply a per-token rate limit and a coarse per-IP limit. Build on the existing
`internal/ratelimit` middleware (it already supports per-endpoint and per-IP
limits) rather than introducing a new mechanism.
- Use `http.MaxBytesReader` or equivalent for body cap.
- Do not log request bodies. If an error needs context, log only source id,
event type, provider, path count, and sanitized bounded error text.
- Never include token in event logs or regular app logs.
- Preserve current duplicate suppression. Duplicate webhook deliveries should
at worst create suppressed Autoscan events, not duplicate scan storms.
- Honor configured burst capacity in both in-memory and Redis rate-limit
backends; season-pack imports commonly deliver more than one event per second.
- Keep transient ingest failures in the durable delivery queue with bounded
exponential backoff. A short-interval hidden task reclaims due or abandoned
leases and retries them independently of the poll cadence.
- **Webhook deliveries must NOT be single-flighted and dropped.** The
running-event exclusion in `CreateEvent` (advisory lock + running-row check)
exists to stop overlapping polls of the same marker window. Poll cycles can
safely be skipped because the marker window is re-read next cycle; a webhook
delivery has no marker, so a dropped delivery is lost forever. Sonarr fires
one webhook per import in rapid succession (a season pack is many imports in
seconds), so "another event is running" is the normal case during a burst.
`CreateEvent` takes a flag (or a separate path) that skips the running-event
exclusion for webhook deliveries; the duplicate-scan guard for concurrent
ingests is the suppressor's atomic `ShouldScan` claim, which webhook
ingestion reuses. Poll-mode behavior is unchanged. If a `hybrid` mode ships
later, the rule is "poll defers to webhook", never "webhook drops".
## Admin UI
Extend `AdminAutoscan` / `SourcesPanel`.
Source row/edit controls:
- Delivery mode segmented control: `Poll` / `Webhook`.
- Poll mode:
- Existing connection picker, poll interval, rewrites, sync suggestions.
- Webhook mode:
- Provider selector: `Auto`, `Sonarr`, `Radarr`.
- Webhook URL display with copy button.
- Rotate URL action.
- Last received / last error status.
- Connection picker hidden or clearly optional. If a connection is bound, it
is used only for existing rewrite suggestions, not for ingestion.
- Keep path rewrites visible in both modes; webhook payload paths are arr-side
paths and still need host-side rewrites in Docker/mount-mismatch setups.
Connections panel:
- No required changes, except copy should not imply a connection is mandatory
for webhook-only sources.
Add-source picker:
- The built-in `silo.autoscan.arr-webhook` entry appears alongside installed
plugin capabilities (supplied by the composite lister) with a display name
like "Sonarr/Radarr Webhook". Picking it creates a webhook-mode source.
Activity panel:
- Show delivery mode and provider event type on event rows (the schema adds
both fields in v1).
## Event metadata (required in v1)
Add fields to `autoscan_events`:
| column | type / default | purpose |
|---|---|---|
| `delivery_mode` | text NOT NULL DEFAULT `'poll'` | poll/webhook event source |
| `provider_event_type` | text NOT NULL DEFAULT '' | arr event type |
These ship in v1, in the same migration as the source/endpoint changes. They
are not required for the enqueue path, but debugging webhook intake without
knowing which delivery produced an event is exactly the pain this feature hits
in week one, and the cost is two columns in a migration being written anyway.
## Rollout
1. Add migrations for source delivery mode, webhook endpoint table, and
`autoscan_events` delivery-mode/provider-event-type columns.
2. Add the composite `ScanSourceLister` exposing the built-in
`silo.autoscan.arr-webhook` identity.
3. Add repository methods for webhook endpoint create/get-by-hash/rotate/delete
and source delivery-mode fields, plus the `CreateEvent` no-exclusion path
for webhook deliveries.
4. Refactor `autoscan.Service` to share the consume path and add
`IngestChanges`.
5. Add ARR webhook parser fixtures.
6. Add public route and handler.
7. Add admin endpoint fields and endpoint-management routes.
8. Update frontend types/hooks and `SourcesPanel`.
9. Keep all existing source rows as poll mode.
10. Update docs or setup text after review.
## Testing
Backend:
- Migration applies and rolls back.
- Existing poll-mode service tests still pass.
- `PollOnce` skips webhook-mode sources.
- The composite lister exposes the built-in identity; source creation accepts
it for webhook mode and rejects poll mode for it.
- `IngestChanges` applies rewrites and enqueues the same targets as polling for
equivalent changes.
- Actionable deliveries are persisted before processing; transient failures
return `202`, remain queued, and are consumed by the retry task.
- Delivery leases are ownership-guarded and reclaimable after worker failure.
- Concurrent/burst webhook deliveries for one source all ingest — none are
dropped by the running-event exclusion; duplicate targets are suppressed by
the suppressor claim.
- Duplicate webhook deliveries are suppressed by existing debounce logic.
- Webhook events record `delivery_mode` and `provider_event_type`.
- Disabled source and disabled global setting return accepted/no-op behavior.
- Invalid token returns 404; malformed payload for valid token returns 400.
- Body cap returns 413.
- Rate limit returns 429.
- Redis rate limiting honors the configured webhook burst allowance.
- Upgrade payloads reconcile both the new file and arr's `deletedFiles` paths.
- Token rotation invalidates old token and accepts new token.
- Parser fixture tests for Sonarr Download, Rename, EpisodeFileDelete, Test.
- Parser fixture tests for Radarr Download, Rename, MovieFileDelete, Test.
- Unknown event type is accepted/no-op.
- No logs include the token or raw request body.
Frontend:
- Source can switch from poll to webhook mode without selecting a connection.
- Webhook URL renders only for configured endpoint.
- Rotate action replaces the URL.
- Provider selector writes `source_config.webhook_provider`.
- Poll mode behavior remains unchanged.
- Rewrite editor remains available in webhook mode.
Validation commands:
```bash
GOWORK=off go test ./internal/autoscan ./internal/api/handlers
cd web && pnpm run lint
cd web && pnpm run format:check
make verify-local-paths
```
Broader validation after implementation:
```bash
GOWORK=off go test ./...
cd web && pnpm run build
```
## Resolved review decisions (2026-07-09)
1. **Built-in source identity.** Webhook mode uses a host-discovered
`silo.autoscan.arr-webhook` identity, not the installed ARR plugin's
capability. Create-time validation requires an installed capability, so
plugin-bound identity would force installing a plugin the webhook path never
calls; the composite-lister alternative is ~20 lines and everything
downstream consumes only `(plugin_id, capability_id, display_name)`.
2. **Webhook URL stays redisplayable** in admin-only responses. Low token blast
radius (worst case: spurious scans of paths already inside Silo libraries);
one-time display would force rotations after any config loss.
3. **No `hybrid` in v1.** Purely additive later; shipping it now forces
poll/webhook overlap semantics to be solved before any field experience.
4. **Unknown event types return 202 no-op.** Arr adds event types over time; a
400 makes arr mark the webhook unhealthy and nag the operator about
something Silo intentionally ignores.
5. **`autoscan_events.delivery_mode` and `provider_event_type` ship in v1.**
Two columns in a migration being written anyway; needed for week-one
debugging of webhook intake.
6. **Parser lives in `internal/autoscan/arrwebhook`.** Fixture-testable, keeps
handlers thin, matches the repo's domain-package organization.
7. **Webhook deliveries are never single-flight-dropped** (raised during
review, not an original question). Poll skips are safe because the marker
window is re-read; webhooks have no marker, and Sonarr bursts (season packs)
make a running event the normal case. See "Security and reliability".
## References
- Current Autoscan plugin architecture:
`docs/superpowers/specs/2026-06-02-autoscan-plugin-architecture-design.md`
- Current ARR polling design:
`docs/superpowers/specs/2026-06-02-autoscan-arr-polling-design.md`
- Current rewrite sync design:
`docs/superpowers/specs/2026-06-02-autoscan-rewrite-sync-design.md`
- Current SDK pull contract: sibling `silo-plugin-sdk` repo,
`proto/silo/plugin/v1/scan_source.proto`
- Current ARR plugin: sibling `silo-plugin-autoscan-arr` repo
- Servarr path/event references:
- Sonarr custom scripts: <https://wiki.servarr.com/sonarr/custom-scripts>
- Radarr custom scripts: <https://wiki.servarr.com/radarr/custom-scripts>
+28
View File
@@ -119,6 +119,7 @@ func NewMiddleware(w Writer, nodeID string) func(http.Handler) http.Handler {
pathPattern = route
}
}
path = RedactSecretPathParams(r, path)
// After handler chain completes, auth middleware has populated lc
entry := LogEntry{
@@ -143,6 +144,33 @@ func NewMiddleware(w Writer, nodeID string) func(http.Handler) http.Handler {
}
}
// RedactSecretPathParams strips bearer credentials from a request path before
// it reaches any log sink. Public webhook endpoints authenticate via a secret
// path segment (chi route params named "token" or "secret" — autoscan webhook
// intake, webhook-sync); logging the raw path would persist the credential in
// app logs and the activity table. Must be called after the handler chain ran
// so the chi route context is populated.
func RedactSecretPathParams(r *http.Request, path string) string {
routeCtx := chi.RouteContext(r.Context())
if routeCtx == nil {
return path
}
for i, key := range routeCtx.URLParams.Keys {
if key != "token" && key != "secret" {
continue
}
if i >= len(routeCtx.URLParams.Values) {
continue
}
value := routeCtx.URLParams.Values[i]
if value == "" {
continue
}
path = strings.Replace(path, "/"+value, "/[redacted]", 1)
}
return path
}
// isStreamChunk returns true if the path looks like a stream segment/manifest
// chunk rather than a session-start request.
func isStreamChunk(path string) bool {
+58
View File
@@ -0,0 +1,58 @@
package activitylog
import (
"context"
"net/http/httptest"
"testing"
"github.com/go-chi/chi/v5"
)
func TestRedactSecretPathParams(t *testing.T) {
cases := []struct {
name string
params map[string]string
path string
want string
}{
{
name: "autoscan webhook token",
params: map[string]string{"token": "2riBxLS4UcC8w8WdRoge8F0f8LOLkz4yPsPNA01gWBY"},
path: "/api/v1/autoscan/webhooks/2riBxLS4UcC8w8WdRoge8F0f8LOLkz4yPsPNA01gWBY",
want: "/api/v1/autoscan/webhooks/[redacted]",
},
{
name: "webhook-sync secret",
params: map[string]string{"secret": "c98bffdbe95b5be34141e2a7f0327e1f"},
path: "/api/v1/webhook-sync/webhooks/c98bffdbe95b5be34141e2a7f0327e1f",
want: "/api/v1/webhook-sync/webhooks/[redacted]",
},
{
name: "non-secret params untouched",
params: map[string]string{"id": "abc123"},
path: "/api/v1/admin/autoscan/sources/abc123",
want: "/api/v1/admin/autoscan/sources/abc123",
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
r := httptest.NewRequest("POST", tc.path, nil)
routeCtx := chi.NewRouteContext()
for k, v := range tc.params {
routeCtx.URLParams.Add(k, v)
}
r = r.WithContext(context.WithValue(r.Context(), chi.RouteCtxKey, routeCtx))
if got := RedactSecretPathParams(r, tc.path); got != tc.want {
t.Fatalf("redacted path = %q, want %q", got, tc.want)
}
})
}
}
func TestRedactSecretPathParamsNoRouteContext(t *testing.T) {
r := httptest.NewRequest("POST", "/api/v1/autoscan/webhooks/tok", nil)
if got := RedactSecretPathParams(r, r.URL.Path); got != r.URL.Path {
t.Fatalf("path without route context must pass through, got %q", got)
}
}
+4 -1
View File
@@ -149,7 +149,10 @@ func BuildAutoscanService(
scantrigger.NewResolver(folderRepo),
queue,
autoscan.NewRedisSuppressor(redisClient),
PluginScanSourceLister{installationStore},
autoscan.WithBuiltinSources(
PluginScanSourceLister{installationStore},
autoscan.BuiltinArrWebhookSource(),
),
)
// Wire the connection-test + rewrite-suggester deps: a (long-timeout)
// arr root-folder/status client and a Silo media-folder lister.
+200 -18
View File
@@ -39,6 +39,15 @@ type autoscanStore interface {
ListRunningEvents(ctx context.Context) ([]autoscan.Event, error)
GetQueueSummary(ctx context.Context) (autoscan.QueueSummary, error)
LatestEventAt(ctx context.Context) (*time.Time, error)
CreateWebhookEndpoint(ctx context.Context, sourceID string) (autoscan.WebhookEndpoint, string, error)
RotateWebhookEndpoint(ctx context.Context, sourceID string) (autoscan.WebhookEndpoint, string, error)
DeleteWebhookEndpoint(ctx context.Context, sourceID string) error
GetWebhookEndpoint(ctx context.Context, sourceID string) (autoscan.WebhookEndpoint, error)
ListWebhookEndpoints(ctx context.Context) ([]autoscan.WebhookEndpoint, error)
RevealWebhookToken(ctx context.Context, sourceID string) (string, error)
ResolveWebhookToken(ctx context.Context, token string) (autoscan.Source, autoscan.WebhookEndpoint, error)
TouchWebhookReceived(ctx context.Context, sourceID string) error
RecordWebhookError(ctx context.Context, sourceID, msg string) error
}
// autoscanTriggerer is the subset of *autoscan.Service the handler needs.
@@ -55,6 +64,9 @@ type autoscanTriggerer interface {
// SuggestRewrites proposes path rewrites by matching the source's arr root
// folders against Silo's media folders.
SuggestRewrites(ctx context.Context, sourceID string) (autoscan.RewriteSuggestions, error)
// IngestChanges feeds webhook-delivered changes through the shared
// rewrite/resolve/suppress/enqueue pipeline.
IngestChanges(ctx context.Context, in autoscan.ChangeIngest) (autoscan.IngestResult, error)
}
// autoscanTriggerUpdater reconfigures the poll task's schedule when the default
@@ -69,11 +81,14 @@ type autoscanTriggerUpdater interface {
const autoscanPollTaskKey = "autoscan_poll"
// AutoscanHandler serves the admin-only autoscan settings/connections/sources/
// trigger API.
// trigger API plus the public webhook delivery endpoint.
type AutoscanHandler struct {
repo autoscanStore
svc autoscanTriggerer
triggers autoscanTriggerUpdater // optional; nil skips rescheduling
// publicURL is the server's externally reachable base URL; when empty,
// webhook URLs are returned as absolute paths for the UI to complete.
publicURL string
}
func NewAutoscanHandler(repo autoscanStore, svc autoscanTriggerer) *AutoscanHandler {
@@ -88,6 +103,13 @@ func (h *AutoscanHandler) SetTriggerUpdater(t autoscanTriggerUpdater) {
h.triggers = t
}
// SetPublicURL wires the server's public base URL so webhook URLs are returned
// fully qualified. Optional: when unset, responses carry the absolute path and
// the admin UI prefixes its own origin.
func (h *AutoscanHandler) SetPublicURL(publicURL string) {
h.publicURL = strings.TrimRight(strings.TrimSpace(publicURL), "/")
}
// --- Settings ---
type autoscanSettingsResponse struct {
@@ -307,19 +329,29 @@ func (h *AutoscanHandler) HandleDeleteConnection(w http.ResponseWriter, r *http.
// --- Sources ---
// autoscanSourceResponse is the client view of a source. It carries no
// credentials: the connection link is by id only.
// credentials: the connection link is by id only. The webhook_* fields are
// admin-only setup surface; webhook_url deliberately stays redisplayable (the
// token's blast radius is spurious scans of already-registered library paths,
// and one-time display would force a rotation after any config loss).
type autoscanSourceResponse struct {
ID string `json:"id"`
PluginID string `json:"plugin_id"`
CapabilityID string `json:"capability_id"`
ConnectionID *string `json:"connection_id"`
Enabled bool `json:"enabled"`
PollIntervalSeconds *int `json:"poll_interval_seconds,omitempty"`
PathRewrites []autoscan.PathRewrite `json:"path_rewrites"`
SourceConfig map[string]string `json:"source_config"`
Label string `json:"label"`
LastRunAt *time.Time `json:"last_run_at,omitempty"`
LastError *string `json:"last_error,omitempty"`
ID string `json:"id"`
PluginID string `json:"plugin_id"`
CapabilityID string `json:"capability_id"`
ConnectionID *string `json:"connection_id"`
Enabled bool `json:"enabled"`
DeliveryMode string `json:"delivery_mode"`
PollIntervalSeconds *int `json:"poll_interval_seconds,omitempty"`
PathRewrites []autoscan.PathRewrite `json:"path_rewrites"`
SourceConfig map[string]string `json:"source_config"`
Label string `json:"label"`
LastRunAt *time.Time `json:"last_run_at,omitempty"`
LastError *string `json:"last_error,omitempty"`
WebhookConfigured bool `json:"webhook_configured"`
WebhookURL string `json:"webhook_url,omitempty"`
WebhookSecretSuffix string `json:"webhook_secret_suffix,omitempty"`
WebhookLastReceivedAt *time.Time `json:"webhook_last_received_at,omitempty"`
WebhookLastErrorAt *time.Time `json:"webhook_last_error_at,omitempty"`
WebhookLastErrorMessage string `json:"webhook_last_error_message,omitempty"`
}
func sourceResponse(s autoscan.Source) autoscanSourceResponse {
@@ -330,12 +362,17 @@ func sourceResponse(s autoscan.Source) autoscanSourceResponse {
rewrites = []autoscan.PathRewrite{}
}
config := normalizeSourceConfig(s.SourceConfig)
deliveryMode := s.DeliveryMode
if deliveryMode == "" {
deliveryMode = autoscan.DeliveryModePoll
}
return autoscanSourceResponse{
ID: s.ID,
PluginID: s.PluginID,
CapabilityID: s.CapabilityID,
ConnectionID: s.ConnectionID,
Enabled: s.Enabled,
DeliveryMode: deliveryMode,
PollIntervalSeconds: s.PollIntervalSeconds,
PathRewrites: rewrites,
SourceConfig: config,
@@ -345,6 +382,45 @@ func sourceResponse(s autoscan.Source) autoscanSourceResponse {
}
}
// webhookURLFor builds the delivery URL for a token: fully qualified when the
// server knows its public URL, otherwise an absolute path the admin UI
// completes with its own origin.
func (h *AutoscanHandler) webhookURLFor(token string) string {
return h.publicURL + "/api/v1/autoscan/webhooks/" + token
}
// attachWebhookState decorates a source response with its endpoint status and
// redisplayable URL. A reveal failure (e.g. cipher/key drift) degrades to
// status-without-URL rather than failing the response.
func (h *AutoscanHandler) attachWebhookState(ctx context.Context, resp *autoscanSourceResponse, endpoint autoscan.WebhookEndpoint) {
resp.WebhookConfigured = true
resp.WebhookSecretSuffix = endpoint.SecretSuffix
resp.WebhookLastReceivedAt = endpoint.LastReceivedAt
resp.WebhookLastErrorAt = endpoint.LastErrorAt
resp.WebhookLastErrorMessage = endpoint.LastErrorMessage
token, err := h.repo.RevealWebhookToken(ctx, endpoint.SourceID)
if err != nil {
slog.WarnContext(ctx, "autoscan: reveal webhook token failed", "component", "api", "source_id", endpoint.SourceID, "err", err)
return
}
resp.WebhookURL = h.webhookURLFor(token)
}
// sourceResponseWithWebhook builds a single source response including webhook
// endpoint state (used by the single-source create/update/webhook handlers).
func (h *AutoscanHandler) sourceResponseWithWebhook(ctx context.Context, s autoscan.Source) autoscanSourceResponse {
resp := sourceResponse(s)
endpoint, err := h.repo.GetWebhookEndpoint(ctx, s.ID)
if err != nil {
if !errors.Is(err, autoscan.ErrNotFound) {
slog.WarnContext(ctx, "autoscan: load webhook endpoint failed", "component", "api", "source_id", s.ID, "err", err)
}
return resp
}
h.attachWebhookState(ctx, &resp, endpoint)
return resp
}
func (h *AutoscanHandler) HandleListSources(w http.ResponseWriter, r *http.Request) {
// Sources are operator-created (no auto-seed): just list what exists. Use
// GET /scan-source-plugins for the Add-source picker of installed capabilities.
@@ -353,9 +429,23 @@ func (h *AutoscanHandler) HandleListSources(w http.ResponseWriter, r *http.Reque
writeAutoscanError(w, err)
return
}
// One batched endpoint query for the whole listing (not per source).
endpoints, err := h.repo.ListWebhookEndpoints(r.Context())
if err != nil {
writeAutoscanError(w, err)
return
}
endpointBySource := make(map[string]autoscan.WebhookEndpoint, len(endpoints))
for _, e := range endpoints {
endpointBySource[e.SourceID] = e
}
out := make([]autoscanSourceResponse, 0, len(sources))
for _, s := range sources {
out = append(out, sourceResponse(s))
resp := sourceResponse(s)
if endpoint, ok := endpointBySource[s.ID]; ok {
h.attachWebhookState(r.Context(), &resp, endpoint)
}
out = append(out, resp)
}
writeJSON(w, http.StatusOK, struct {
Sources []autoscanSourceResponse `json:"sources"`
@@ -402,12 +492,55 @@ type autoscanCreateSourceInput struct {
CapabilityID string `json:"capability_id"`
ConnectionID *string `json:"connection_id"`
Enabled bool `json:"enabled"`
DeliveryMode string `json:"delivery_mode"`
PollIntervalSeconds *int `json:"poll_interval_seconds"`
PathRewrites []autoscan.PathRewrite `json:"path_rewrites"`
SourceConfig map[string]string `json:"source_config"`
Label string `json:"label"`
}
// resolveDeliveryMode validates a requested delivery mode against the source
// identity: webhook mode exists only for the built-in ARR webhook identity
// (which cannot poll — it has no plugin), and an empty mode defaults to
// whichever mode the identity supports.
func resolveDeliveryMode(requested, pluginID, capabilityID string) (string, error) {
isBuiltinWebhook := autoscan.IsBuiltinArrWebhookIdentity(pluginID, capabilityID)
switch strings.ToLower(strings.TrimSpace(requested)) {
case "":
if isBuiltinWebhook {
return autoscan.DeliveryModeWebhook, nil
}
return autoscan.DeliveryModePoll, nil
case autoscan.DeliveryModePoll:
if isBuiltinWebhook {
return "", fmt.Errorf("the built-in Sonarr/Radarr webhook source has no plugin to poll; delivery_mode must be webhook")
}
return autoscan.DeliveryModePoll, nil
case autoscan.DeliveryModeWebhook:
if !isBuiltinWebhook {
return "", fmt.Errorf("delivery_mode webhook is only valid for the built-in Sonarr/Radarr webhook source (%s/%s)",
autoscan.BuiltinArrWebhookPluginID, autoscan.BuiltinArrWebhookCapabilityID)
}
return autoscan.DeliveryModeWebhook, nil
default:
return "", fmt.Errorf("delivery_mode must be poll or webhook")
}
}
// validateWebhookProvider checks the optional source_config.webhook_provider
// value used by webhook payload parsing.
func validateWebhookProvider(config map[string]string) error {
provider, ok := config["webhook_provider"]
if !ok {
return nil
}
switch strings.ToLower(strings.TrimSpace(provider)) {
case "", "auto", "sonarr", "radarr":
return nil
}
return fmt.Errorf("source_config.webhook_provider must be auto, sonarr, or radarr")
}
// HandleCreateSource creates a new source bound to an installed scan_source
// capability. Many sources may share one capability (e.g. one Sonarr plugin
// fronting several arr servers, one source per connection). The capability must
@@ -436,9 +569,25 @@ func (h *AutoscanHandler) HandleCreateSource(w http.ResponseWriter, r *http.Requ
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
return
}
deliveryMode, err := resolveDeliveryMode(in.DeliveryMode, pluginID, capID)
if err != nil {
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
return
}
sourceConfig := normalizeSourceConfig(in.SourceConfig)
if deliveryMode == autoscan.DeliveryModeWebhook {
if err := validateWebhookProvider(sourceConfig); err != nil {
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
return
}
if provider, ok := sourceConfig["webhook_provider"]; ok {
sourceConfig["webhook_provider"] = strings.ToLower(strings.TrimSpace(provider))
}
}
// Validate the (plugin_id, capability_id) is a currently-installed
// scan_source capability — a source may only be created against an installed
// plugin capability.
// plugin capability. The built-in webhook identity passes because the
// composite lister always supplies it.
available, err := h.svc.ListAvailableScanSources(r.Context())
if err != nil {
writeAutoscanError(w, err)
@@ -454,9 +603,10 @@ func (h *AutoscanHandler) HandleCreateSource(w http.ResponseWriter, r *http.Requ
CapabilityID: capID,
ConnectionID: connArg,
Enabled: in.Enabled,
DeliveryMode: deliveryMode,
PollIntervalSeconds: in.PollIntervalSeconds,
PathRewrites: normalizePathRewrites(in.PathRewrites),
SourceConfig: normalizeSourceConfig(in.SourceConfig),
SourceConfig: sourceConfig,
Label: autoscan.NormalizeSourceLabel(in.Label),
})
if err != nil {
@@ -491,6 +641,7 @@ func scanSourceInstalled(available []autoscan.AvailableScanSource, pluginID, cap
type autoscanSourceInput struct {
ConnectionID *string `json:"connection_id"`
Enabled bool `json:"enabled"`
DeliveryMode string `json:"delivery_mode"`
PollIntervalSeconds *int `json:"poll_interval_seconds"`
PathRewrites []autoscan.PathRewrite `json:"path_rewrites"`
SourceConfig map[string]string `json:"source_config"`
@@ -549,6 +700,32 @@ func (h *AutoscanHandler) HandleUpdateSource(w http.ResponseWriter, r *http.Requ
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
return
}
// Delivery mode is validated against the existing row's immutable identity:
// an empty input keeps the stored mode.
existing, err := h.repo.GetSource(r.Context(), id)
if err != nil {
writeAutoscanError(w, err)
return
}
requestedMode := in.DeliveryMode
if strings.TrimSpace(requestedMode) == "" {
requestedMode = existing.DeliveryMode
}
deliveryMode, err := resolveDeliveryMode(requestedMode, existing.PluginID, existing.CapabilityID)
if err != nil {
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
return
}
sourceConfig := normalizeSourceConfig(in.SourceConfig)
if deliveryMode == autoscan.DeliveryModeWebhook {
if err := validateWebhookProvider(sourceConfig); err != nil {
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
return
}
if provider, ok := sourceConfig["webhook_provider"]; ok {
sourceConfig["webhook_provider"] = strings.ToLower(strings.TrimSpace(provider))
}
}
// connection_id is a full-state field: nil means unbind, a UUID string
// means bind. A whitespace-only string is normalised to nil (unbound).
connArg := normalizeConnectionID(in.ConnectionID)
@@ -558,16 +735,17 @@ func (h *AutoscanHandler) HandleUpdateSource(w http.ResponseWriter, r *http.Requ
ID: id,
ConnectionID: connArg,
Enabled: in.Enabled,
DeliveryMode: deliveryMode,
PollIntervalSeconds: in.PollIntervalSeconds,
PathRewrites: normalizePathRewrites(in.PathRewrites),
SourceConfig: normalizeSourceConfig(in.SourceConfig),
SourceConfig: sourceConfig,
Label: autoscan.NormalizeSourceLabel(in.Label),
})
if err != nil {
writeAutoscanError(w, err)
return
}
writeJSON(w, http.StatusOK, sourceResponse(updated))
writeJSON(w, http.StatusOK, h.sourceResponseWithWebhook(r.Context(), updated))
}
// normalizeConnectionID maps a full-state connection_id input to a stored value:
@@ -719,6 +897,8 @@ type autoscanEventResponse struct {
CompletedAt time.Time `json:"completed_at"`
DurationMS int64 `json:"duration_ms"`
Status string `json:"status"`
DeliveryMode string `json:"delivery_mode"`
ProviderEvent string `json:"provider_event_type,omitempty"`
ChangesReturned int `json:"changes_returned"`
ChangesResolved int `json:"changes_resolved"`
TargetsClaimed int `json:"targets_claimed"`
@@ -762,6 +942,8 @@ func eventResponse(event autoscan.EventWithRuns) autoscanEventResponse {
CompletedAt: e.CompletedAt,
DurationMS: e.DurationMS,
Status: string(e.Status),
DeliveryMode: e.DeliveryMode,
ProviderEvent: e.ProviderEventType,
ChangesReturned: e.ChangesReturned,
ChangesResolved: e.ChangesResolved,
TargetsClaimed: e.TargetsClaimed,
+82
View File
@@ -34,6 +34,16 @@ type fakeAutoscanStore struct {
listRunningFn func() ([]autoscan.Event, error)
queueSummaryFn func() (autoscan.QueueSummary, error)
latestEventAtFn func() (*time.Time, error)
createWebhookFn func(string) (autoscan.WebhookEndpoint, string, error)
rotateWebhookFn func(string) (autoscan.WebhookEndpoint, string, error)
deleteWebhookFn func(string) error
getWebhookFn func(string) (autoscan.WebhookEndpoint, error)
listWebhooksFn func() ([]autoscan.WebhookEndpoint, error)
revealTokenFn func(string) (string, error)
resolveTokenFn func(string) (autoscan.Source, autoscan.WebhookEndpoint, error)
touchedSources []string
webhookErrs map[string]string
}
func (f *fakeAutoscanStore) GetSettings(context.Context) (autoscan.Settings, error) {
@@ -162,6 +172,68 @@ func (f *fakeAutoscanStore) LatestEventAt(context.Context) (*time.Time, error) {
return nil, nil
}
func (f *fakeAutoscanStore) CreateWebhookEndpoint(_ context.Context, sourceID string) (autoscan.WebhookEndpoint, string, error) {
if f.createWebhookFn != nil {
return f.createWebhookFn(sourceID)
}
return autoscan.WebhookEndpoint{SourceID: sourceID}, "", nil
}
func (f *fakeAutoscanStore) RotateWebhookEndpoint(_ context.Context, sourceID string) (autoscan.WebhookEndpoint, string, error) {
if f.rotateWebhookFn != nil {
return f.rotateWebhookFn(sourceID)
}
return autoscan.WebhookEndpoint{SourceID: sourceID}, "", nil
}
func (f *fakeAutoscanStore) DeleteWebhookEndpoint(_ context.Context, sourceID string) error {
if f.deleteWebhookFn != nil {
return f.deleteWebhookFn(sourceID)
}
return nil
}
func (f *fakeAutoscanStore) GetWebhookEndpoint(_ context.Context, sourceID string) (autoscan.WebhookEndpoint, error) {
if f.getWebhookFn != nil {
return f.getWebhookFn(sourceID)
}
return autoscan.WebhookEndpoint{}, autoscan.ErrNotFound
}
func (f *fakeAutoscanStore) ListWebhookEndpoints(context.Context) ([]autoscan.WebhookEndpoint, error) {
if f.listWebhooksFn != nil {
return f.listWebhooksFn()
}
return nil, nil
}
func (f *fakeAutoscanStore) RevealWebhookToken(_ context.Context, sourceID string) (string, error) {
if f.revealTokenFn != nil {
return f.revealTokenFn(sourceID)
}
return "", autoscan.ErrNotFound
}
func (f *fakeAutoscanStore) ResolveWebhookToken(_ context.Context, token string) (autoscan.Source, autoscan.WebhookEndpoint, error) {
if f.resolveTokenFn != nil {
return f.resolveTokenFn(token)
}
return autoscan.Source{}, autoscan.WebhookEndpoint{}, autoscan.ErrNotFound
}
func (f *fakeAutoscanStore) TouchWebhookReceived(_ context.Context, sourceID string) error {
f.touchedSources = append(f.touchedSources, sourceID)
return nil
}
func (f *fakeAutoscanStore) RecordWebhookError(_ context.Context, sourceID, msg string) error {
if f.webhookErrs == nil {
f.webhookErrs = map[string]string{}
}
f.webhookErrs[sourceID] = msg
return nil
}
// fakeTriggerUpdater records the last UpdateTriggers call so a test can assert
// the handler reschedules the poll task with the new interval.
type fakeTriggerUpdater struct {
@@ -197,6 +269,11 @@ type fakeAutoscanTriggerer struct {
// send happens-before the test's receive, so reading `called` afterwards is
// race-free.
done chan struct{}
// ingested records IngestChanges calls; ingestResult/ingestErr drive the
// response.
ingested []autoscan.ChangeIngest
ingestResult autoscan.IngestResult
ingestErr error
}
func (f *fakeAutoscanTriggerer) PollOnce(context.Context) error {
@@ -223,6 +300,11 @@ func (f *fakeAutoscanTriggerer) SuggestRewrites(context.Context, string) (autosc
return f.suggestions, f.suggestErr
}
func (f *fakeAutoscanTriggerer) IngestChanges(_ context.Context, in autoscan.ChangeIngest) (autoscan.IngestResult, error) {
f.ingested = append(f.ingested, in)
return f.ingestResult, f.ingestErr
}
func newAutoscanRequest(method, target, body, id string) *http.Request {
var r *http.Request
if body != "" {
+182
View File
@@ -0,0 +1,182 @@
package handlers
import (
"errors"
"io"
"log/slog"
"net/http"
"strings"
"time"
"github.com/go-chi/chi/v5"
"github.com/Silo-Server/silo-server/internal/autoscan"
"github.com/Silo-Server/silo-server/internal/autoscan/arrwebhook"
)
// maxWebhookBodyBytes caps public webhook request bodies. Real Sonarr/Radarr
// payloads are a few KiB; season packs with many episode files stay well under
// this.
const maxWebhookBodyBytes = 256 * 1024
// --- Admin webhook endpoint management ---
// HandleCreateSourceWebhook creates the source's webhook endpoint if missing
// and returns the source view including the delivery URL.
// POST /admin/autoscan/sources/{id}/webhook
func (h *AutoscanHandler) HandleCreateSourceWebhook(w http.ResponseWriter, r *http.Request) {
id := strings.TrimSpace(chi.URLParam(r, "id"))
source, err := h.repo.GetSource(r.Context(), id)
if err != nil {
writeAutoscanError(w, err)
return
}
if source.DeliveryMode != autoscan.DeliveryModeWebhook {
writeError(w, http.StatusBadRequest, "bad_request", "source is not in webhook delivery mode")
return
}
if _, _, err := h.repo.CreateWebhookEndpoint(r.Context(), id); err != nil {
writeAutoscanError(w, err)
return
}
writeJSON(w, http.StatusOK, h.sourceResponseWithWebhook(r.Context(), source))
}
// HandleRotateSourceWebhook replaces the endpoint's token; the old URL stops
// working immediately.
// POST /admin/autoscan/sources/{id}/webhook/rotate
func (h *AutoscanHandler) HandleRotateSourceWebhook(w http.ResponseWriter, r *http.Request) {
id := strings.TrimSpace(chi.URLParam(r, "id"))
source, err := h.repo.GetSource(r.Context(), id)
if err != nil {
writeAutoscanError(w, err)
return
}
if _, _, err := h.repo.RotateWebhookEndpoint(r.Context(), id); err != nil {
writeAutoscanError(w, err)
return
}
writeJSON(w, http.StatusOK, h.sourceResponseWithWebhook(r.Context(), source))
}
// HandleDeleteSourceWebhook removes the endpoint; future deliveries to its URL
// return 404.
// DELETE /admin/autoscan/sources/{id}/webhook
func (h *AutoscanHandler) HandleDeleteSourceWebhook(w http.ResponseWriter, r *http.Request) {
id := strings.TrimSpace(chi.URLParam(r, "id"))
if err := h.repo.DeleteWebhookEndpoint(r.Context(), id); err != nil {
writeAutoscanError(w, err)
return
}
w.WriteHeader(http.StatusNoContent)
}
// --- Public delivery endpoint ---
// webhookAccepted is the 202 body for every accepted delivery (including
// no-ops), so senders can't distinguish disabled sources from active ones.
type webhookAccepted struct {
Status string `json:"status"`
}
// HandleWebhookDelivery accepts a Sonarr/Radarr webhook POST. The bearer token
// in the path authenticates the delivery. Responses are deliberately coarse:
// 404 for any unknown token (no existence hints), 202 for anything accepted —
// including test events, unsupported event types, and disabled sources — so
// arr never marks a configured webhook unhealthy for a Silo-side state it
// cannot act on. Actionable deliveries are durably queued before the 202, so
// transient ingest failures are retried inside Silo rather than delegated to
// arr (which does not replay failed notification events).
//
// The token, request URL, and body are never logged.
func (h *AutoscanHandler) HandleWebhookDelivery(w http.ResponseWriter, r *http.Request) {
receivedAt := time.Now()
token := chi.URLParam(r, "token")
source, _, err := h.repo.ResolveWebhookToken(r.Context(), token)
if err != nil {
if errors.Is(err, autoscan.ErrNotFound) {
writeError(w, http.StatusNotFound, "not_found", "Not found")
return
}
writeAutoscanError(w, err)
return
}
body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, maxWebhookBodyBytes))
if err != nil {
var maxErr *http.MaxBytesError
if errors.As(err, &maxErr) {
writeError(w, http.StatusRequestEntityTooLarge, "payload_too_large", "Request body exceeds the webhook payload cap")
return
}
writeError(w, http.StatusBadRequest, "bad_request", "Could not read request body")
return
}
parsed, err := arrwebhook.Parse(source.SourceConfig["webhook_provider"], body)
if err != nil {
// Parse errors carry no payload content; recording and returning them
// is safe. The token stays out of every message.
_ = h.repo.RecordWebhookError(r.Context(), source.ID, err.Error())
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
return
}
// Test events, unsupported event types, disabled sources, and globally
// disabled autoscan all acknowledge without enqueueing. Stamping
// last_received_at on every valid delivery keeps the admin "webhook is
// wired up" signal honest even while disabled.
settings, err := h.repo.GetSettings(r.Context())
if err != nil {
writeAutoscanError(w, err)
return
}
if terr := h.repo.TouchWebhookReceived(r.Context(), source.ID); terr != nil {
slog.WarnContext(r.Context(), "autoscan: touch webhook received failed", "component", "api", "source_id", source.ID, "err", terr)
}
if parsed.Test || len(parsed.Changes) == 0 || !settings.Enabled || !source.Enabled {
writeJSON(w, http.StatusAccepted, webhookAccepted{Status: "accepted"})
return
}
result, err := h.svc.IngestChanges(r.Context(), autoscan.ChangeIngest{
SourceID: source.ID,
ProviderEventType: parsed.EventType,
Changes: parsed.Changes,
ReceivedAt: receivedAt,
})
if err != nil {
// The only error from IngestChanges is failure to durably accept the
// delivery. Processing errors are queued internally and return Pending.
slog.WarnContext(r.Context(), "autoscan: webhook delivery acceptance failed", "component", "api",
"source_id", source.ID,
"provider", parsed.Provider,
"event_type", parsed.EventType,
"paths", len(parsed.Changes),
"err", err,
)
_ = h.repo.RecordWebhookError(r.Context(), source.ID, err.Error())
writeError(w, http.StatusInternalServerError, "internal_error", "Could not durably accept delivery")
return
}
if result.Pending {
slog.WarnContext(r.Context(), "autoscan: webhook delivery queued for retry", "component", "api",
"source_id", source.ID,
"provider", parsed.Provider,
"event_type", parsed.EventType,
"paths", len(parsed.Changes),
)
}
slog.DebugContext(r.Context(), "autoscan: webhook delivery ingested", "component", "api",
"source_id", source.ID,
"provider", parsed.Provider,
"event_type", parsed.EventType,
"paths", len(parsed.Changes),
"enqueued", result.Enqueued,
"suppressed", result.Suppressed,
"unresolved", result.Unresolved,
"pending", result.Pending,
)
writeJSON(w, http.StatusAccepted, webhookAccepted{Status: "accepted"})
}
@@ -0,0 +1,424 @@
package handlers
import (
"context"
"log/slog"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"github.com/go-chi/chi/v5"
"github.com/Silo-Server/silo-server/internal/autoscan"
)
const testWebhookToken = "wh_test_token_abcdef123456"
func webhookSource(enabled bool) autoscan.Source {
return autoscan.Source{
ID: "src-1",
PluginID: autoscan.BuiltinArrWebhookPluginID,
CapabilityID: autoscan.BuiltinArrWebhookCapabilityID,
Enabled: enabled,
DeliveryMode: autoscan.DeliveryModeWebhook,
SourceConfig: map[string]string{"webhook_provider": "auto"},
}
}
// webhookStore returns a store whose resolveTokenFn accepts testWebhookToken.
func webhookStore(source autoscan.Source, settingsEnabled bool) *fakeAutoscanStore {
return &fakeAutoscanStore{
getSettingsFn: func() (autoscan.Settings, error) {
return autoscan.Settings{Enabled: settingsEnabled, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60}, nil
},
resolveTokenFn: func(token string) (autoscan.Source, autoscan.WebhookEndpoint, error) {
if token == testWebhookToken {
return source, autoscan.WebhookEndpoint{SourceID: source.ID}, nil
}
return autoscan.Source{}, autoscan.WebhookEndpoint{}, autoscan.ErrNotFound
},
}
}
func newWebhookDeliveryRequest(token, body string) *http.Request {
r := httptest.NewRequest(http.MethodPost, "/api/v1/autoscan/webhooks/"+token, strings.NewReader(body))
routeCtx := chi.NewRouteContext()
routeCtx.URLParams.Add("token", token)
return r.WithContext(context.WithValue(r.Context(), chi.RouteCtxKey, routeCtx))
}
const sonarrDownloadBody = `{
"eventType": "Download",
"series": {"path": "/data/tv/Show"},
"episodeFile": {"path": "/data/tv/Show/Season 01/e01.mkv"}
}`
func TestWebhookDeliveryIngestsAndAccepts(t *testing.T) {
store := webhookStore(webhookSource(true), true)
svc := &fakeAutoscanTriggerer{ingestResult: autoscan.IngestResult{Enqueued: 1}}
h := NewAutoscanHandler(store, svc)
rec := httptest.NewRecorder()
h.HandleWebhookDelivery(rec, newWebhookDeliveryRequest(testWebhookToken, sonarrDownloadBody))
if rec.Code != http.StatusAccepted {
t.Fatalf("status = %d, want 202 (%s)", rec.Code, rec.Body.String())
}
if len(svc.ingested) != 1 {
t.Fatalf("ingested = %+v, want one call", svc.ingested)
}
in := svc.ingested[0]
if in.SourceID != "src-1" || in.ProviderEventType != "Download" || len(in.Changes) != 1 {
t.Fatalf("unexpected ingest input: %+v", in)
}
if in.Changes[0].SourcePath != "/data/tv/Show/Season 01/e01.mkv" {
t.Fatalf("change path = %q", in.Changes[0].SourcePath)
}
if len(store.touchedSources) != 1 || store.touchedSources[0] != "src-1" {
t.Fatalf("touched = %+v, want src-1", store.touchedSources)
}
}
func TestWebhookDeliveryUnknownTokenIs404(t *testing.T) {
store := webhookStore(webhookSource(true), true)
svc := &fakeAutoscanTriggerer{}
h := NewAutoscanHandler(store, svc)
rec := httptest.NewRecorder()
h.HandleWebhookDelivery(rec, newWebhookDeliveryRequest("bogus-token", sonarrDownloadBody))
if rec.Code != http.StatusNotFound {
t.Fatalf("status = %d, want 404", rec.Code)
}
if len(svc.ingested) != 0 {
t.Fatalf("unknown token must not ingest, got %+v", svc.ingested)
}
}
func TestWebhookDeliveryTestEventIsAcceptedNoOp(t *testing.T) {
store := webhookStore(webhookSource(true), true)
svc := &fakeAutoscanTriggerer{}
h := NewAutoscanHandler(store, svc)
rec := httptest.NewRecorder()
h.HandleWebhookDelivery(rec, newWebhookDeliveryRequest(testWebhookToken,
`{"eventType": "Test", "series": {"path": "C:\\testpath"}}`))
if rec.Code != http.StatusAccepted {
t.Fatalf("status = %d, want 202", rec.Code)
}
if len(svc.ingested) != 0 {
t.Fatalf("test event must not ingest, got %+v", svc.ingested)
}
if len(store.touchedSources) != 1 {
t.Fatalf("test event must stamp last_received_at, touched = %+v", store.touchedSources)
}
}
func TestWebhookDeliveryUnknownEventTypeIsAcceptedNoOp(t *testing.T) {
store := webhookStore(webhookSource(true), true)
svc := &fakeAutoscanTriggerer{}
h := NewAutoscanHandler(store, svc)
rec := httptest.NewRecorder()
h.HandleWebhookDelivery(rec, newWebhookDeliveryRequest(testWebhookToken,
`{"eventType": "Health", "series": {"path": "/data/tv/Show"}}`))
if rec.Code != http.StatusAccepted {
t.Fatalf("status = %d, want 202", rec.Code)
}
if len(svc.ingested) != 0 {
t.Fatalf("unknown event must not ingest, got %+v", svc.ingested)
}
}
func TestWebhookDeliveryDisabledSourceIsAcceptedNoOp(t *testing.T) {
for name, tc := range map[string]struct {
sourceEnabled bool
settingsEnabled bool
}{
"source disabled": {sourceEnabled: false, settingsEnabled: true},
"autoscan disabled": {sourceEnabled: true, settingsEnabled: false},
} {
t.Run(name, func(t *testing.T) {
store := webhookStore(webhookSource(tc.sourceEnabled), tc.settingsEnabled)
svc := &fakeAutoscanTriggerer{}
h := NewAutoscanHandler(store, svc)
rec := httptest.NewRecorder()
h.HandleWebhookDelivery(rec, newWebhookDeliveryRequest(testWebhookToken, sonarrDownloadBody))
if rec.Code != http.StatusAccepted {
t.Fatalf("status = %d, want 202", rec.Code)
}
if len(svc.ingested) != 0 {
t.Fatalf("disabled state must not ingest, got %+v", svc.ingested)
}
if len(store.touchedSources) != 1 {
t.Fatalf("valid delivery must stamp last_received_at even while disabled")
}
})
}
}
func TestWebhookDeliveryMalformedBodyIs400(t *testing.T) {
store := webhookStore(webhookSource(true), true)
svc := &fakeAutoscanTriggerer{}
h := NewAutoscanHandler(store, svc)
rec := httptest.NewRecorder()
h.HandleWebhookDelivery(rec, newWebhookDeliveryRequest(testWebhookToken, "{not json"))
if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want 400", rec.Code)
}
if store.webhookErrs["src-1"] == "" {
t.Fatal("parse failure must be recorded on the endpoint")
}
}
func TestWebhookDeliveryOversizedBodyIs413(t *testing.T) {
store := webhookStore(webhookSource(true), true)
h := NewAutoscanHandler(store, &fakeAutoscanTriggerer{})
big := `{"eventType": "Download", "pad": "` + strings.Repeat("x", maxWebhookBodyBytes+1) + `"}`
rec := httptest.NewRecorder()
h.HandleWebhookDelivery(rec, newWebhookDeliveryRequest(testWebhookToken, big))
if rec.Code != http.StatusRequestEntityTooLarge {
t.Fatalf("status = %d, want 413", rec.Code)
}
}
func TestWebhookDeliveryDurableAcceptanceFailureIs500(t *testing.T) {
store := webhookStore(webhookSource(true), true)
svc := &fakeAutoscanTriggerer{ingestErr: context.DeadlineExceeded}
h := NewAutoscanHandler(store, svc)
rec := httptest.NewRecorder()
h.HandleWebhookDelivery(rec, newWebhookDeliveryRequest(testWebhookToken, sonarrDownloadBody))
if rec.Code != http.StatusInternalServerError {
t.Fatalf("status = %d, want 500 when the delivery could not be persisted", rec.Code)
}
if store.webhookErrs["src-1"] == "" {
t.Fatal("ingest failure must be recorded on the endpoint")
}
}
func TestWebhookDeliveryPendingRetryIsAccepted(t *testing.T) {
store := webhookStore(webhookSource(true), true)
svc := &fakeAutoscanTriggerer{ingestResult: autoscan.IngestResult{Pending: true}}
h := NewAutoscanHandler(store, svc)
rec := httptest.NewRecorder()
h.HandleWebhookDelivery(rec, newWebhookDeliveryRequest(testWebhookToken, sonarrDownloadBody))
if rec.Code != http.StatusAccepted {
t.Fatalf("status = %d, want 202 for a durably queued retry", rec.Code)
}
}
// recordingLogHandler captures every log record produced during a test.
type recordingLogHandler struct {
mu sync.Mutex
entries []string
}
func (h *recordingLogHandler) Enabled(context.Context, slog.Level) bool { return true }
func (h *recordingLogHandler) Handle(_ context.Context, r slog.Record) error {
var b strings.Builder
b.WriteString(r.Message)
r.Attrs(func(a slog.Attr) bool {
b.WriteString(" ")
b.WriteString(a.String())
return true
})
h.mu.Lock()
h.entries = append(h.entries, b.String())
h.mu.Unlock()
return nil
}
func (h *recordingLogHandler) WithAttrs([]slog.Attr) slog.Handler { return h }
func (h *recordingLogHandler) WithGroup(string) slog.Handler { return h }
func TestWebhookDeliveryNeverLogsToken(t *testing.T) {
recorder := &recordingLogHandler{}
prev := slog.Default()
slog.SetDefault(slog.New(recorder))
defer slog.SetDefault(prev)
store := webhookStore(webhookSource(true), true)
// Force both the failure and success logging paths.
for _, svc := range []*fakeAutoscanTriggerer{
{ingestErr: context.DeadlineExceeded},
{ingestResult: autoscan.IngestResult{Enqueued: 1}},
} {
h := NewAutoscanHandler(store, svc)
rec := httptest.NewRecorder()
h.HandleWebhookDelivery(rec, newWebhookDeliveryRequest(testWebhookToken, sonarrDownloadBody))
}
recorder.mu.Lock()
defer recorder.mu.Unlock()
for _, entry := range recorder.entries {
if strings.Contains(entry, testWebhookToken) {
t.Fatalf("log entry leaks the webhook token: %s", entry)
}
if strings.Contains(entry, "/data/tv/Show/Season 01") {
t.Fatalf("log entry leaks payload content: %s", entry)
}
}
}
func TestResolveDeliveryMode(t *testing.T) {
builtin := autoscan.BuiltinArrWebhookPluginID
builtinCap := autoscan.BuiltinArrWebhookCapabilityID
cases := []struct {
name string
requested string
pluginID string
capID string
want string
wantErr bool
}{
{"plugin defaults to poll", "", "sonarr", "arr", autoscan.DeliveryModePoll, false},
{"builtin defaults to webhook", "", builtin, builtinCap, autoscan.DeliveryModeWebhook, false},
{"plugin poll ok", "poll", "sonarr", "arr", autoscan.DeliveryModePoll, false},
{"builtin webhook ok", "webhook", builtin, builtinCap, autoscan.DeliveryModeWebhook, false},
{"plugin webhook rejected", "webhook", "sonarr", "arr", "", true},
{"builtin poll rejected", "poll", builtin, builtinCap, "", true},
{"garbage rejected", "carrier-pigeon", "sonarr", "arr", "", true},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got, err := resolveDeliveryMode(tc.requested, tc.pluginID, tc.capID)
if tc.wantErr != (err != nil) {
t.Fatalf("err = %v, wantErr = %v", err, tc.wantErr)
}
if got != tc.want {
t.Fatalf("mode = %q, want %q", got, tc.want)
}
})
}
}
func TestValidateWebhookProvider(t *testing.T) {
for _, ok := range []string{"auto", "sonarr", "radarr", "", "Sonarr"} {
if err := validateWebhookProvider(map[string]string{"webhook_provider": ok}); err != nil {
t.Fatalf("provider %q must validate: %v", ok, err)
}
}
if err := validateWebhookProvider(map[string]string{"webhook_provider": "lidarr"}); err == nil {
t.Fatal("unsupported provider must be rejected")
}
if err := validateWebhookProvider(map[string]string{}); err != nil {
t.Fatalf("absent provider must validate: %v", err)
}
}
func TestUpdatePollSourceAllowsPluginWebhookProviderKey(t *testing.T) {
existing := autoscan.Source{
ID: "src-1",
PluginID: "example.plugin",
CapabilityID: "source",
DeliveryMode: autoscan.DeliveryModePoll,
}
var saved autoscan.Source
store := &fakeAutoscanStore{
getSourceFn: func(string) (autoscan.Source, error) { return existing, nil },
updateSourceFn: func(source autoscan.Source) (autoscan.Source, error) {
saved = source
return source, nil
},
}
h := NewAutoscanHandler(store, &fakeAutoscanTriggerer{})
body := `{"enabled":true,"delivery_mode":"poll","path_rewrites":[],"source_config":{"webhook_provider":"plugin-specific"}}`
rec := httptest.NewRecorder()
h.HandleUpdateSource(rec, newAutoscanRequest(http.MethodPut, "/admin/autoscan/sources/src-1", body, "src-1"))
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200 (%s)", rec.Code, rec.Body.String())
}
if got := saved.SourceConfig["webhook_provider"]; got != "plugin-specific" {
t.Fatalf("plugin source config = %q, want unchanged", got)
}
}
// --- Admin endpoint management ---
func TestCreateSourceWebhookRequiresWebhookMode(t *testing.T) {
store := &fakeAutoscanStore{
getSourceFn: func(id string) (autoscan.Source, error) {
return autoscan.Source{ID: id, DeliveryMode: autoscan.DeliveryModePoll}, nil
},
}
h := NewAutoscanHandler(store, &fakeAutoscanTriggerer{})
rec := httptest.NewRecorder()
h.HandleCreateSourceWebhook(rec, newAutoscanRequest(http.MethodPost, "/admin/autoscan/sources/s1/webhook", "", "s1"))
if rec.Code != http.StatusBadRequest {
t.Fatalf("status = %d, want 400 for poll-mode source", rec.Code)
}
}
func TestCreateSourceWebhookReturnsURL(t *testing.T) {
source := webhookSource(true)
store := &fakeAutoscanStore{
getSourceFn: func(string) (autoscan.Source, error) { return source, nil },
createWebhookFn: func(sourceID string) (autoscan.WebhookEndpoint, string, error) {
return autoscan.WebhookEndpoint{SourceID: sourceID, SecretSuffix: "def456"}, testWebhookToken, nil
},
getWebhookFn: func(sourceID string) (autoscan.WebhookEndpoint, error) {
return autoscan.WebhookEndpoint{SourceID: sourceID, SecretSuffix: "def456"}, nil
},
revealTokenFn: func(string) (string, error) { return testWebhookToken, nil },
}
h := NewAutoscanHandler(store, &fakeAutoscanTriggerer{})
h.SetPublicURL("https://silo.example/")
rec := httptest.NewRecorder()
h.HandleCreateSourceWebhook(rec, newAutoscanRequest(http.MethodPost, "/admin/autoscan/sources/src-1/webhook", "", "src-1"))
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200 (%s)", rec.Code, rec.Body.String())
}
body := rec.Body.String()
wantURL := `"webhook_url":"https://silo.example/api/v1/autoscan/webhooks/` + testWebhookToken + `"`
if !strings.Contains(body, wantURL) {
t.Fatalf("response missing %s: %s", wantURL, body)
}
if !strings.Contains(body, `"webhook_configured":true`) || !strings.Contains(body, `"webhook_secret_suffix":"def456"`) {
t.Fatalf("response missing webhook status fields: %s", body)
}
}
func TestSourceResponsesOmitWebhookURLWhenRevealFails(t *testing.T) {
source := webhookSource(true)
store := &fakeAutoscanStore{
listSourcesFn: func() ([]autoscan.Source, error) { return []autoscan.Source{source}, nil },
listWebhooksFn: func() ([]autoscan.WebhookEndpoint, error) {
return []autoscan.WebhookEndpoint{{SourceID: source.ID, SecretSuffix: "def456"}}, nil
},
revealTokenFn: func(string) (string, error) { return "", context.DeadlineExceeded },
}
h := NewAutoscanHandler(store, &fakeAutoscanTriggerer{})
rec := httptest.NewRecorder()
h.HandleListSources(rec, newAutoscanRequest(http.MethodGet, "/admin/autoscan/sources", "", ""))
if rec.Code != http.StatusOK {
t.Fatalf("status = %d, want 200", rec.Code)
}
body := rec.Body.String()
if strings.Contains(body, "webhook_url") {
t.Fatalf("reveal failure must omit webhook_url, got %s", body)
}
if !strings.Contains(body, `"webhook_configured":true`) {
t.Fatalf("status fields must survive reveal failure: %s", body)
}
}
+1 -1
View File
@@ -52,7 +52,7 @@ func RequestLogger(nodeID string) func(http.Handler) http.Handler {
"component", "api",
"request_id", chimw.GetReqID(r.Context()),
"method", r.Method,
"path", r.URL.Path,
"path", activitylog.RedactSecretPathParams(r, r.URL.Path),
"path_pattern", pathPattern,
"status", wrapped.status,
"duration_ms", time.Since(start).Milliseconds(),
+18
View File
@@ -662,6 +662,8 @@ func NewRouter(deps Dependencies) chi.Router {
if deps.TaskManager != nil {
autoscanHandler.SetTriggerUpdater(deps.TaskManager)
}
// Fully qualify webhook URLs when the public base URL is known.
autoscanHandler.SetPublicURL(deps.PublicURL)
}
if deps.PersonRepo != nil {
@@ -1684,6 +1686,19 @@ func NewRouter(deps Dependencies) chi.Router {
})
}
// Autoscan webhook intake: public — Sonarr/Radarr POST here without a
// Silo session; the URL's bearer token authenticates the delivery and
// maps it to its Autoscan source. Rate limited per-IP (plus the
// "autoscan_webhook" per-endpoint limit) since it is unauthenticated.
if autoscanHandler != nil {
if deps.RateLimitMW != nil {
r.With(deps.RateLimitMW.AuthEndpointHandler("autoscan_webhook")).
Post("/autoscan/webhooks/{token}", autoscanHandler.HandleWebhookDelivery)
} else {
r.Post("/autoscan/webhooks/{token}", autoscanHandler.HandleWebhookDelivery)
}
}
// Discord account-link OAuth callback: public — Discord redirects the
// browser here without credentials; the one-time link-state row
// authenticates the request and maps it back to the initiating
@@ -2808,6 +2823,9 @@ func NewRouter(deps Dependencies) chi.Router {
r.Put("/autoscan/sources/{id}", autoscanHandler.HandleUpdateSource)
r.Delete("/autoscan/sources/{id}", autoscanHandler.HandleDeleteSource)
r.Get("/autoscan/sources/{id}/rewrite-suggestions", autoscanHandler.HandleRewriteSuggestions)
r.Post("/autoscan/sources/{id}/webhook", autoscanHandler.HandleCreateSourceWebhook)
r.Post("/autoscan/sources/{id}/webhook/rotate", autoscanHandler.HandleRotateSourceWebhook)
r.Delete("/autoscan/sources/{id}/webhook", autoscanHandler.HandleDeleteSourceWebhook)
r.Get("/autoscan/scans", autoscanHandler.HandleListScans)
r.Get("/autoscan/events", autoscanHandler.HandleListEvents)
r.Post("/autoscan/trigger", autoscanHandler.HandleTrigger)
+204
View File
@@ -0,0 +1,204 @@
// Package arrwebhook parses Sonarr/Radarr webhook payloads into autoscan
// changes. It is deliberately host-side and handler-free: input is the raw
// request body, output is the provider/event classification plus the file or
// subtree paths to feed autoscan.Service.IngestChanges. Payload-derived
// content never appears in returned error messages, so they are safe to log
// and to echo in HTTP responses.
package arrwebhook
import (
"encoding/json"
"errors"
"strings"
"github.com/Silo-Server/silo-server/internal/autoscan"
)
// Provider names accepted by Parse and stored in source_config.webhook_provider.
const (
ProviderAuto = "auto"
ProviderSonarr = "sonarr"
ProviderRadarr = "radarr"
)
// ErrMalformedPayload reports a body that is not valid JSON or has no
// eventType. Handlers map it to 400.
var ErrMalformedPayload = errors.New("arrwebhook: malformed payload")
// ErrUnknownProvider reports a payload whose provider could not be inferred in
// auto mode. Handlers map it to 400.
var ErrUnknownProvider = errors.New("arrwebhook: cannot infer provider from payload")
// ParsedWebhook is one classified delivery. Unsupported event types parse
// successfully with no changes (callers no-op them): arr adds event types over
// time and an unknown one must not make the webhook look unhealthy.
type ParsedWebhook struct {
Provider string
EventType string
Test bool
Changes []autoscan.Change
}
// fileRef is the common shape of arr webhook file resources (episodeFile,
// movieFile, and the renamed* variants).
type fileRef struct {
Path string `json:"path"`
PreviousPath string `json:"previousPath"`
}
type payload struct {
EventType string `json:"eventType"`
Series *struct {
Path string `json:"path"`
} `json:"series"`
EpisodeFile *fileRef `json:"episodeFile"`
EpisodeFiles []fileRef `json:"episodeFiles"`
RenamedEpisodeFiles []fileRef `json:"renamedEpisodeFiles"`
Movie *struct {
Path string `json:"path"`
FolderPath string `json:"folderPath"`
} `json:"movie"`
MovieFile *fileRef `json:"movieFile"`
MovieFiles []fileRef `json:"movieFiles"`
RenamedMovieFiles []fileRef `json:"renamedMovieFiles"`
// Download/import payloads include the files replaced by an upgrade. They
// must be reconciled as vanished paths even when a separate "file delete for
// upgrade" notification is not enabled in arr.
DeletedFiles []fileRef `json:"deletedFiles"`
}
// importEventTypes are the events that carry newly imported files.
var importEventTypes = map[string]bool{
"Download": true,
"Import": true,
"Upgrade": true,
"DownloadComplete": true,
}
// deleteEventTypes are the events that carry a removed file path.
var deleteEventTypes = map[string]bool{
"EpisodeFileDelete": true,
"MovieFileDelete": true,
}
// Parse classifies a webhook body. provider is "sonarr", "radarr", or "auto"
// (empty means auto); in auto mode the provider is inferred from the payload's
// series/movie shape. Inference failure is an error only for non-test events
// that would otherwise produce work.
func Parse(provider string, body []byte) (ParsedWebhook, error) {
var p payload
if err := json.Unmarshal(body, &p); err != nil {
return ParsedWebhook{}, ErrMalformedPayload
}
eventType := strings.TrimSpace(p.EventType)
if eventType == "" {
return ParsedWebhook{}, ErrMalformedPayload
}
out := ParsedWebhook{EventType: eventType, Provider: resolveProvider(provider, p)}
if strings.EqualFold(eventType, "Test") {
out.Test = true
return out, nil
}
switch {
case importEventTypes[eventType]:
out.Changes = filePaths(p, false)
if len(out.Changes) == 0 {
out.Changes = subtreeFallback(p)
}
case eventType == "Rename":
out.Changes = filePaths(p, true)
case deleteEventTypes[eventType]:
out.Changes = filePaths(p, false)
default:
// Unsupported event type: valid delivery, nothing to do.
return out, nil
}
if out.Provider == "" {
return ParsedWebhook{}, ErrUnknownProvider
}
return out, nil
}
// resolveProvider returns the explicit provider when set, otherwise infers
// sonarr/radarr from the payload shape ("" when inference fails).
func resolveProvider(provider string, p payload) string {
switch strings.ToLower(strings.TrimSpace(provider)) {
case ProviderSonarr:
return ProviderSonarr
case ProviderRadarr:
return ProviderRadarr
}
switch {
case p.Series != nil || p.EpisodeFile != nil || len(p.EpisodeFiles) > 0 || len(p.RenamedEpisodeFiles) > 0:
return ProviderSonarr
case p.Movie != nil || p.MovieFile != nil || len(p.MovieFiles) > 0 || len(p.RenamedMovieFiles) > 0:
return ProviderRadarr
}
return ""
}
// filePaths collects every file path in the payload as file-scope changes,
// deduped in order. includePrevious additionally collects previousPath values
// (rename events): the previous path may no longer exist, which the host's
// resolve pipeline already handles via its vanished-path fallback.
func filePaths(p payload, includePrevious bool) []autoscan.Change {
var refs []fileRef
if p.EpisodeFile != nil {
refs = append(refs, *p.EpisodeFile)
}
refs = append(refs, p.EpisodeFiles...)
refs = append(refs, p.RenamedEpisodeFiles...)
if p.MovieFile != nil {
refs = append(refs, *p.MovieFile)
}
refs = append(refs, p.MovieFiles...)
refs = append(refs, p.RenamedMovieFiles...)
refs = append(refs, p.DeletedFiles...)
seen := make(map[string]struct{})
var changes []autoscan.Change
add := func(path string) {
path = strings.TrimSpace(path)
if path == "" {
return
}
if _, dup := seen[path]; dup {
return
}
seen[path] = struct{}{}
changes = append(changes, autoscan.Change{SourcePath: path, Scope: autoscan.ChangeScopeFile})
}
for _, ref := range refs {
add(ref.Path)
if includePrevious {
add(ref.PreviousPath)
}
}
return changes
}
// subtreeFallback maps an import event with no file details to the series or
// movie folder as one subtree-scope change.
func subtreeFallback(p payload) []autoscan.Change {
path := ""
switch {
case p.Series != nil:
path = p.Series.Path
case p.Movie != nil:
path = p.Movie.FolderPath
if strings.TrimSpace(path) == "" {
path = p.Movie.Path
}
}
path = strings.TrimSpace(path)
if path == "" {
return nil
}
return []autoscan.Change{{SourcePath: path, Scope: autoscan.ChangeScopeSubtree}}
}
+245
View File
@@ -0,0 +1,245 @@
package arrwebhook
import (
"errors"
"os"
"path/filepath"
"testing"
"github.com/Silo-Server/silo-server/internal/autoscan"
)
func fixture(t *testing.T, name string) []byte {
t.Helper()
body, err := os.ReadFile(filepath.Join("testdata", name))
if err != nil {
t.Fatalf("read fixture %s: %v", name, err)
}
return body
}
func paths(changes []autoscan.Change) []string {
out := make([]string, 0, len(changes))
for _, c := range changes {
out = append(out, c.SourcePath)
}
return out
}
func TestParseSonarrDownload(t *testing.T) {
parsed, err := Parse(ProviderAuto, fixture(t, "sonarr_download.json"))
if err != nil {
t.Fatalf("Parse: %v", err)
}
if parsed.Provider != ProviderSonarr || parsed.EventType != "Download" || parsed.Test {
t.Fatalf("unexpected classification: %+v", parsed)
}
want := "/data/tv/Example Show/Season 02/Example Show - S02E01 - The One That Imports.mkv"
if len(parsed.Changes) != 1 || parsed.Changes[0].SourcePath != want || parsed.Changes[0].Scope != autoscan.ChangeScopeFile {
t.Fatalf("changes = %+v", parsed.Changes)
}
}
func TestParseSonarrRenameIncludesPreviousPaths(t *testing.T) {
parsed, err := Parse(ProviderAuto, fixture(t, "sonarr_rename.json"))
if err != nil {
t.Fatalf("Parse: %v", err)
}
got := paths(parsed.Changes)
want := []string{
"/data/tv/Example Show/Season 02/Example Show - S02E01.mkv",
"/data/tv/Example Show/Season 02/old-name-e01.mkv",
"/data/tv/Example Show/Season 02/Example Show - S02E02.mkv",
"/data/tv/Example Show/Season 02/old-name-e02.mkv",
}
if len(got) != len(want) {
t.Fatalf("paths = %v, want %v", got, want)
}
for i := range want {
if got[i] != want[i] {
t.Fatalf("paths[%d] = %q, want %q", i, got[i], want[i])
}
}
for _, c := range parsed.Changes {
if c.Scope != autoscan.ChangeScopeFile {
t.Fatalf("rename changes must be file scope, got %+v", c)
}
}
}
func TestParseSonarrEpisodeFileDelete(t *testing.T) {
parsed, err := Parse(ProviderAuto, fixture(t, "sonarr_episodefile_delete.json"))
if err != nil {
t.Fatalf("Parse: %v", err)
}
if parsed.EventType != "EpisodeFileDelete" {
t.Fatalf("event type = %q", parsed.EventType)
}
if len(parsed.Changes) != 1 || parsed.Changes[0].SourcePath != "/data/tv/Example Show/Season 02/Example Show - S02E01.mkv" {
t.Fatalf("changes = %+v", parsed.Changes)
}
}
func TestParseSonarrTest(t *testing.T) {
parsed, err := Parse(ProviderAuto, fixture(t, "sonarr_test.json"))
if err != nil {
t.Fatalf("Parse: %v", err)
}
if !parsed.Test || len(parsed.Changes) != 0 {
t.Fatalf("test event must be a no-op: %+v", parsed)
}
if parsed.Provider != ProviderSonarr {
t.Fatalf("provider = %q", parsed.Provider)
}
}
func TestParseRadarrDownload(t *testing.T) {
parsed, err := Parse(ProviderAuto, fixture(t, "radarr_download.json"))
if err != nil {
t.Fatalf("Parse: %v", err)
}
if parsed.Provider != ProviderRadarr || parsed.EventType != "Download" {
t.Fatalf("unexpected classification: %+v", parsed)
}
want := "/data/movies/Example Movie (2024)/Example Movie (2024) Bluray-1080p.mkv"
if len(parsed.Changes) != 1 || parsed.Changes[0].SourcePath != want {
t.Fatalf("changes = %+v", parsed.Changes)
}
}
func TestParseUpgradeIncludesReplacedFiles(t *testing.T) {
for name, body := range map[string]string{
"sonarr": `{
"eventType": "Download",
"series": {"path": "/data/tv/Show"},
"episodeFile": {"path": "/data/tv/Show/new.mkv"},
"deletedFiles": [{"path": "/data/tv/Show/old.mkv"}],
"isUpgrade": true
}`,
"radarr": `{
"eventType": "Download",
"movie": {"folderPath": "/data/movies/Movie"},
"movieFile": {"path": "/data/movies/Movie/new.mkv"},
"deletedFiles": [{"path": "/data/movies/Movie/old.mkv"}],
"isUpgrade": true
}`,
} {
t.Run(name, func(t *testing.T) {
parsed, err := Parse(ProviderAuto, []byte(body))
if err != nil {
t.Fatalf("Parse: %v", err)
}
got := paths(parsed.Changes)
if len(got) != 2 || got[1] == "" || got[1] == got[0] {
t.Fatalf("upgrade paths = %v, want new and replaced file", got)
}
for _, change := range parsed.Changes {
if change.Scope != autoscan.ChangeScopeFile {
t.Fatalf("upgrade change = %+v, want file scope", change)
}
}
})
}
}
func TestParseRadarrRename(t *testing.T) {
parsed, err := Parse(ProviderAuto, fixture(t, "radarr_rename.json"))
if err != nil {
t.Fatalf("Parse: %v", err)
}
got := paths(parsed.Changes)
want := []string{
"/data/movies/Example Movie (2024)/Example Movie (2024) Bluray-1080p.mkv",
"/data/movies/Example Movie (2024)/old-release-name.mkv",
}
if len(got) != 2 || got[0] != want[0] || got[1] != want[1] {
t.Fatalf("paths = %v, want %v", got, want)
}
}
func TestParseRadarrMovieFileDelete(t *testing.T) {
parsed, err := Parse(ProviderAuto, fixture(t, "radarr_moviefile_delete.json"))
if err != nil {
t.Fatalf("Parse: %v", err)
}
if parsed.EventType != "MovieFileDelete" || len(parsed.Changes) != 1 {
t.Fatalf("unexpected parse: %+v", parsed)
}
}
func TestParseRadarrTest(t *testing.T) {
parsed, err := Parse(ProviderAuto, fixture(t, "radarr_test.json"))
if err != nil {
t.Fatalf("Parse: %v", err)
}
if !parsed.Test || parsed.Provider != ProviderRadarr {
t.Fatalf("unexpected parse: %+v", parsed)
}
}
func TestParseUnknownEventIsNoOpNotError(t *testing.T) {
parsed, err := Parse(ProviderSonarr, fixture(t, "unknown_event.json"))
if err != nil {
t.Fatalf("unknown event types must parse: %v", err)
}
if parsed.EventType != "Health" || parsed.Test || len(parsed.Changes) != 0 {
t.Fatalf("unknown event must be a no-op: %+v", parsed)
}
}
func TestParseMalformedBody(t *testing.T) {
if _, err := Parse(ProviderAuto, []byte("{not json")); !errors.Is(err, ErrMalformedPayload) {
t.Fatalf("want ErrMalformedPayload, got %v", err)
}
if _, err := Parse(ProviderAuto, []byte(`{"noEventType": true}`)); !errors.Is(err, ErrMalformedPayload) {
t.Fatalf("missing eventType must be malformed, got %v", err)
}
}
func TestParseAutoInferenceFailure(t *testing.T) {
// A work-producing event with neither series nor movie shape cannot be
// attributed in auto mode.
body := []byte(`{"eventType": "Download"}`)
if _, err := Parse(ProviderAuto, body); !errors.Is(err, ErrUnknownProvider) {
t.Fatalf("want ErrUnknownProvider, got %v", err)
}
// The same payload with an explicit provider parses (zero changes).
parsed, err := Parse(ProviderSonarr, body)
if err != nil {
t.Fatalf("explicit provider must parse: %v", err)
}
if len(parsed.Changes) != 0 {
t.Fatalf("changes = %+v, want none", parsed.Changes)
}
}
func TestParseImportFallsBackToSubtree(t *testing.T) {
body := []byte(`{
"eventType": "Download",
"series": {"path": "/data/tv/Example Show"}
}`)
parsed, err := Parse(ProviderAuto, body)
if err != nil {
t.Fatalf("Parse: %v", err)
}
if len(parsed.Changes) != 1 ||
parsed.Changes[0].SourcePath != "/data/tv/Example Show" ||
parsed.Changes[0].Scope != autoscan.ChangeScopeSubtree {
t.Fatalf("changes = %+v, want one subtree fallback", parsed.Changes)
}
}
func TestParseDedupesExactPaths(t *testing.T) {
body := []byte(`{
"eventType": "Download",
"episodeFile": {"path": "/data/tv/S/e1.mkv"},
"episodeFiles": [{"path": "/data/tv/S/e1.mkv"}, {"path": "/data/tv/S/e2.mkv"}]
}`)
parsed, err := Parse(ProviderAuto, body)
if err != nil {
t.Fatalf("Parse: %v", err)
}
if got := paths(parsed.Changes); len(got) != 2 || got[0] != "/data/tv/S/e1.mkv" || got[1] != "/data/tv/S/e2.mkv" {
t.Fatalf("paths = %v, want deduped pair", got)
}
}
@@ -0,0 +1,30 @@
{
"movie": {
"id": 44,
"title": "Example Movie",
"year": 2024,
"folderPath": "/data/movies/Example Movie (2024)",
"tmdbId": 654321
},
"remoteMovie": {
"tmdbId": 654321,
"imdbId": "tt7654321",
"title": "Example Movie",
"year": 2024
},
"movieFile": {
"id": 771,
"relativePath": "Example Movie (2024) Bluray-1080p.mkv",
"path": "/data/movies/Example Movie (2024)/Example Movie (2024) Bluray-1080p.mkv",
"quality": "Bluray-1080p",
"qualityVersion": 1,
"releaseGroup": "GROUP",
"sceneName": "Example.Movie.2024.1080p.BluRay",
"size": 4294967296
},
"isUpgrade": false,
"downloadClient": "qbittorrent",
"downloadId": "def456",
"eventType": "Download",
"instanceName": "Radarr"
}
@@ -0,0 +1,20 @@
{
"movie": {
"id": 44,
"title": "Example Movie",
"year": 2024,
"folderPath": "/data/movies/Example Movie (2024)",
"tmdbId": 654321
},
"movieFile": {
"id": 771,
"relativePath": "Example Movie (2024) Bluray-1080p.mkv",
"path": "/data/movies/Example Movie (2024)/Example Movie (2024) Bluray-1080p.mkv",
"quality": "Bluray-1080p",
"qualityVersion": 1,
"size": 4294967296
},
"deleteReason": "deleted",
"eventType": "MovieFileDelete",
"instanceName": "Radarr"
}
@@ -0,0 +1,20 @@
{
"movie": {
"id": 44,
"title": "Example Movie",
"year": 2024,
"folderPath": "/data/movies/Example Movie (2024)",
"tmdbId": 654321
},
"renamedMovieFiles": [
{
"id": 771,
"previousRelativePath": "old-release-name.mkv",
"previousPath": "/data/movies/Example Movie (2024)/old-release-name.mkv",
"relativePath": "Example Movie (2024) Bluray-1080p.mkv",
"path": "/data/movies/Example Movie (2024)/Example Movie (2024) Bluray-1080p.mkv"
}
],
"eventType": "Rename",
"instanceName": "Radarr"
}
+22
View File
@@ -0,0 +1,22 @@
{
"movie": {
"id": 1,
"title": "Test Title",
"year": 1970,
"folderPath": "C:\\testpath",
"tmdbId": 1234
},
"remoteMovie": {
"tmdbId": 1234,
"imdbId": "tt1234",
"title": "Test title",
"year": 1970
},
"release": {
"quality": "Test Quality",
"qualityVersion": 1,
"size": 9999999
},
"eventType": "Test",
"instanceName": "Radarr"
}
@@ -0,0 +1,32 @@
{
"series": {
"id": 12,
"title": "Example Show",
"path": "/data/tv/Example Show",
"tvdbId": 123456,
"type": "standard"
},
"episodes": [
{
"id": 401,
"episodeNumber": 1,
"seasonNumber": 2,
"title": "The One That Imports"
}
],
"episodeFile": {
"id": 991,
"relativePath": "Season 02/Example Show - S02E01 - The One That Imports.mkv",
"path": "/data/tv/Example Show/Season 02/Example Show - S02E01 - The One That Imports.mkv",
"quality": "WEBDL-1080p",
"qualityVersion": 1,
"releaseGroup": "GROUP",
"sceneName": "Example.Show.S02E01.1080p.WEB-DL",
"size": 2147483648
},
"isUpgrade": false,
"downloadClient": "sabnzbd",
"downloadId": "abc123",
"eventType": "Download",
"instanceName": "Sonarr"
}
@@ -0,0 +1,28 @@
{
"series": {
"id": 12,
"title": "Example Show",
"path": "/data/tv/Example Show",
"tvdbId": 123456,
"type": "standard"
},
"episodes": [
{
"id": 401,
"episodeNumber": 1,
"seasonNumber": 2,
"title": "The One That Got Deleted"
}
],
"episodeFile": {
"id": 991,
"relativePath": "Season 02/Example Show - S02E01.mkv",
"path": "/data/tv/Example Show/Season 02/Example Show - S02E01.mkv",
"quality": "WEBDL-1080p",
"qualityVersion": 1,
"size": 2147483648
},
"deleteReason": "upgrade",
"eventType": "EpisodeFileDelete",
"instanceName": "Sonarr"
}
@@ -0,0 +1,27 @@
{
"series": {
"id": 12,
"title": "Example Show",
"path": "/data/tv/Example Show",
"tvdbId": 123456,
"type": "standard"
},
"renamedEpisodeFiles": [
{
"id": 991,
"previousRelativePath": "Season 02/old-name-e01.mkv",
"previousPath": "/data/tv/Example Show/Season 02/old-name-e01.mkv",
"relativePath": "Season 02/Example Show - S02E01.mkv",
"path": "/data/tv/Example Show/Season 02/Example Show - S02E01.mkv"
},
{
"id": 992,
"previousRelativePath": "Season 02/old-name-e02.mkv",
"previousPath": "/data/tv/Example Show/Season 02/old-name-e02.mkv",
"relativePath": "Season 02/Example Show - S02E02.mkv",
"path": "/data/tv/Example Show/Season 02/Example Show - S02E02.mkv"
}
],
"eventType": "Rename",
"instanceName": "Sonarr"
}
+19
View File
@@ -0,0 +1,19 @@
{
"series": {
"id": 1,
"title": "Test Title",
"path": "C:\\testpath",
"tvdbId": 1234,
"type": "standard"
},
"episodes": [
{
"id": 123,
"episodeNumber": 1,
"seasonNumber": 1,
"title": "Test title"
}
],
"eventType": "Test",
"instanceName": "Sonarr"
}
@@ -0,0 +1,8 @@
{
"level": "warning",
"message": "Indexers unavailable due to failures",
"type": "IndexerStatusCheck",
"wikiUrl": "https://wiki.servarr.com/sonarr/system#indexers-are-unavailable-due-to-failures",
"eventType": "Health",
"instanceName": "Sonarr"
}
+52
View File
@@ -5,6 +5,33 @@ import (
"fmt"
)
// Built-in source identities are host-discovered scan-source entries that need
// no plugin installation. The ARR webhook identity backs webhook-mode sources:
// Sonarr/Radarr POST directly to Silo, the host parses the payload, and the
// plugin provider is never invoked for it.
const (
BuiltinArrWebhookPluginID = "silo.autoscan.arr-webhook"
BuiltinArrWebhookCapabilityID = "arr-webhook"
builtinArrWebhookDisplayName = "Sonarr/Radarr Webhook"
)
// BuiltinArrWebhookSource returns the host-built-in ARR webhook source
// identity offered by the Add-source picker alongside installed plugin
// capabilities.
func BuiltinArrWebhookSource() DiscoveredSource {
return DiscoveredSource{
PluginID: BuiltinArrWebhookPluginID,
CapabilityID: BuiltinArrWebhookCapabilityID,
DisplayName: builtinArrWebhookDisplayName,
}
}
// IsBuiltinArrWebhookIdentity reports whether (pluginID, capabilityID) is the
// built-in ARR webhook source identity.
func IsBuiltinArrWebhookIdentity(pluginID, capabilityID string) bool {
return pluginID == BuiltinArrWebhookPluginID && capabilityID == BuiltinArrWebhookCapabilityID
}
// DiscoveredSource identifies one installed scan_source.v1 capability instance,
// enriched with the metadata the Add-source picker needs (plugin id + a
// human-friendly display name).
@@ -26,6 +53,31 @@ type ScanSourceLister interface {
ListScanSources(ctx context.Context) ([]DiscoveredSource, error)
}
// WithBuiltinSources wraps a ScanSourceLister so host-built-in source
// identities appear beside installed plugin capabilities. inner may be nil
// (only the builtins are returned), matching the Service's nil-lister
// tolerance.
func WithBuiltinSources(inner ScanSourceLister, builtins ...DiscoveredSource) ScanSourceLister {
return builtinSourceLister{inner: inner, builtins: builtins}
}
type builtinSourceLister struct {
inner ScanSourceLister
builtins []DiscoveredSource
}
func (l builtinSourceLister) ListScanSources(ctx context.Context) ([]DiscoveredSource, error) {
var out []DiscoveredSource
if l.inner != nil {
discovered, err := l.inner.ListScanSources(ctx)
if err != nil {
return nil, err
}
out = discovered
}
return append(out, l.builtins...), nil
}
// AvailableScanSource is one installed scan_source capability an operator can
// create a source against (the Add-source picker list).
type AvailableScanSource struct {
+42
View File
@@ -31,6 +31,48 @@ func TestListAvailableScanSourcesEnumeratesInstalled(t *testing.T) {
}
}
func TestWithBuiltinSourcesAppendsToInner(t *testing.T) {
inner := fakeLister{sources: []DiscoveredSource{
{PluginID: "sonarr", CapabilityID: "arr-a", DisplayName: "Sonarr"},
}}
lister := WithBuiltinSources(inner, BuiltinArrWebhookSource())
discovered, err := lister.ListScanSources(context.Background())
if err != nil {
t.Fatalf("ListScanSources: %v", err)
}
if len(discovered) != 2 {
t.Fatalf("expected 2 discovered, got %d: %+v", len(discovered), discovered)
}
if discovered[0].PluginID != "sonarr" {
t.Fatalf("plugin entries must pass through first, got %+v", discovered[0])
}
if discovered[1] != BuiltinArrWebhookSource() {
t.Fatalf("expected builtin appended, got %+v", discovered[1])
}
}
func TestWithBuiltinSourcesNilInner(t *testing.T) {
lister := WithBuiltinSources(nil, BuiltinArrWebhookSource())
discovered, err := lister.ListScanSources(context.Background())
if err != nil {
t.Fatalf("ListScanSources: %v", err)
}
if len(discovered) != 1 || discovered[0] != BuiltinArrWebhookSource() {
t.Fatalf("expected only builtin, got %+v", discovered)
}
}
func TestIsBuiltinArrWebhookIdentity(t *testing.T) {
if !IsBuiltinArrWebhookIdentity(BuiltinArrWebhookPluginID, BuiltinArrWebhookCapabilityID) {
t.Fatal("builtin identity must match")
}
if IsBuiltinArrWebhookIdentity("sonarr", "arr") {
t.Fatal("plugin identity must not match builtin")
}
}
func TestListAvailableScanSourcesNilListerEmpty(t *testing.T) {
svc := &Service{lister: nil}
available, err := svc.ListAvailableScanSources(context.Background())
+50 -37
View File
@@ -221,7 +221,7 @@ func (r *Repository) GetConnection(ctx context.Context, id string) (Connection,
// --- Sources ---
const sourceColumns = `id, plugin_id, capability_id, connection_id, enabled,
const sourceColumns = `id, plugin_id, capability_id, connection_id, enabled, delivery_mode,
poll_interval_seconds, path_rewrites, source_config, label, marker, last_run_at, last_error`
func scanSource(row interface{ Scan(...any) error }) (Source, error) {
@@ -229,7 +229,7 @@ func scanSource(row interface{ Scan(...any) error }) (Source, error) {
var pathRewrites []byte
var sourceConfig []byte
if err := row.Scan(&s.ID, &s.PluginID, &s.CapabilityID, &s.ConnectionID,
&s.Enabled, &s.PollIntervalSeconds, &pathRewrites, &sourceConfig, &s.Label, &s.Marker, &s.LastRunAt, &s.LastError); err != nil {
&s.Enabled, &s.DeliveryMode, &s.PollIntervalSeconds, &pathRewrites, &sourceConfig, &s.Label, &s.Marker, &s.LastRunAt, &s.LastError); err != nil {
return Source{}, err
}
rewrites, err := unmarshalPathRewrites(pathRewrites)
@@ -313,6 +313,15 @@ func connectionIDArg(id *string) any {
return nullable(*id)
}
// deliveryModeArg normalizes an unset delivery mode to the poll default so
// pre-webhook callers keep today's behavior.
func deliveryModeArg(mode string) string {
if strings.TrimSpace(mode) == "" {
return DeliveryModePoll
}
return mode
}
// CreateSource inserts a new autoscan source row. One installed scan_source
// capability can back many sources (e.g. one Sonarr plugin install fronting 4
// arr servers, one source per connection), so this is a plain INSERT with a
@@ -329,11 +338,11 @@ func (r *Repository) CreateSource(ctx context.Context, s Source) (Source, error)
}
row := r.pool.QueryRow(ctx, `
INSERT INTO autoscan_sources (
plugin_id, capability_id, connection_id, enabled, poll_interval_seconds, path_rewrites, source_config, label
plugin_id, capability_id, connection_id, enabled, delivery_mode, poll_interval_seconds, path_rewrites, source_config, label
)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
RETURNING `+sourceColumns,
s.PluginID, s.CapabilityID, connectionIDArg(s.ConnectionID), s.Enabled, s.PollIntervalSeconds, rewrites, sourceConfig, s.Label)
s.PluginID, s.CapabilityID, connectionIDArg(s.ConnectionID), s.Enabled, deliveryModeArg(s.DeliveryMode), s.PollIntervalSeconds, rewrites, sourceConfig, s.Label)
out, err := scanSource(row)
if err != nil {
if connID, ok := connectionFKViolation(err, s.ConnectionID); ok {
@@ -361,14 +370,15 @@ func (r *Repository) UpdateSource(ctx context.Context, s Source) (Source, error)
UPDATE autoscan_sources
SET connection_id = $2,
enabled = $3,
poll_interval_seconds = $4,
path_rewrites = $5,
source_config = $6,
label = $7,
delivery_mode = $4,
poll_interval_seconds = $5,
path_rewrites = $6,
source_config = $7,
label = $8,
updated_at = now()
WHERE id = $1
RETURNING `+sourceColumns,
s.ID, connectionIDArg(s.ConnectionID), s.Enabled, s.PollIntervalSeconds, rewrites, sourceConfig, s.Label)
s.ID, connectionIDArg(s.ConnectionID), s.Enabled, deliveryModeArg(s.DeliveryMode), s.PollIntervalSeconds, rewrites, sourceConfig, s.Label)
out, err := scanSource(row)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
@@ -511,8 +521,8 @@ func (r *Repository) RecordError(ctx context.Context, sourceID, msg string) erro
}
const eventColumns = `id, source_id, plugin_id, capability_id, started_at, completed_at,
duration_ms, status, changes_returned, changes_resolved, targets_claimed, scans_created,
scans_reused, scans_suppressed, error_message, marker_before, marker_after`
duration_ms, status, delivery_mode, provider_event_type, changes_returned, changes_resolved,
targets_claimed, scans_created, scans_reused, scans_suppressed, error_message, marker_before, marker_after`
func scanEvent(row interface{ Scan(...any) error }) (Event, error) {
var e Event
@@ -526,6 +536,8 @@ func scanEvent(row interface{ Scan(...any) error }) (Event, error) {
&e.CompletedAt,
&e.DurationMS,
&status,
&e.DeliveryMode,
&e.ProviderEventType,
&e.ChangesReturned,
&e.ChangesResolved,
&e.TargetsClaimed,
@@ -542,28 +554,43 @@ func scanEvent(row interface{ Scan(...any) error }) (Event, error) {
return e, nil
}
const insertEventSQL = `
INSERT INTO autoscan_events (
source_id, plugin_id, capability_id, started_at, completed_at,
duration_ms, status, delivery_mode, provider_event_type, error_message, marker_before
)
VALUES ($1, $2, $3, $4, $4, 0, $5, $6, $7, $8, $9)
RETURNING id`
func (r *Repository) CreateEvent(ctx context.Context, in EventCreate) (int64, error) {
started := in.StartedAt
if started.IsZero() {
started = time.Now()
}
sourceID := strings.TrimSpace(in.SourceID)
if sourceID == "" {
row := r.pool.QueryRow(ctx, `
INSERT INTO autoscan_events (
source_id, plugin_id, capability_id, started_at, completed_at,
duration_ms, status, error_message, marker_before
)
VALUES ($1, $2, $3, $4, $4, 0, $5, $6, $7)
RETURNING id`,
nil,
insertArgs := func(sourceID any) []any {
return []any{
sourceID,
in.PluginID,
in.CapabilityID,
started,
string(EventStatusRunning),
deliveryModeArg(in.DeliveryMode),
in.ProviderEventType,
"",
nullable(in.MarkerBefore),
)
}
}
// Sourceless events (ad hoc triggers) and events that opt out of the
// running-event exclusion (webhook deliveries — no marker window to
// re-read, so they must never be dropped) insert without the advisory
// lock + running check.
if sourceID == "" || in.SkipRunningCheck {
var sourceArg any
if sourceID != "" {
sourceArg = sourceID
}
row := r.pool.QueryRow(ctx, insertEventSQL, insertArgs(sourceArg)...)
var id int64
if err := row.Scan(&id); err != nil {
return 0, fmt.Errorf("create autoscan event: %w", err)
@@ -599,21 +626,7 @@ func (r *Repository) CreateEvent(ctx context.Context, in EventCreate) (int64, er
return 0, fmt.Errorf("check running autoscan event: %w", err)
}
row := tx.QueryRow(ctx, `
INSERT INTO autoscan_events (
source_id, plugin_id, capability_id, started_at, completed_at,
duration_ms, status, error_message, marker_before
)
VALUES ($1, $2, $3, $4, $4, 0, $5, $6, $7)
RETURNING id`,
sourceID,
in.PluginID,
in.CapabilityID,
started,
string(EventStatusRunning),
"",
nullable(in.MarkerBefore),
)
row := tx.QueryRow(ctx, insertEventSQL, insertArgs(sourceID)...)
var id int64
if err := row.Scan(&id); err != nil {
return 0, fmt.Errorf("create autoscan event: %w", err)
+343 -108
View File
@@ -8,6 +8,7 @@ import (
"time"
"github.com/Silo-Server/silo-server/internal/scantrigger"
"github.com/google/uuid"
)
const (
@@ -31,6 +32,12 @@ type Store interface {
RecordError(ctx context.Context, sourceID, msg string) error
CreateEvent(ctx context.Context, event EventCreate) (int64, error)
FinishEvent(ctx context.Context, event EventFinish) error
CreateWebhookDelivery(ctx context.Context, in ChangeIngest) (WebhookDelivery, error)
ClaimWebhookDeliveries(ctx context.Context, workerID string, limit int) ([]WebhookDelivery, error)
CompleteWebhookDelivery(ctx context.Context, id int64, lockedBy string) error
RetryWebhookDelivery(ctx context.Context, id int64, lockedBy string, delay time.Duration, msg string) error
RecordWebhookError(ctx context.Context, sourceID, msg string) error
ClearWebhookError(ctx context.Context, sourceID string) error
}
// Resolver maps a Silo-native path to a concrete scan target (library folder).
@@ -149,6 +156,11 @@ func (s *Service) PollOnce(ctx context.Context) error {
now := time.Now()
for _, src := range sources {
// Webhook sources are fed by IngestChanges when the provider POSTs to
// their endpoint; there is nothing to poll and no plugin to invoke.
if src.DeliveryMode == DeliveryModeWebhook {
continue
}
// Honor the per-source poll interval as a "poll at most every N seconds"
// floor: the global task fires at the default cadence, so a source with a
// longer interval is skipped until enough time has elapsed.
@@ -205,119 +217,342 @@ func (s *Service) PollOnce(ctx context.Context) error {
continue // do NOT advance marker
}
rewritten := rewriteChanges(changes, src.PathRewrites)
targets, claimed, resolvedAny, stats := s.resolveAndClaim(ctx, rewritten, ttl)
if len(targets) > maxAutoscanTargetsPerPoll {
collapsed := collapseTargetsToLibraryScans(targets)
slog.WarnContext(ctx, "autoscan: collapsed large scan target batch to library scans", "component", "autoscan",
"source_id", src.ID,
"targets", len(targets),
"collapsed_targets", len(collapsed),
"limit", maxAutoscanTargetsPerPoll,
)
targets = collapsed
}
var enqueue EnqueueResult
if len(targets) > 0 {
var eerr error
enqueue, eerr = s.enqueueScanTargets(ctx, targets, eventID)
if eerr != nil {
s.releaseClaims(ctx, claimed)
slog.WarnContext(ctx, "autoscan: enqueue failed", "component", "autoscan", "source_id", src.ID, "err", eerr)
s.finishEvent(ctx, eventID, EventFinish{
Status: EventStatusError,
ChangesReturned: len(changes),
ChangesResolved: stats.ChangesResolved,
TargetsClaimed: stats.TargetsClaimed,
ScansSuppressed: stats.Suppressed,
ErrorMessage: eerr.Error(),
MarkerAfter: marker,
})
continue // do NOT advance marker
}
}
// Advancing the marker is what tells the provider "I've consumed up to
// here". Advance it ONLY when the work it represents is genuinely done:
// - provider returned ZERO paths → nothing to do, advance normally.
// - returned paths AND ≥1 resolved → enqueued (above), advance.
// - returned paths, resolved but all
// suppressed (recently scanned /
// debounced) → work is effectively done; advance.
// - ANY resolve attempt failed INTERNALLY (resolver/database fault —
// possibly transient), whether or not other paths resolved → hold the
// marker and record the error; a later poll re-reads the same window
// once the fault clears. Advancing would silently skip the failed
// paths. Targets that DID resolve were already enqueued above; the
// re-read at worst re-scans them, which is safe.
// - returned paths AND NOTHING resolved → every path is outside Silo's
// libraries (RequestError) — a benign, expected condition for
// whole-volume filesystem watchers (e.g. CephFS), which observe
// folders that are not registered as Silo libraries. Holding here
// would pin the marker and permanently stall autoscan, so advance;
// finish the event as "unresolved" so the condition stays visible in
// poll history.
// (Some-but-not-all resolving still advances when the unresolved
// remainder is merely outside Silo's libraries.)
//
// NOTE: len(targets)==0 alone is NOT misconfiguration — paths can resolve
// yet be fully suppressed. Gate on resolvedAny, not targets.
if stats.TransientErrors > 0 {
msg := fmt.Sprintf("%d resolve attempt(s) failed internally (%d of %d path(s) resolved) — holding marker to retry", stats.TransientErrors, stats.ChangesResolved, len(changes))
if rerr := s.store.RecordError(ctx, src.ID, msg); rerr != nil {
slog.WarnContext(ctx, "autoscan: record error failed", "component", "autoscan", "source_id", src.ID, "err", rerr)
}
s.finishEvent(ctx, eventID, EventFinish{
Status: EventStatusError,
ChangesReturned: len(changes),
ChangesResolved: stats.ChangesResolved,
TargetsClaimed: stats.TargetsClaimed,
ScansCreated: enqueue.Created,
ScansReused: enqueue.Reused,
ScansSuppressed: stats.Suppressed,
ErrorMessage: msg,
MarkerAfter: marker,
})
continue // do NOT advance marker
}
status := EventStatusSuccess
var statusMsg string
if len(changes) > 0 && !resolvedAny {
status = EventStatusUnresolved
statusMsg = fmt.Sprintf("returned %d path(s) but none matched a Silo library folder — advanced past them", len(changes))
slog.WarnContext(ctx, "autoscan: returned paths matched no library folder — advancing marker",
"source_id", src.ID, "changes", len(changes))
}
if aerr := s.store.AdvanceMarker(ctx, src.ID, next); aerr != nil {
slog.WarnContext(ctx, "autoscan: advance marker failed", "component", "autoscan", "source_id", src.ID, "err", aerr)
s.finishEvent(ctx, eventID, EventFinish{
Status: EventStatusError,
ChangesReturned: len(changes),
ChangesResolved: stats.ChangesResolved,
TargetsClaimed: stats.TargetsClaimed,
ScansCreated: enqueue.Created,
ScansReused: enqueue.Reused,
ScansSuppressed: stats.Suppressed,
ErrorMessage: aerr.Error(),
MarkerAfter: marker,
})
continue
}
s.finishEvent(ctx, eventID, EventFinish{
Status: status,
ChangesReturned: len(changes),
ChangesResolved: stats.ChangesResolved,
TargetsClaimed: stats.TargetsClaimed,
ScansCreated: enqueue.Created,
ScansReused: enqueue.Reused,
ScansSuppressed: stats.Suppressed,
ErrorMessage: statusMsg,
MarkerAfter: next,
// consumeSourceChanges finishes the event and does all per-source error
// logging/recording; per-source failures never abort the poll loop.
_, _ = s.consumeSourceChanges(ctx, src, changes, consumeOptions{
EventID: eventID,
TTL: ttl,
Marker: marker,
NextMarker: next,
AdvanceMarker: true,
})
}
return nil
}
// consumeOptions parameterizes the shared consume path for its two callers.
// Marker/NextMarker/AdvanceMarker apply to poll-mode consumption only: webhook
// deliveries carry no marker window, never advance one, and hold nothing on
// failure (arr retries the delivery instead).
type consumeOptions struct {
EventID int64
TTL time.Duration
Marker string // poll: the window's opening marker, held on failure
NextMarker string // poll: the provider's next marker
AdvanceMarker bool // poll: true; webhook: false
}
// consumeResult reports what one consume pass did, for callers that surface
// the outcome (webhook delivery responses).
type consumeResult struct {
Status EventStatus
Enqueue EnqueueResult
Stats resolveStats
ResolvedAny bool
}
// consumeSourceChanges is the shared back half of both delivery modes: it
// rewrites raw provider paths, resolves and claims scan targets, enqueues
// them, decides the event status, and finishes the event. The returned error
// reports a genuine failure (enqueue fault or transient resolve fault); the
// poll loop ignores it (already logged/recorded per source), webhook ingestion
// propagates it so the delivery can be retried by the sender.
func (s *Service) consumeSourceChanges(ctx context.Context, src Source, changes []Change, opts consumeOptions) (consumeResult, error) {
rewritten := rewriteChanges(changes, src.PathRewrites)
targets, claimed, resolvedAny, stats := s.resolveAndClaim(ctx, rewritten, opts.TTL)
result := consumeResult{Stats: stats, ResolvedAny: resolvedAny}
if len(targets) > maxAutoscanTargetsPerPoll {
collapsed := collapseTargetsToLibraryScans(targets)
slog.WarnContext(ctx, "autoscan: collapsed large scan target batch to library scans", "component", "autoscan",
"source_id", src.ID,
"targets", len(targets),
"collapsed_targets", len(collapsed),
"limit", maxAutoscanTargetsPerPoll,
)
targets = collapsed
}
if len(targets) > 0 {
enqueue, eerr := s.enqueueScanTargets(ctx, targets, opts.EventID)
result.Enqueue = enqueue
if eerr != nil {
s.releaseClaims(ctx, claimed)
slog.WarnContext(ctx, "autoscan: enqueue failed", "component", "autoscan", "source_id", src.ID, "err", eerr)
result.Status = EventStatusError
s.finishEvent(ctx, opts.EventID, EventFinish{
Status: EventStatusError,
ChangesReturned: len(changes),
ChangesResolved: stats.ChangesResolved,
TargetsClaimed: stats.TargetsClaimed,
ScansSuppressed: stats.Suppressed,
ErrorMessage: eerr.Error(),
MarkerAfter: opts.Marker,
})
return result, eerr // do NOT advance marker
}
}
// Advancing the marker is what tells the provider "I've consumed up to
// here". Advance it ONLY when the work it represents is genuinely done:
// - provider returned ZERO paths → nothing to do, advance normally.
// - returned paths AND ≥1 resolved → enqueued (above), advance.
// - returned paths, resolved but all
// suppressed (recently scanned /
// debounced) → work is effectively done; advance.
// - ANY resolve attempt failed INTERNALLY (resolver/database fault —
// possibly transient), whether or not other paths resolved → hold the
// marker and record the error; a later poll re-reads the same window
// once the fault clears. Advancing would silently skip the failed
// paths. Targets that DID resolve were already enqueued above; the
// re-read at worst re-scans them, which is safe.
// - returned paths AND NOTHING resolved → every path is outside Silo's
// libraries (RequestError) — a benign, expected condition for
// whole-volume filesystem watchers (e.g. CephFS), which observe
// folders that are not registered as Silo libraries. Holding here
// would pin the marker and permanently stall autoscan, so advance;
// finish the event as "unresolved" so the condition stays visible in
// poll history.
// (Some-but-not-all resolving still advances when the unresolved
// remainder is merely outside Silo's libraries.)
//
// Webhook deliveries have no marker: a transient resolve fault still
// finishes the event as error (the sender retries the delivery; a
// duplicate is at worst suppressed), and the unresolved case is the same
// benign "paths outside Silo's libraries" signal.
//
// NOTE: len(targets)==0 alone is NOT misconfiguration — paths can resolve
// yet be fully suppressed. Gate on resolvedAny, not targets.
if stats.TransientErrors > 0 {
msg := fmt.Sprintf("%d resolve attempt(s) failed internally (%d of %d path(s) resolved)", stats.TransientErrors, stats.ChangesResolved, len(changes))
if opts.AdvanceMarker {
msg += " — holding marker to retry"
}
if rerr := s.store.RecordError(ctx, src.ID, msg); rerr != nil {
slog.WarnContext(ctx, "autoscan: record error failed", "component", "autoscan", "source_id", src.ID, "err", rerr)
}
result.Status = EventStatusError
s.finishEvent(ctx, opts.EventID, EventFinish{
Status: EventStatusError,
ChangesReturned: len(changes),
ChangesResolved: stats.ChangesResolved,
TargetsClaimed: stats.TargetsClaimed,
ScansCreated: result.Enqueue.Created,
ScansReused: result.Enqueue.Reused,
ScansSuppressed: stats.Suppressed,
ErrorMessage: msg,
MarkerAfter: opts.Marker,
})
return result, errors.New(msg) // do NOT advance marker
}
status := EventStatusSuccess
var statusMsg string
if len(changes) > 0 && !resolvedAny {
status = EventStatusUnresolved
statusMsg = fmt.Sprintf("returned %d path(s) but none matched a Silo library folder", len(changes))
if opts.AdvanceMarker {
statusMsg += " — advanced past them"
slog.WarnContext(ctx, "autoscan: returned paths matched no library folder — advancing marker",
"source_id", src.ID, "changes", len(changes))
} else {
slog.WarnContext(ctx, "autoscan: webhook paths matched no library folder",
"source_id", src.ID, "changes", len(changes))
}
}
if opts.AdvanceMarker {
if aerr := s.store.AdvanceMarker(ctx, src.ID, opts.NextMarker); aerr != nil {
slog.WarnContext(ctx, "autoscan: advance marker failed", "component", "autoscan", "source_id", src.ID, "err", aerr)
result.Status = EventStatusError
s.finishEvent(ctx, opts.EventID, EventFinish{
Status: EventStatusError,
ChangesReturned: len(changes),
ChangesResolved: stats.ChangesResolved,
TargetsClaimed: stats.TargetsClaimed,
ScansCreated: result.Enqueue.Created,
ScansReused: result.Enqueue.Reused,
ScansSuppressed: stats.Suppressed,
ErrorMessage: aerr.Error(),
MarkerAfter: opts.Marker,
})
return result, aerr
}
}
result.Status = status
s.finishEvent(ctx, opts.EventID, EventFinish{
Status: status,
ChangesReturned: len(changes),
ChangesResolved: stats.ChangesResolved,
TargetsClaimed: stats.TargetsClaimed,
ScansCreated: result.Enqueue.Created,
ScansReused: result.Enqueue.Reused,
ScansSuppressed: stats.Suppressed,
ErrorMessage: statusMsg,
MarkerAfter: opts.NextMarker,
})
return result, nil
}
// ChangeIngest is one webhook delivery's worth of changes for a source.
type ChangeIngest struct {
SourceID string
ProviderEventType string
Changes []Change
ReceivedAt time.Time
}
// IngestResult summarizes what a webhook delivery produced.
type IngestResult struct {
Enqueued int
Suppressed int
Unresolved bool
// Pending reports that the delivery was accepted durably but its immediate
// ingest failed. The retry task will process it again inside Silo.
Pending bool
}
var errWebhookDeliveryDisabled = errors.New("autoscan: webhook delivery disabled")
const (
webhookRetryBaseDelay = 5 * time.Second
webhookRetryMaxDelay = 10 * time.Minute
)
func webhookRetryDelay(attempt int) time.Duration {
if attempt < 1 {
attempt = 1
}
delay := webhookRetryBaseDelay
for i := 1; i < attempt && delay < webhookRetryMaxDelay; i++ {
delay *= 2
if delay >= webhookRetryMaxDelay {
return webhookRetryMaxDelay
}
}
return delay
}
// IngestChanges durably records a webhook delivery before attempting the same
// rewrite/resolve/suppress/enqueue/event pipeline as polling. A transient
// processing failure leaves the delivery queued for an internal retry and is
// surfaced as Pending rather than depending on Sonarr/Radarr to replay it.
func (s *Service) IngestChanges(ctx context.Context, in ChangeIngest) (IngestResult, error) {
delivery, err := s.store.CreateWebhookDelivery(ctx, in)
if err != nil {
return IngestResult{}, err
}
return s.processWebhookDelivery(ctx, delivery), nil
}
// RetryPendingWebhookDeliveries claims and consumes a bounded batch of durable
// deliveries. Repository leases make concurrent nodes safe; individual ingest
// failures are rescheduled and do not abort the rest of the batch.
func (s *Service) RetryPendingWebhookDeliveries(ctx context.Context, limit int) (int, error) {
deliveries, err := s.store.ClaimWebhookDeliveries(ctx, uuid.NewString(), limit)
if err != nil {
return 0, err
}
for _, delivery := range deliveries {
s.processWebhookDelivery(ctx, delivery)
}
return len(deliveries), nil
}
func (s *Service) processWebhookDelivery(ctx context.Context, delivery WebhookDelivery) IngestResult {
result, ingestErr := s.ingestChangesNow(ctx, ChangeIngest{
SourceID: delivery.SourceID,
ProviderEventType: delivery.ProviderEventType,
Changes: delivery.Changes,
ReceivedAt: delivery.ReceivedAt,
})
if ingestErr == nil {
if err := s.store.CompleteWebhookDelivery(ctx, delivery.ID, delivery.LockedBy); err != nil {
result.Pending = true
slog.WarnContext(ctx, "autoscan: complete webhook delivery failed", "component", "autoscan", "delivery_id", delivery.ID, "err", err)
} else {
if err := s.store.ClearWebhookError(ctx, delivery.SourceID); err != nil {
slog.WarnContext(ctx, "autoscan: clear webhook error failed", "component", "autoscan", "source_id", delivery.SourceID, "err", err)
}
}
return result
}
if errors.Is(ingestErr, errWebhookDeliveryDisabled) {
// A source/global disable is a pause, not a reason to discard work that
// was already accepted while enabled. Keep it pending without surfacing a
// delivery error and try again after the operator re-enables Autoscan.
result.Pending = true
if err := s.store.RetryWebhookDelivery(ctx, delivery.ID, delivery.LockedBy, time.Minute, ""); err != nil {
slog.WarnContext(ctx, "autoscan: pause webhook retry failed", "component", "autoscan", "delivery_id", delivery.ID, "err", err)
}
return result
}
result.Pending = true
if err := s.store.RetryWebhookDelivery(
ctx,
delivery.ID,
delivery.LockedBy,
webhookRetryDelay(delivery.AttemptCount),
ingestErr.Error(),
); err != nil {
// The durable row remains leased and becomes reclaimable when the lease
// expires, so a bookkeeping failure here still cannot lose the delivery.
slog.WarnContext(ctx, "autoscan: schedule webhook retry failed", "component", "autoscan", "delivery_id", delivery.ID, "err", err)
}
if err := s.store.RecordWebhookError(ctx, delivery.SourceID, ingestErr.Error()); err != nil {
slog.WarnContext(ctx, "autoscan: record webhook error failed", "component", "autoscan", "source_id", delivery.SourceID, "err", err)
}
return result
}
// ingestChangesNow performs one webhook consume attempt. It is deliberately
// separate from IngestChanges so retries do not create nested delivery rows.
func (s *Service) ingestChangesNow(ctx context.Context, in ChangeIngest) (IngestResult, error) {
settings, err := s.store.GetSettings(ctx)
if err != nil {
return IngestResult{}, err
}
if !settings.Enabled {
return IngestResult{}, errWebhookDeliveryDisabled
}
src, err := s.store.GetSource(ctx, in.SourceID)
if err != nil {
return IngestResult{}, err
}
if src.DeliveryMode != DeliveryModeWebhook {
return IngestResult{}, fmt.Errorf("autoscan: source %s is not a webhook source", src.ID)
}
if !src.Enabled {
return IngestResult{}, errWebhookDeliveryDisabled
}
startedAt := in.ReceivedAt
if startedAt.IsZero() {
startedAt = time.Now()
}
// Event bookkeeping failure does not block the scan work (same resilience
// as polling): eventID 0 makes finishEvent a no-op and enqueues without an
// event link.
eventID, err := s.store.CreateEvent(ctx, EventCreate{
SourceID: src.ID,
PluginID: src.PluginID,
CapabilityID: src.CapabilityID,
StartedAt: startedAt,
DeliveryMode: DeliveryModeWebhook,
ProviderEventType: in.ProviderEventType,
SkipRunningCheck: true,
})
if err != nil {
slog.WarnContext(ctx, "autoscan: create webhook event failed", "component", "autoscan", "source_id", src.ID, "err", err)
eventID = 0
}
result, cerr := s.consumeSourceChanges(ctx, src, in.Changes, consumeOptions{
EventID: eventID,
TTL: time.Duration(settings.DebounceSeconds) * time.Second,
})
return IngestResult{
Enqueued: result.Enqueue.Created + result.Enqueue.Reused,
Suppressed: result.Stats.Suppressed,
Unresolved: result.Status == EventStatusUnresolved,
}, cerr
}
func (s *Service) enqueueScanTargets(ctx context.Context, targets []scantrigger.Target, eventID int64) (EnqueueResult, error) {
var result EnqueueResult
if len(targets) == 0 {
+220
View File
@@ -0,0 +1,220 @@
package autoscan
import (
"context"
"errors"
"testing"
"github.com/Silo-Server/silo-server/internal/scantrigger"
)
func webhookTestStore() *fakeStore {
return &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{
ID: "s1",
PluginID: BuiltinArrWebhookPluginID,
CapabilityID: BuiltinArrWebhookCapabilityID,
Enabled: true,
DeliveryMode: DeliveryModeWebhook,
}},
}
}
func TestIngestChangesEnqueuesLikePolling(t *testing.T) {
store := webhookTestStore()
prov := &fakeProvider{}
q := &recordingQueuer{}
svc := newService(store, prov, q, allowSuppressor{})
result, err := svc.IngestChanges(context.Background(), ChangeIngest{
SourceID: "s1",
ProviderEventType: "Download",
Changes: []Change{
{SourcePath: "/mnt/media/Show/Season 01/e01.mkv", Scope: ChangeScopeFile},
{SourcePath: "/mnt/media/Show/Season 01/e02.mkv", Scope: ChangeScopeFile},
},
})
if err != nil {
t.Fatalf("IngestChanges: %v", err)
}
if result.Enqueued != 2 || result.Suppressed != 0 || result.Unresolved {
t.Fatalf("unexpected result: %+v", result)
}
if len(q.enqueued) != 2 {
t.Fatalf("enqueued = %+v, want 2 file targets", q.enqueued)
}
if prov.calls != 0 {
t.Fatalf("provider must never be invoked for webhook ingestion, got %d calls", prov.calls)
}
if len(store.advanced) != 0 {
t.Fatalf("webhook ingestion must not advance markers, got %+v", store.advanced)
}
if len(store.createdEvents) != 1 {
t.Fatalf("created events = %+v, want 1", store.createdEvents)
}
created := store.createdEvents[0]
if created.DeliveryMode != DeliveryModeWebhook || created.ProviderEventType != "Download" || !created.SkipRunningCheck {
t.Fatalf("webhook event create misconfigured: %+v", created)
}
if len(store.events) != 1 || store.events[0].Status != EventStatusSuccess {
t.Fatalf("finished events = %+v, want one success", store.events)
}
if store.events[0].MarkerAfter != "" {
t.Fatalf("webhook events must not carry markers, got %q", store.events[0].MarkerAfter)
}
if len(store.deliveries) != 1 || len(store.completed) != 1 {
t.Fatalf("durable lifecycle deliveries=%d completed=%v", len(store.deliveries), store.completed)
}
}
type failOnceResolver struct {
calls int
}
func (r *failOnceResolver) Resolve(ctx context.Context, req scantrigger.Request) (*scantrigger.Target, error) {
r.calls++
if r.calls == 1 {
return nil, errors.New("temporary resolver failure")
}
return (fakeResolver{}).Resolve(ctx, req)
}
func (r *failOnceResolver) ResolveMissingSubtree(ctx context.Context, path, trigger string) (*scantrigger.Target, error) {
return (fakeResolver{}).ResolveMissingSubtree(ctx, path, trigger)
}
func (r *failOnceResolver) ResolveVanishedPath(ctx context.Context, path, trigger string) (*scantrigger.Target, error) {
return (fakeResolver{}).ResolveVanishedPath(ctx, path, trigger)
}
func TestRetryPendingWebhookDeliveriesRecoversTransientFailure(t *testing.T) {
store := webhookTestStore()
queue := &recordingQueuer{}
resolver := &failOnceResolver{}
svc := NewService(store, &fakeProvider{}, passthroughConnRes{}, resolver, queue, allowSuppressor{}, nil)
result, err := svc.IngestChanges(context.Background(), ChangeIngest{
SourceID: "s1",
ProviderEventType: "Download",
Changes: []Change{{SourcePath: "/mnt/media/Movie/movie.mkv", Scope: ChangeScopeFile}},
})
if err != nil || !result.Pending {
t.Fatalf("initial result = %+v, err = %v, want pending", result, err)
}
processed, err := svc.RetryPendingWebhookDeliveries(context.Background(), 10)
if err != nil {
t.Fatalf("RetryPendingWebhookDeliveries: %v", err)
}
if processed != 1 || len(queue.enqueued) != 1 {
t.Fatalf("processed=%d enqueued=%d, want 1", processed, len(queue.enqueued))
}
if len(store.completed) != 1 {
t.Fatalf("completed = %v, want retried delivery", store.completed)
}
if store.webhookErrors["s1"] != "" {
t.Fatalf("webhook error not cleared: %q", store.webhookErrors["s1"])
}
}
func TestIngestChangesRejectsPollSource(t *testing.T) {
store := webhookTestStore()
store.sources[0].DeliveryMode = DeliveryModePoll
svc := newService(store, &fakeProvider{}, &recordingQueuer{}, allowSuppressor{})
if _, err := svc.ingestChangesNow(context.Background(), ChangeIngest{SourceID: "s1"}); err == nil {
t.Fatal("webhook consume must reject non-webhook sources")
}
}
func TestIngestChangesUnresolvedPathsAreBenign(t *testing.T) {
store := webhookTestStore()
svc := NewService(store, &fakeProvider{}, passthroughConnRes{}, unresolvableResolver{}, &recordingQueuer{}, allowSuppressor{}, nil)
result, err := svc.IngestChanges(context.Background(), ChangeIngest{
SourceID: "s1",
ProviderEventType: "Download",
Changes: []Change{{SourcePath: "/outside/library/file.mkv", Scope: ChangeScopeFile}},
})
if err != nil {
t.Fatalf("unresolved paths must not be an error: %v", err)
}
if !result.Unresolved || result.Enqueued != 0 {
t.Fatalf("unexpected result: %+v", result)
}
if len(store.events) != 1 || store.events[0].Status != EventStatusUnresolved {
t.Fatalf("finished events = %+v, want one unresolved", store.events)
}
}
func TestIngestChangesTransientResolveFailureQueuesRetry(t *testing.T) {
store := webhookTestStore()
svc := NewService(store, &fakeProvider{}, passthroughConnRes{}, transientFailureResolver{}, &recordingQueuer{}, allowSuppressor{}, nil)
result, err := svc.IngestChanges(context.Background(), ChangeIngest{
SourceID: "s1",
ProviderEventType: "Download",
Changes: []Change{{SourcePath: "/mnt/media/Movie/movie.mkv", Scope: ChangeScopeFile}},
})
if err != nil {
t.Fatalf("durably accepted delivery must not depend on sender retry: %v", err)
}
if !result.Pending {
t.Fatalf("result = %+v, want pending retry", result)
}
if len(store.events) != 1 || store.events[0].Status != EventStatusError {
t.Fatalf("finished events = %+v, want one error", store.events)
}
if store.recorded["s1"] == "" {
t.Fatal("transient failure must be recorded on the source")
}
if len(store.retried) != 1 || store.retried[0].SourceID != "s1" {
t.Fatalf("durable retries = %+v, want source s1", store.retried)
}
if store.webhookErrors["s1"] == "" {
t.Fatal("transient failure must be visible on the webhook endpoint")
}
}
func TestIngestChangesSuppressedDuplicates(t *testing.T) {
store := webhookTestStore()
q := &recordingQueuer{}
svc := NewService(store, &fakeProvider{}, passthroughConnRes{}, fakeResolver{}, q, denySuppressor{}, nil)
result, err := svc.IngestChanges(context.Background(), ChangeIngest{
SourceID: "s1",
ProviderEventType: "Download",
Changes: []Change{{SourcePath: "/mnt/media/Movie/movie.mkv", Scope: ChangeScopeFile}},
})
if err != nil {
t.Fatalf("IngestChanges: %v", err)
}
if result.Suppressed != 1 || result.Enqueued != 0 || result.Unresolved {
t.Fatalf("unexpected result: %+v", result)
}
if len(q.enqueued) != 0 {
t.Fatalf("suppressed delivery must not enqueue, got %+v", q.enqueued)
}
if len(store.events) != 1 || store.events[0].Status != EventStatusSuccess {
t.Fatalf("finished events = %+v, want one success", store.events)
}
}
func TestPollOnceSkipsWebhookSources(t *testing.T) {
store := webhookTestStore()
prov := &fakeProvider{paths: map[string][]string{BuiltinArrWebhookCapabilityID: {"/mnt/media/Movie/movie.mkv"}}}
q := &recordingQueuer{}
svc := newService(store, prov, q, allowSuppressor{})
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if prov.calls != 0 {
t.Fatalf("provider calls = %d, want 0 for webhook sources", prov.calls)
}
if len(store.createdEvents) != 0 || len(q.enqueued) != 0 {
t.Fatalf("webhook sources must be skipped by polling entirely (events=%d, enqueued=%d)",
len(store.createdEvents), len(q.enqueued))
}
}
+56
View File
@@ -22,6 +22,10 @@ type fakeStore struct {
createdEvents []EventCreate
events []EventFinish
createEventErr error
deliveries []WebhookDelivery
retried []WebhookDelivery
completed []int64
webhookErrors map[string]string
}
func (f *fakeStore) GetSettings(context.Context) (Settings, error) { return f.settings, nil }
@@ -64,6 +68,58 @@ func (f *fakeStore) FinishEvent(_ context.Context, event EventFinish) error {
f.events = append(f.events, event)
return nil
}
func (f *fakeStore) CreateWebhookDelivery(_ context.Context, in ChangeIngest) (WebhookDelivery, error) {
delivery := WebhookDelivery{
ID: int64(len(f.deliveries) + 1),
SourceID: in.SourceID,
ProviderEventType: in.ProviderEventType,
Changes: append([]Change(nil), in.Changes...),
ReceivedAt: in.ReceivedAt,
AttemptCount: 1,
LockedBy: "immediate",
}
f.deliveries = append(f.deliveries, delivery)
return delivery, nil
}
func (f *fakeStore) ClaimWebhookDeliveries(_ context.Context, workerID string, limit int) ([]WebhookDelivery, error) {
if limit > len(f.retried) {
limit = len(f.retried)
}
out := append([]WebhookDelivery(nil), f.retried[:limit]...)
f.retried = f.retried[limit:]
for i := range out {
out[i].AttemptCount++
out[i].LockedBy = workerID
}
return out, nil
}
func (f *fakeStore) CompleteWebhookDelivery(_ context.Context, id int64, _ string) error {
f.completed = append(f.completed, id)
return nil
}
func (f *fakeStore) RetryWebhookDelivery(_ context.Context, id int64, lockedBy string, _ time.Duration, _ string) error {
for _, delivery := range f.deliveries {
if delivery.ID == id {
delivery.LockedBy = lockedBy
f.retried = append(f.retried, delivery)
break
}
}
return nil
}
func (f *fakeStore) RecordWebhookError(_ context.Context, sourceID, msg string) error {
if f.webhookErrors == nil {
f.webhookErrors = map[string]string{}
}
f.webhookErrors[sourceID] = msg
return nil
}
func (f *fakeStore) ClearWebhookError(_ context.Context, sourceID string) error {
if f.webhookErrors != nil {
delete(f.webhookErrors, sourceID)
}
return nil
}
// fakeProvider implements ScanSourceProvider, keyed by capability ID.
type fakeProvider struct {
+70 -24
View File
@@ -41,10 +41,18 @@ const (
// Change is one raw provider/source-namespace path returned by a scan_source
// plugin. The host applies PathRewrites before resolving the changed path.
type Change struct {
SourcePath string
Scope ChangeScope
SourcePath string `json:"source_path"`
Scope ChangeScope `json:"scope"`
}
// DeliveryMode selects how a source's changes reach the host: the host polls
// the plugin (`poll`, the default), or the provider POSTs to a Silo webhook
// endpoint (`webhook`) and the plugin is never invoked.
const (
DeliveryModePoll = "poll"
DeliveryModeWebhook = "webhook"
)
// Source ties a scan_source plugin capability to a connection plus the
// host-owned scheduling/bookkeeping state.
type Source struct {
@@ -53,6 +61,7 @@ type Source struct {
CapabilityID string
ConnectionID *string // nil until an operator binds a connection
Enabled bool
DeliveryMode string // DeliveryModePoll or DeliveryModeWebhook
PollIntervalSeconds *int // nil => use settings default
PathRewrites []PathRewrite // host-owned raw->Silo prefix rewrites
SourceConfig map[string]string
@@ -62,6 +71,34 @@ type Source struct {
LastError *string
}
// WebhookEndpoint is the admin-visible state of a source's webhook endpoint.
// The plaintext token and its lookup hash never live on this struct; creation
// and rotation return the token separately, and redisplay goes through
// RevealWebhookToken.
type WebhookEndpoint struct {
SourceID string
SecretSuffix string
CreatedAt time.Time
RotatedAt *time.Time
LastReceivedAt *time.Time
LastErrorAt *time.Time
LastErrorMessage string
}
// WebhookDelivery is one durably accepted ARR webhook delivery waiting to be
// consumed. LockedBy is an opaque lease owner; repository completion/failure
// updates are ownership-guarded so an expired worker cannot finalize a delivery
// that another node reclaimed.
type WebhookDelivery struct {
ID int64
SourceID string
ProviderEventType string
Changes []Change
ReceivedAt time.Time
AttemptCount int
LockedBy string
}
type EventStatus string
const (
@@ -72,31 +109,40 @@ const (
)
type Event struct {
ID int64
SourceID *string
PluginID string
CapabilityID string
StartedAt time.Time
CompletedAt time.Time
DurationMS int64
Status EventStatus
ChangesReturned int
ChangesResolved int
TargetsClaimed int
ScansCreated int
ScansReused int
ScansSuppressed int
ErrorMessage string
MarkerBefore *string
MarkerAfter *string
ID int64
SourceID *string
PluginID string
CapabilityID string
StartedAt time.Time
CompletedAt time.Time
DurationMS int64
Status EventStatus
DeliveryMode string
ProviderEventType string
ChangesReturned int
ChangesResolved int
TargetsClaimed int
ScansCreated int
ScansReused int
ScansSuppressed int
ErrorMessage string
MarkerBefore *string
MarkerAfter *string
}
type EventCreate struct {
SourceID string
PluginID string
CapabilityID string
StartedAt time.Time
MarkerBefore string
SourceID string
PluginID string
CapabilityID string
StartedAt time.Time
MarkerBefore string
DeliveryMode string // "" defaults to DeliveryModePoll
ProviderEventType string
// SkipRunningCheck bypasses the running-event single-flight exclusion.
// Poll cycles may be skipped when an event is already running (their
// marker window is re-read next cycle); webhook deliveries carry no
// marker, so dropping one loses it forever — they always set this.
SkipRunningCheck bool
}
type EventFinish struct {
+150
View File
@@ -0,0 +1,150 @@
package autoscan
import (
"context"
"encoding/json"
"fmt"
"time"
"github.com/google/uuid"
)
const webhookDeliveryLease = 5 * time.Minute
func postgresInterval(d time.Duration) string {
return fmt.Sprintf("%f seconds", d.Seconds())
}
func scanWebhookDelivery(row interface{ Scan(...any) error }) (WebhookDelivery, error) {
var (
delivery WebhookDelivery
changes []byte
)
if err := row.Scan(
&delivery.ID,
&delivery.SourceID,
&delivery.ProviderEventType,
&changes,
&delivery.ReceivedAt,
&delivery.AttemptCount,
&delivery.LockedBy,
); err != nil {
return WebhookDelivery{}, err
}
if err := json.Unmarshal(changes, &delivery.Changes); err != nil {
return WebhookDelivery{}, fmt.Errorf("decode autoscan webhook delivery changes: %w", err)
}
return delivery, nil
}
// CreateWebhookDelivery durably accepts a delivery and leases it to the caller
// for an immediate ingest attempt. If the caller exits before finalizing it, a
// retry worker can reclaim it after webhookDeliveryLease.
func (r *Repository) CreateWebhookDelivery(ctx context.Context, in ChangeIngest) (WebhookDelivery, error) {
changes, err := json.Marshal(in.Changes)
if err != nil {
return WebhookDelivery{}, fmt.Errorf("encode autoscan webhook delivery changes: %w", err)
}
receivedAt := in.ReceivedAt
if receivedAt.IsZero() {
receivedAt = time.Now()
}
lockedBy := uuid.NewString()
row := r.pool.QueryRow(ctx, `
INSERT INTO autoscan_webhook_deliveries (
source_id, provider_event_type, changes, received_at,
attempt_count, next_attempt_at, locked_at, locked_by
)
VALUES ($1, $2, $3, $4, 1, now(), now(), $5)
RETURNING id, source_id, provider_event_type, changes, received_at,
attempt_count, locked_by`,
in.SourceID, in.ProviderEventType, changes, receivedAt, lockedBy)
delivery, err := scanWebhookDelivery(row)
if err != nil {
return WebhookDelivery{}, fmt.Errorf("create autoscan webhook delivery: %w", err)
}
return delivery, nil
}
// ClaimWebhookDeliveries leases due or abandoned deliveries to one worker.
// FOR UPDATE SKIP LOCKED keeps concurrent nodes from ingesting the same row.
func (r *Repository) ClaimWebhookDeliveries(ctx context.Context, workerID string, limit int) ([]WebhookDelivery, error) {
if limit <= 0 {
return []WebhookDelivery{}, nil
}
rows, err := r.pool.Query(ctx, `
WITH due AS (
SELECT id
FROM autoscan_webhook_deliveries
WHERE next_attempt_at <= now()
AND (locked_at IS NULL OR locked_at < now() - $2::interval)
ORDER BY next_attempt_at ASC, id ASC
LIMIT $1
FOR UPDATE SKIP LOCKED
)
UPDATE autoscan_webhook_deliveries d
SET attempt_count = d.attempt_count + 1,
locked_at = now(),
locked_by = $3,
updated_at = now()
FROM due
WHERE d.id = due.id
RETURNING d.id, d.source_id, d.provider_event_type, d.changes,
d.received_at, d.attempt_count, d.locked_by`,
limit, postgresInterval(webhookDeliveryLease), workerID)
if err != nil {
return nil, fmt.Errorf("claim autoscan webhook deliveries: %w", err)
}
defer rows.Close()
deliveries := make([]WebhookDelivery, 0, limit)
for rows.Next() {
delivery, err := scanWebhookDelivery(rows)
if err != nil {
return nil, err
}
deliveries = append(deliveries, delivery)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("iterate autoscan webhook deliveries: %w", err)
}
return deliveries, nil
}
// CompleteWebhookDelivery removes a successfully consumed delivery. The lease
// owner guard prevents stale workers from deleting a row another node reclaimed.
func (r *Repository) CompleteWebhookDelivery(ctx context.Context, id int64, lockedBy string) error {
tag, err := r.pool.Exec(ctx, `
DELETE FROM autoscan_webhook_deliveries
WHERE id = $1 AND locked_by = $2`, id, lockedBy)
if err != nil {
return fmt.Errorf("complete autoscan webhook delivery: %w", err)
}
if tag.RowsAffected() == 0 {
return fmt.Errorf("%w: webhook delivery %d lease", ErrNotFound, id)
}
return nil
}
// RetryWebhookDelivery releases a failed delivery back to the durable queue
// after a bounded delay. The lease owner guard makes late workers harmless.
func (r *Repository) RetryWebhookDelivery(ctx context.Context, id int64, lockedBy string, delay time.Duration, msg string) error {
if delay < 0 {
delay = 0
}
tag, err := r.pool.Exec(ctx, `
UPDATE autoscan_webhook_deliveries
SET next_attempt_at = now() + $3::interval,
locked_at = NULL,
locked_by = '',
last_error = $4,
updated_at = now()
WHERE id = $1 AND locked_by = $2`,
id, lockedBy, postgresInterval(delay), truncateUTF8(msg, maxLastErrorLen))
if err != nil {
return fmt.Errorf("retry autoscan webhook delivery: %w", err)
}
if tag.RowsAffected() == 0 {
return fmt.Errorf("%w: webhook delivery %d lease", ErrNotFound, id)
}
return nil
}
+278
View File
@@ -0,0 +1,278 @@
package autoscan
import (
"context"
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"errors"
"fmt"
"strings"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgconn"
"github.com/Silo-Server/silo-server/internal/secret"
)
// webhookSecretAAD binds an autoscan_webhook_endpoints secret_ref ciphertext to
// its owning source id.
func webhookSecretAAD(sourceID string) string {
return secret.RowAAD("autoscan_webhook_endpoints", "secret_ref", sourceID)
}
// webhookSecretSuffixLen is how many trailing token characters are stored for
// UI/debug display.
const webhookSecretSuffixLen = 6
// newWebhookToken generates a URL-safe bearer token plus its SHA-256 lookup
// hash and display suffix. The token is 32 random bytes; plain SHA-256 is
// sufficient for lookup because the preimage is high-entropy random data.
func newWebhookToken() (token, hash, suffix string, err error) {
raw := make([]byte, 32)
if _, err := rand.Read(raw); err != nil {
return "", "", "", fmt.Errorf("generate webhook token: %w", err)
}
token = base64.RawURLEncoding.EncodeToString(raw)
hash = hashWebhookToken(token)
return token, hash, token[len(token)-webhookSecretSuffixLen:], nil
}
// hashWebhookToken maps a raw bearer token to its secret_hash lookup value.
func hashWebhookToken(token string) string {
sum := sha256.Sum256([]byte(token))
return hex.EncodeToString(sum[:])
}
const webhookEndpointColumns = `source_id, secret_suffix, created_at, rotated_at,
last_received_at, last_error_at, last_error_message`
func scanWebhookEndpoint(row interface{ Scan(...any) error }) (WebhookEndpoint, error) {
var e WebhookEndpoint
if err := row.Scan(&e.SourceID, &e.SecretSuffix, &e.CreatedAt, &e.RotatedAt,
&e.LastReceivedAt, &e.LastErrorAt, &e.LastErrorMessage); err != nil {
return WebhookEndpoint{}, err
}
return e, nil
}
// CreateWebhookEndpoint creates a source's webhook endpoint and returns it with
// the plaintext bearer token. When the endpoint already exists it is returned
// unchanged with an empty token (the token is only handed out on create/rotate;
// redisplay goes through RevealWebhookToken). An unknown source maps to
// ErrNotFound.
func (r *Repository) CreateWebhookEndpoint(ctx context.Context, sourceID string) (WebhookEndpoint, string, error) {
token, hash, suffix, err := newWebhookToken()
if err != nil {
return WebhookEndpoint{}, "", err
}
ref, err := r.cipher.Encrypt(token, webhookSecretAAD(sourceID))
if err != nil {
return WebhookEndpoint{}, "", fmt.Errorf("encrypt webhook token: %w", err)
}
row := r.pool.QueryRow(ctx, `
INSERT INTO autoscan_webhook_endpoints (source_id, secret_hash, secret_ref, secret_suffix)
VALUES ($1, $2, $3, $4)
ON CONFLICT (source_id) DO NOTHING
RETURNING `+webhookEndpointColumns,
sourceID, hash, ref, suffix)
endpoint, err := scanWebhookEndpoint(row)
if err == nil {
return endpoint, token, nil
}
if errors.Is(err, pgx.ErrNoRows) {
// Conflict: the endpoint already exists — return it, discard the
// unused freshly generated token.
existing, gerr := r.GetWebhookEndpoint(ctx, sourceID)
if gerr != nil {
return WebhookEndpoint{}, "", gerr
}
return existing, "", nil
}
var pgErr *pgconn.PgError
if errors.As(err, &pgErr) && pgErr.Code == "23503" {
return WebhookEndpoint{}, "", fmt.Errorf("%w: source %s", ErrNotFound, sourceID)
}
return WebhookEndpoint{}, "", fmt.Errorf("create autoscan webhook endpoint: %w", err)
}
// RotateWebhookEndpoint replaces the endpoint's bearer token, invalidating the
// old URL immediately, and returns the new plaintext token. An unknown
// source/endpoint maps to ErrNotFound.
func (r *Repository) RotateWebhookEndpoint(ctx context.Context, sourceID string) (WebhookEndpoint, string, error) {
token, hash, suffix, err := newWebhookToken()
if err != nil {
return WebhookEndpoint{}, "", err
}
ref, err := r.cipher.Encrypt(token, webhookSecretAAD(sourceID))
if err != nil {
return WebhookEndpoint{}, "", fmt.Errorf("encrypt webhook token: %w", err)
}
row := r.pool.QueryRow(ctx, `
UPDATE autoscan_webhook_endpoints
SET secret_hash = $2, secret_ref = $3, secret_suffix = $4, rotated_at = now()
WHERE source_id = $1
RETURNING `+webhookEndpointColumns,
sourceID, hash, ref, suffix)
endpoint, err := scanWebhookEndpoint(row)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return WebhookEndpoint{}, "", fmt.Errorf("%w: webhook endpoint for source %s", ErrNotFound, sourceID)
}
return WebhookEndpoint{}, "", fmt.Errorf("rotate autoscan webhook endpoint: %w", err)
}
return endpoint, token, nil
}
// DeleteWebhookEndpoint removes a source's webhook endpoint; future deliveries
// to the old URL return not found. An unknown endpoint maps to ErrNotFound.
func (r *Repository) DeleteWebhookEndpoint(ctx context.Context, sourceID string) error {
tag, err := r.pool.Exec(ctx, `DELETE FROM autoscan_webhook_endpoints WHERE source_id = $1`, sourceID)
if err != nil {
return fmt.Errorf("delete autoscan webhook endpoint: %w", err)
}
if tag.RowsAffected() == 0 {
return fmt.Errorf("%w: webhook endpoint for source %s", ErrNotFound, sourceID)
}
return nil
}
// GetWebhookEndpoint loads a source's webhook endpoint state. An unknown
// endpoint maps to ErrNotFound.
func (r *Repository) GetWebhookEndpoint(ctx context.Context, sourceID string) (WebhookEndpoint, error) {
row := r.pool.QueryRow(ctx, `SELECT `+webhookEndpointColumns+`
FROM autoscan_webhook_endpoints WHERE source_id = $1`, sourceID)
endpoint, err := scanWebhookEndpoint(row)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return WebhookEndpoint{}, fmt.Errorf("%w: webhook endpoint for source %s", ErrNotFound, sourceID)
}
return WebhookEndpoint{}, fmt.Errorf("get autoscan webhook endpoint: %w", err)
}
return endpoint, nil
}
// ListWebhookEndpoints returns every webhook endpoint (one batched query for
// admin source listings).
func (r *Repository) ListWebhookEndpoints(ctx context.Context) ([]WebhookEndpoint, error) {
rows, err := r.pool.Query(ctx, `SELECT `+webhookEndpointColumns+`
FROM autoscan_webhook_endpoints`)
if err != nil {
return nil, fmt.Errorf("list autoscan webhook endpoints: %w", err)
}
defer rows.Close()
var out []WebhookEndpoint
for rows.Next() {
endpoint, err := scanWebhookEndpoint(rows)
if err != nil {
return nil, err
}
out = append(out, endpoint)
}
return out, rows.Err()
}
// RevealWebhookToken decrypts the stored bearer token for admin redisplay.
func (r *Repository) RevealWebhookToken(ctx context.Context, sourceID string) (string, error) {
var ref string
err := r.pool.QueryRow(ctx, `SELECT secret_ref
FROM autoscan_webhook_endpoints WHERE source_id = $1`, sourceID).Scan(&ref)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return "", fmt.Errorf("%w: webhook endpoint for source %s", ErrNotFound, sourceID)
}
return "", fmt.Errorf("reveal autoscan webhook token: %w", err)
}
token, err := r.cipher.Decrypt(ref, webhookSecretAAD(sourceID))
if err != nil {
return "", fmt.Errorf("decrypt webhook token: %w", err)
}
return token, nil
}
// ResolveWebhookToken maps a raw delivery token to its owning source and
// endpoint. Lookup is constant-shape: the token is hashed and matched against
// secret_hash; unknown, deleted, and never-existed tokens are indistinguishable
// and all map to ErrNotFound.
func (r *Repository) ResolveWebhookToken(ctx context.Context, token string) (Source, WebhookEndpoint, error) {
token = strings.TrimSpace(token)
if token == "" {
return Source{}, WebhookEndpoint{}, fmt.Errorf("%w: webhook token", ErrNotFound)
}
row := r.pool.QueryRow(ctx, `
SELECT s.id, s.plugin_id, s.capability_id, s.connection_id, s.enabled, s.delivery_mode,
s.poll_interval_seconds, s.path_rewrites, s.source_config, s.label, s.marker, s.last_run_at, s.last_error,
e.source_id, e.secret_suffix, e.created_at, e.rotated_at,
e.last_received_at, e.last_error_at, e.last_error_message
FROM autoscan_webhook_endpoints e
JOIN autoscan_sources s ON s.id = e.source_id
WHERE e.secret_hash = $1`,
hashWebhookToken(token))
var (
src Source
endpoint WebhookEndpoint
pathRewrites []byte
sourceConfig []byte
)
if err := row.Scan(&src.ID, &src.PluginID, &src.CapabilityID, &src.ConnectionID,
&src.Enabled, &src.DeliveryMode, &src.PollIntervalSeconds, &pathRewrites, &sourceConfig,
&src.Label, &src.Marker, &src.LastRunAt, &src.LastError,
&endpoint.SourceID, &endpoint.SecretSuffix, &endpoint.CreatedAt, &endpoint.RotatedAt,
&endpoint.LastReceivedAt, &endpoint.LastErrorAt, &endpoint.LastErrorMessage); err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return Source{}, WebhookEndpoint{}, fmt.Errorf("%w: webhook token", ErrNotFound)
}
return Source{}, WebhookEndpoint{}, fmt.Errorf("resolve autoscan webhook token: %w", err)
}
rewrites, err := unmarshalPathRewrites(pathRewrites)
if err != nil {
return Source{}, WebhookEndpoint{}, err
}
src.PathRewrites = rewrites
config, err := unmarshalSourceConfig(sourceConfig)
if err != nil {
return Source{}, WebhookEndpoint{}, err
}
src.SourceConfig = config
return src, endpoint, nil
}
// TouchWebhookReceived stamps the endpoint's last valid delivery time.
func (r *Repository) TouchWebhookReceived(ctx context.Context, sourceID string) error {
_, err := r.pool.Exec(ctx, `
UPDATE autoscan_webhook_endpoints
SET last_received_at = now()
WHERE source_id = $1`, sourceID)
if err != nil {
return fmt.Errorf("touch autoscan webhook received: %w", err)
}
return nil
}
// RecordWebhookError stores the endpoint's most recent parse/ingest failure
// (length-bounded; the caller sanitizes payload-derived content).
func (r *Repository) RecordWebhookError(ctx context.Context, sourceID, msg string) error {
_, err := r.pool.Exec(ctx, `
UPDATE autoscan_webhook_endpoints
SET last_error_at = now(), last_error_message = $2
WHERE source_id = $1`, sourceID, truncateUTF8(msg, maxLastErrorLen))
if err != nil {
return fmt.Errorf("record autoscan webhook error: %w", err)
}
return nil
}
// ClearWebhookError removes stale delivery failure state after a queued retry
// succeeds. last_received_at remains the time the provider actually delivered
// the webhook rather than being rewritten as a processing timestamp.
func (r *Repository) ClearWebhookError(ctx context.Context, sourceID string) error {
_, err := r.pool.Exec(ctx, `
UPDATE autoscan_webhook_endpoints
SET last_error_at = NULL, last_error_message = ''
WHERE source_id = $1`, sourceID)
if err != nil {
return fmt.Errorf("clear autoscan webhook error: %w", err)
}
return nil
}
+248
View File
@@ -0,0 +1,248 @@
package autoscan
import (
"context"
"errors"
"os"
"strings"
"testing"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/Silo-Server/silo-server/internal/secret"
)
// newWebhookDBTest connects to SILO_TEST_DATABASE_URL (skipping when unset or
// unmigrated) and returns a repository plus a fresh webhook-mode source row.
func newWebhookDBTest(t *testing.T) (context.Context, *Repository, Source) {
t.Helper()
dsn := os.Getenv("SILO_TEST_DATABASE_URL")
if dsn == "" {
t.Skip("SILO_TEST_DATABASE_URL is not set")
}
ctx := context.Background()
pool, err := pgxpool.New(ctx, dsn)
if err != nil {
t.Fatalf("connect test database: %v", err)
}
t.Cleanup(pool.Close)
var tableName *string
if err := pool.QueryRow(ctx, `SELECT to_regclass('public.autoscan_webhook_endpoints')::text`).Scan(&tableName); err != nil {
t.Fatalf("check autoscan_webhook_endpoints table: %v", err)
}
if tableName == nil || *tableName == "" {
t.Skip("test database has not applied the autoscan webhook intake migration")
}
cipher, err := secret.New([]byte("0123456789abcdef0123456789abcdef"))
if err != nil {
t.Fatalf("new cipher: %v", err)
}
repo := NewRepository(pool, cipher)
src, err := repo.CreateSource(ctx, Source{
PluginID: BuiltinArrWebhookPluginID,
CapabilityID: BuiltinArrWebhookCapabilityID,
Enabled: true,
DeliveryMode: DeliveryModeWebhook,
})
if err != nil {
t.Fatalf("create source: %v", err)
}
t.Cleanup(func() { _ = repo.DeleteSource(ctx, src.ID) })
return ctx, repo, src
}
func TestWebhookEndpointLifecycle(t *testing.T) {
ctx, repo, src := newWebhookDBTest(t)
if src.DeliveryMode != DeliveryModeWebhook {
t.Fatalf("expected webhook delivery mode persisted, got %q", src.DeliveryMode)
}
endpoint, token, err := repo.CreateWebhookEndpoint(ctx, src.ID)
if err != nil {
t.Fatalf("create endpoint: %v", err)
}
if token == "" {
t.Fatal("create must return a plaintext token")
}
if !strings.HasSuffix(token, endpoint.SecretSuffix) || len(endpoint.SecretSuffix) != webhookSecretSuffixLen {
t.Fatalf("suffix %q does not match token tail", endpoint.SecretSuffix)
}
// Idempotent create: existing endpoint returned, no new token.
again, token2, err := repo.CreateWebhookEndpoint(ctx, src.ID)
if err != nil {
t.Fatalf("re-create endpoint: %v", err)
}
if token2 != "" || again.SecretSuffix != endpoint.SecretSuffix {
t.Fatalf("re-create must return existing endpoint with empty token, got token=%q suffix=%q", token2, again.SecretSuffix)
}
resolvedSrc, _, err := repo.ResolveWebhookToken(ctx, token)
if err != nil {
t.Fatalf("resolve token: %v", err)
}
if resolvedSrc.ID != src.ID {
t.Fatalf("resolved source %s, want %s", resolvedSrc.ID, src.ID)
}
revealed, err := repo.RevealWebhookToken(ctx, src.ID)
if err != nil {
t.Fatalf("reveal token: %v", err)
}
if revealed != token {
t.Fatal("revealed token must round-trip the original")
}
rotated, newToken, err := repo.RotateWebhookEndpoint(ctx, src.ID)
if err != nil {
t.Fatalf("rotate endpoint: %v", err)
}
if newToken == "" || newToken == token {
t.Fatal("rotate must return a fresh token")
}
if rotated.RotatedAt == nil {
t.Fatal("rotate must stamp rotated_at")
}
if _, _, err := repo.ResolveWebhookToken(ctx, token); !errors.Is(err, ErrNotFound) {
t.Fatalf("old token must be invalidated, got %v", err)
}
if _, _, err := repo.ResolveWebhookToken(ctx, newToken); err != nil {
t.Fatalf("new token must resolve: %v", err)
}
if err := repo.TouchWebhookReceived(ctx, src.ID); err != nil {
t.Fatalf("touch received: %v", err)
}
if err := repo.RecordWebhookError(ctx, src.ID, "boom"); err != nil {
t.Fatalf("record error: %v", err)
}
got, err := repo.GetWebhookEndpoint(ctx, src.ID)
if err != nil {
t.Fatalf("get endpoint: %v", err)
}
if got.LastReceivedAt == nil || got.LastErrorAt == nil || got.LastErrorMessage != "boom" {
t.Fatalf("bookkeeping fields not persisted: %+v", got)
}
// Cascade: deleting the source removes the endpoint.
if err := repo.DeleteSource(ctx, src.ID); err != nil {
t.Fatalf("delete source: %v", err)
}
if _, err := repo.GetWebhookEndpoint(ctx, src.ID); !errors.Is(err, ErrNotFound) {
t.Fatalf("endpoint must cascade-delete with source, got %v", err)
}
if err := repo.DeleteWebhookEndpoint(ctx, src.ID); !errors.Is(err, ErrNotFound) {
t.Fatalf("delete of missing endpoint must be ErrNotFound, got %v", err)
}
}
func TestCreateEventSkipRunningCheck(t *testing.T) {
ctx, repo, src := newWebhookDBTest(t)
first, err := repo.CreateEvent(ctx, EventCreate{
SourceID: src.ID,
PluginID: src.PluginID,
CapabilityID: src.CapabilityID,
DeliveryMode: DeliveryModeWebhook,
ProviderEventType: "Download",
SkipRunningCheck: true,
})
if err != nil {
t.Fatalf("create first event: %v", err)
}
// Poll-style creation is excluded while the first event runs...
if _, err := repo.CreateEvent(ctx, EventCreate{
SourceID: src.ID,
PluginID: src.PluginID,
CapabilityID: src.CapabilityID,
}); !errors.Is(err, ErrPollAlreadyRunning) {
t.Fatalf("poll create must hit running exclusion, got %v", err)
}
// ...but a webhook delivery is never dropped.
second, err := repo.CreateEvent(ctx, EventCreate{
SourceID: src.ID,
PluginID: src.PluginID,
CapabilityID: src.CapabilityID,
DeliveryMode: DeliveryModeWebhook,
ProviderEventType: "Download",
SkipRunningCheck: true,
})
if err != nil {
t.Fatalf("concurrent webhook event must not be dropped: %v", err)
}
for _, id := range []int64{first, second} {
if err := repo.FinishEvent(ctx, EventFinish{ID: id, Status: EventStatusSuccess}); err != nil {
t.Fatalf("finish event %d: %v", id, err)
}
}
events, err := repo.ListEvents(ctx, EventListFilter{SourceID: src.ID})
if err != nil {
t.Fatalf("list events: %v", err)
}
if len(events) != 2 {
t.Fatalf("expected 2 events, got %d", len(events))
}
for _, e := range events {
if e.Event.DeliveryMode != DeliveryModeWebhook || e.Event.ProviderEventType != "Download" {
t.Fatalf("event metadata not persisted: %+v", e.Event)
}
}
}
func TestWebhookDeliveryDurableRetryLifecycle(t *testing.T) {
ctx, repo, src := newWebhookDBTest(t)
var tableName *string
if err := repo.pool.QueryRow(ctx, `SELECT to_regclass('public.autoscan_webhook_deliveries')::text`).Scan(&tableName); err != nil {
t.Fatalf("check autoscan_webhook_deliveries table: %v", err)
}
if tableName == nil || *tableName == "" {
t.Skip("test database has not applied the autoscan webhook delivery queue migration")
}
delivery, err := repo.CreateWebhookDelivery(ctx, ChangeIngest{
SourceID: src.ID,
ProviderEventType: "Download",
Changes: []Change{{SourcePath: "/data/movie.mkv", Scope: ChangeScopeFile}},
})
if err != nil {
t.Fatalf("create delivery: %v", err)
}
if delivery.ID == 0 || delivery.AttemptCount != 1 || delivery.LockedBy == "" {
t.Fatalf("created delivery = %+v", delivery)
}
if err := repo.RetryWebhookDelivery(ctx, delivery.ID, delivery.LockedBy, 0, "temporary"); err != nil {
t.Fatalf("schedule retry: %v", err)
}
claimed, err := repo.ClaimWebhookDeliveries(ctx, "worker-2", 10)
if err != nil {
t.Fatalf("claim delivery: %v", err)
}
if len(claimed) != 1 || claimed[0].ID != delivery.ID || claimed[0].AttemptCount != 2 || claimed[0].LockedBy != "worker-2" {
t.Fatalf("claimed = %+v", claimed)
}
if len(claimed[0].Changes) != 1 || claimed[0].Changes[0].SourcePath != "/data/movie.mkv" {
t.Fatalf("changes did not round-trip: %+v", claimed[0].Changes)
}
if err := repo.CompleteWebhookDelivery(ctx, delivery.ID, "stale-worker"); !errors.Is(err, ErrNotFound) {
t.Fatalf("stale lease completion must be rejected, got %v", err)
}
if err := repo.CompleteWebhookDelivery(ctx, delivery.ID, "worker-2"); err != nil {
t.Fatalf("complete delivery: %v", err)
}
claimed, err = repo.ClaimWebhookDeliveries(ctx, "worker-3", 10)
if err != nil {
t.Fatalf("claim after complete: %v", err)
}
if len(claimed) != 0 {
t.Fatalf("completed delivery was reclaimed: %+v", claimed)
}
}
+4 -1
View File
@@ -9,7 +9,7 @@ import (
type Rate struct {
RequestsPerSecond float64
RequestsPerMinute float64
Burst int // max burst for token bucket (in-memory); ignored by Redis
Burst int // immediate burst allowance (token bucket or Redis second window)
}
// AllowResult contains the result of a rate limit check.
@@ -72,6 +72,9 @@ func DefaultConfig() Config {
"device_start": {RequestsPerMinute: 20, Burst: 10},
"device_lookup": {RequestsPerMinute: 60, Burst: 20},
"device_poll": {RequestsPerMinute: 120, Burst: 30},
// Public autoscan webhook intake. Generous: arr fires one delivery
// per imported file, so season packs are legitimate bursts.
"autoscan_webhook": {RequestsPerMinute: 60, Burst: 30},
},
}
}
+22 -20
View File
@@ -4,6 +4,7 @@ import (
"context"
"fmt"
"log/slog"
"math"
"time"
"github.com/redis/go-redis/v9"
@@ -55,6 +56,25 @@ type RedisLimiter struct {
client *redis.Client
}
// redisWindowLimits maps the shared rate model onto Redis fixed windows. The
// second window must honor Burst just like the in-memory token bucket; using
// only int(RequestsPerSecond) previously collapsed a 60/minute, burst-30
// webhook limit into one request per second and rejected legitimate batches.
func redisWindowLimits(limit Rate) (secLimit, minLimit, effectiveLimit int) {
secLimit = int(math.Ceil(limit.RequestsPerSecond))
minLimit = int(math.Ceil(limit.RequestsPerMinute))
if limit.Burst > secLimit {
secLimit = limit.Burst
}
if secLimit <= 0 && minLimit > 0 {
secLimit = minLimit
}
if minLimit <= 0 && secLimit > 0 {
minLimit = secLimit * 60
}
return secLimit, minLimit, secLimit
}
// NewRedisLimiter creates a new Redis-backed rate limiter.
func NewRedisLimiter(client *redis.Client) *RedisLimiter {
return &RedisLimiter{client: client}
@@ -68,25 +88,7 @@ func (rl *RedisLimiter) Allow(ctx context.Context, key string, limit Rate) Allow
secKey := fmt.Sprintf("silo:ratelimit:%s:s:%d", key, secTs)
minKey := fmt.Sprintf("silo:ratelimit:%s:m:%d", key, minTs)
secLimit := int(limit.RequestsPerSecond)
minLimit := int(limit.RequestsPerMinute)
// For sub-1 rps rates (e.g. auth endpoints: 5 req/min = 0.083 rps),
// int truncation gives secLimit=0 which makes the Lua script reject
// every request. Use the per-minute limit as the effective limit and
// skip the per-second constraint by setting it to the per-minute limit
// (the per-minute window is the real constraint).
subSecondRate := secLimit == 0 && limit.RequestsPerSecond > 0
if subSecondRate {
secLimit = minLimit
}
// Effective limit for response headers: use per-minute when the rate
// is defined in minutes (sub-second), per-second otherwise.
effectiveLimit := secLimit
if subSecondRate {
effectiveLimit = minLimit
}
secLimit, minLimit, effectiveLimit := redisWindowLimits(limit)
result, err := rateLimitScript.Run(ctx, rl.client,
[]string{secKey, minKey},
@@ -124,7 +126,7 @@ func (rl *RedisLimiter) Allow(ctx context.Context, key string, limit Rate) Allow
}
}
remaining := effectiveLimit - minCount
remaining := minLimit - minCount
if secRemaining := secLimit - secCount; secRemaining < remaining {
remaining = secRemaining
}
+25
View File
@@ -0,0 +1,25 @@
package ratelimit
import "testing"
func TestRedisWindowLimitsHonorBurst(t *testing.T) {
sec, minute, effective := redisWindowLimits(Rate{
RequestsPerSecond: 1,
RequestsPerMinute: 60,
Burst: 30,
})
if sec != 30 || minute != 60 || effective != 30 {
t.Fatalf("limits = (%d, %d, %d), want (30, 60, 30)", sec, minute, effective)
}
}
func TestRedisWindowLimitsKeepMinuteOnlyRatesUsable(t *testing.T) {
sec, minute, _ := redisWindowLimits(Rate{
RequestsPerSecond: 10.0 / 60.0,
RequestsPerMinute: 10,
Burst: 6,
})
if sec != 6 || minute != 10 {
t.Fatalf("limits = (%d, %d), want burst 6 and minute 10", sec, minute)
}
}
@@ -0,0 +1,58 @@
package tasks
import (
"context"
"fmt"
"github.com/Silo-Server/silo-server/internal/taskmanager"
)
const (
autoscanWebhookRetryIntervalMs int64 = 15 * 1000
autoscanWebhookRetryBatch = 100
)
type AutoscanWebhookRetrier interface {
RetryPendingWebhookDeliveries(ctx context.Context, limit int) (int, error)
}
// AutoscanWebhookRetryTask drains the durable webhook inbox independently of
// the scan-source poll cadence. It stays hidden because it is reliability
// plumbing rather than an operator-facing maintenance action.
type AutoscanWebhookRetryTask struct {
retrier AutoscanWebhookRetrier
}
func NewAutoscanWebhookRetryTask(retrier AutoscanWebhookRetrier) *AutoscanWebhookRetryTask {
return &AutoscanWebhookRetryTask{retrier: retrier}
}
func (t *AutoscanWebhookRetryTask) Key() string { return "autoscan_webhook_retry" }
func (t *AutoscanWebhookRetryTask) Name() string { return "Autoscan webhook retry" }
func (t *AutoscanWebhookRetryTask) Description() string {
return "Retry durably accepted Autoscan webhook deliveries"
}
func (t *AutoscanWebhookRetryTask) Category() taskmanager.TaskCategory {
return taskmanager.TaskCategoryLibrary
}
func (t *AutoscanWebhookRetryTask) IsHidden() bool { return true }
func (t *AutoscanWebhookRetryTask) DefaultTriggers() []taskmanager.TriggerConfig {
return []taskmanager.TriggerConfig{{
Type: taskmanager.TriggerTypeInterval,
IntervalMs: autoscanWebhookRetryIntervalMs,
}}
}
func (t *AutoscanWebhookRetryTask) Execute(ctx context.Context, progress taskmanager.ProgressReporter) error {
if t.retrier == nil {
return nil
}
progress.Report(0, "Checking durable webhook deliveries")
processed, err := t.retrier.RetryPendingWebhookDeliveries(ctx, autoscanWebhookRetryBatch)
if err != nil {
return fmt.Errorf("retry autoscan webhook deliveries: %w", err)
}
progress.Report(100, fmt.Sprintf("Processed %d webhook deliveries", processed))
return nil
}
@@ -0,0 +1,39 @@
package tasks
import (
"context"
"errors"
"testing"
)
type fakeAutoscanWebhookRetrier struct {
limit int
processed int
err error
}
func (f *fakeAutoscanWebhookRetrier) RetryPendingWebhookDeliveries(_ context.Context, limit int) (int, error) {
f.limit = limit
return f.processed, f.err
}
func TestAutoscanWebhookRetryTask(t *testing.T) {
retrier := &fakeAutoscanWebhookRetrier{processed: 3}
task := NewAutoscanWebhookRetryTask(retrier)
if err := task.Execute(context.Background(), noopProgressReporter{}); err != nil {
t.Fatalf("Execute: %v", err)
}
if retrier.limit != autoscanWebhookRetryBatch {
t.Fatalf("limit = %d, want %d", retrier.limit, autoscanWebhookRetryBatch)
}
if !task.IsHidden() || task.DefaultTriggers()[0].IntervalMs != autoscanWebhookRetryIntervalMs {
t.Fatalf("unexpected task configuration: hidden=%v triggers=%+v", task.IsHidden(), task.DefaultTriggers())
}
}
func TestAutoscanWebhookRetryTaskPropagatesClaimFailure(t *testing.T) {
task := NewAutoscanWebhookRetryTask(&fakeAutoscanWebhookRetrier{err: errors.New("database unavailable")})
if err := task.Execute(context.Background(), noopProgressReporter{}); err == nil {
t.Fatal("Execute must report retry queue failures")
}
}
@@ -0,0 +1,44 @@
-- +goose Up
-- +goose StatementBegin
ALTER TABLE autoscan_sources
ADD COLUMN delivery_mode text NOT NULL DEFAULT 'poll'
CONSTRAINT autoscan_sources_delivery_mode_check
CHECK (delivery_mode = ANY (ARRAY['poll'::text, 'webhook'::text]));
-- +goose StatementEnd
-- +goose StatementBegin
CREATE TABLE autoscan_webhook_endpoints (
source_id uuid PRIMARY KEY
REFERENCES autoscan_sources(id) ON DELETE CASCADE,
secret_hash text UNIQUE NOT NULL,
secret_ref text NOT NULL,
secret_suffix text NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
rotated_at timestamptz,
last_received_at timestamptz,
last_error_at timestamptz,
last_error_message text NOT NULL DEFAULT ''
);
-- +goose StatementEnd
-- +goose StatementBegin
ALTER TABLE autoscan_events
ADD COLUMN delivery_mode text NOT NULL DEFAULT 'poll',
ADD COLUMN provider_event_type text NOT NULL DEFAULT '';
-- +goose StatementEnd
-- +goose Down
-- +goose StatementBegin
ALTER TABLE autoscan_events
DROP COLUMN provider_event_type,
DROP COLUMN delivery_mode;
-- +goose StatementEnd
-- +goose StatementBegin
DROP TABLE autoscan_webhook_endpoints;
-- +goose StatementEnd
-- +goose StatementBegin
ALTER TABLE autoscan_sources
DROP COLUMN delivery_mode;
-- +goose StatementEnd
@@ -0,0 +1,26 @@
-- +goose Up
-- +goose StatementBegin
CREATE TABLE autoscan_webhook_deliveries (
id bigserial PRIMARY KEY,
source_id uuid NOT NULL
REFERENCES autoscan_sources(id) ON DELETE CASCADE,
provider_event_type text NOT NULL,
changes jsonb NOT NULL,
received_at timestamptz NOT NULL,
attempt_count integer NOT NULL DEFAULT 1,
next_attempt_at timestamptz NOT NULL DEFAULT now(),
locked_at timestamptz,
locked_by text NOT NULL DEFAULT '',
last_error text NOT NULL DEFAULT '',
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX idx_autoscan_webhook_deliveries_due
ON autoscan_webhook_deliveries (next_attempt_at, id);
-- +goose StatementEnd
-- +goose Down
-- +goose StatementBegin
DROP TABLE autoscan_webhook_deliveries;
-- +goose StatementEnd
+15
View File
@@ -2049,23 +2049,35 @@ export interface AutoscanPathRewrite {
to: string;
}
export type AutoscanDeliveryMode = "poll" | "webhook";
export type AutoscanWebhookProvider = "auto" | "sonarr" | "radarr";
export interface AutoscanSource {
id: string;
plugin_id: string;
capability_id: string;
connection_id: string | null;
enabled: boolean;
delivery_mode: AutoscanDeliveryMode;
poll_interval_seconds: number | null;
last_run_at: string | null;
last_error: string | null;
path_rewrites: AutoscanPathRewrite[];
source_config: Record<string, string>;
label: string;
webhook_configured: boolean;
webhook_url?: string;
webhook_secret_suffix?: string;
webhook_last_received_at?: string;
webhook_last_error_at?: string;
webhook_last_error_message?: string;
}
export interface AutoscanSourceInput {
connection_id: string | null;
enabled: boolean;
delivery_mode?: AutoscanDeliveryMode;
poll_interval_seconds: number | null;
path_rewrites?: AutoscanPathRewrite[];
source_config?: Record<string, string>;
@@ -2091,6 +2103,7 @@ export interface AutoscanSourceCreateInput {
capability_id: string;
connection_id?: string | null;
enabled: boolean;
delivery_mode?: AutoscanDeliveryMode;
poll_interval_seconds?: number | null;
path_rewrites: AutoscanPathRewrite[];
source_config?: Record<string, string>;
@@ -2182,6 +2195,8 @@ export interface AutoscanEvent {
completed_at: string;
duration_ms: number;
status: AutoscanEventStatus;
delivery_mode?: AutoscanDeliveryMode;
provider_event_type?: string;
changes_returned: number;
changes_resolved: number;
targets_claimed: number;
+53
View File
@@ -198,6 +198,59 @@ export function useDeleteAutoscanSource() {
});
}
// --- Webhook endpoints ---
export function useCreateAutoscanWebhook() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: (id: string) =>
api<AutoscanSource>(`/admin/autoscan/sources/${encodeURIComponent(id)}/webhook`, {
method: "POST",
}),
onSuccess: () => {
toast.success("Webhook URL created");
queryClient.invalidateQueries({ queryKey: adminKeys.autoscanSources() });
},
onError: (err) => {
toast.error(err instanceof Error ? err.message : "Failed to create webhook URL");
},
});
}
export function useRotateAutoscanWebhook() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: (id: string) =>
api<AutoscanSource>(`/admin/autoscan/sources/${encodeURIComponent(id)}/webhook/rotate`, {
method: "POST",
}),
onSuccess: () => {
toast.success("Webhook URL rotated — update Sonarr/Radarr with the new URL");
queryClient.invalidateQueries({ queryKey: adminKeys.autoscanSources() });
},
onError: (err) => {
toast.error(err instanceof Error ? err.message : "Failed to rotate webhook URL");
},
});
}
export function useDeleteAutoscanWebhook() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: (id: string) =>
api<void>(`/admin/autoscan/sources/${encodeURIComponent(id)}/webhook`, {
method: "DELETE",
}),
onSuccess: () => {
toast.success("Webhook URL deleted");
queryClient.invalidateQueries({ queryKey: adminKeys.autoscanSources() });
},
onError: (err) => {
toast.error(err instanceof Error ? err.message : "Failed to delete webhook URL");
},
});
}
/**
* Test an arr connection. Accepts either an existing connection id, or raw
* credentials (base_url + api_key_ref) / a request integration id for an
+2
View File
@@ -51,12 +51,14 @@ describe("resolveEventSourceName", () => {
capability_id: "arr",
connection_id: "conn-1",
enabled: true,
delivery_mode: "poll",
poll_interval_seconds: null,
last_run_at: null,
last_error: null,
path_rewrites: [],
source_config: {},
label: "",
webhook_configured: false,
};
const lookups: SourceLabelLookups = {
sourceByID: new Map([["src-1", source]]),
+22 -1
View File
@@ -202,6 +202,21 @@ function ScanStatusBadge({ status }: { status: AutoscanScanStatus | ScanRun["sta
);
}
/**
* Marks webhook-delivered events (with the arr event type when known) so they
* read distinctly from poll cycles. Poll events render nothing — poll is the
* default and labeling every row would be noise.
*/
function DeliveryBadge({ event }: { event: AutoscanEvent }) {
if (event.delivery_mode !== "webhook") return null;
return (
<Badge variant="secondary" className="text-xs">
Webhook
{event.provider_event_type ? ` · ${event.provider_event_type}` : ""}
</Badge>
);
}
// Shared desktop table chrome so the queue, scan history, and poll log read as
// one family: clipped rounded border, muted sticky-feeling header band.
function DataTable({ head, children }: { head: ReactNode; children: ReactNode }) {
@@ -606,6 +621,7 @@ function PollEventCard({ event, lookups }: { event: AutoscanEvent; lookups: Sour
<div className="flex flex-wrap items-center gap-2">
<PollStatusBadge status={event.status} />
<span className="font-medium">{pollSourceName(event, lookups)}</span>
<DeliveryBadge event={event} />
</div>
<PollMetricStrip event={event} />
</div>
@@ -662,7 +678,12 @@ function PollEventTable({
<TableCell>
<PollStatusBadge status={event.status} />
</TableCell>
<TableCell className="font-medium">{pollSourceName(event, lookups)}</TableCell>
<TableCell className="font-medium">
<div className="flex flex-wrap items-center gap-1.5">
{pollSourceName(event, lookups)}
<DeliveryBadge event={event} />
</div>
</TableCell>
<TableCell>
<PollMetricStrip event={event} />
{event.error_message ? (
+354 -93
View File
@@ -5,18 +5,22 @@ import {
ChevronDown,
ChevronRight,
Clock,
Copy,
Library as LibraryIcon,
Plus,
RefreshCw,
Trash2,
Webhook,
X,
} from "lucide-react";
import { Link } from "react-router";
import { toast } from "sonner";
import type {
AutoscanPathRewrite,
AutoscanRewriteSuggestions,
AutoscanSource,
AutoscanSourceInput,
AutoscanWebhookProvider,
Library,
} from "@/api/types";
import {
@@ -64,7 +68,9 @@ import {
useAutoscanSources,
useAvailableScanSources,
useCreateAutoscanSource,
useCreateAutoscanWebhook,
useDeleteAutoscanSource,
useRotateAutoscanWebhook,
useUpdateAutoscanSource,
} from "@/hooks/queries/useAutoscan";
import { useAdminLibraries } from "@/hooks/queries/admin/libraries";
@@ -135,6 +141,22 @@ function sourceToRowEdit(source: AutoscanSource): RowEdit {
};
}
const ARR_WEBHOOK_PLUGIN_ID = "silo.autoscan.arr-webhook";
const WEBHOOK_PROVIDER_KEY = "webhook_provider";
function isWebhookSource(source: AutoscanSource): boolean {
return source.delivery_mode === "webhook";
}
function isArrWebhookPlugin(plugin: { plugin_id: string } | undefined): boolean {
return plugin?.plugin_id === ARR_WEBHOOK_PLUGIN_ID;
}
/** Resolve a possibly relative webhook_url against the admin UI's own origin. */
function absoluteWebhookURL(url: string): string {
return url.startsWith("/") ? `${window.location.origin}${url}` : url;
}
const CEPHFS_PLUGIN_ID = "silo.autoscan.cephfs";
const CEPHFS_CAPABILITY_ID = "cephfs";
const DEFAULT_CEPHFS_EXCLUSIONS = [
@@ -687,6 +709,147 @@ function CephFSConfigEditor({
);
}
// ---------------------------------------------------------------------------
// WebhookEndpointSection — webhook-mode replacement for the poll interval
// ---------------------------------------------------------------------------
function WebhookEndpointSection({
source,
provider,
onProviderChange,
isSaving,
}: {
source: AutoscanSource;
provider: AutoscanWebhookProvider;
onProviderChange: (next: AutoscanWebhookProvider) => void;
isSaving: boolean;
}) {
const createWebhook = useCreateAutoscanWebhook();
const rotateWebhook = useRotateAutoscanWebhook();
const [rotateOpen, setRotateOpen] = useState(false);
const url = source.webhook_url ? absoluteWebhookURL(source.webhook_url) : "";
async function copyURL() {
if (!url) return;
try {
await navigator.clipboard.writeText(url);
toast.success("Webhook URL copied");
} catch {
toast.error("Could not copy — select the URL manually");
}
}
return (
<div className="space-y-3">
<div className="space-y-1.5">
<Label className="text-muted-foreground text-xs">Webhook URL</Label>
{source.webhook_configured ? (
<>
<div className="flex items-center gap-1.5">
<Input
readOnly
value={url || `…${source.webhook_secret_suffix ?? ""} (URL unavailable)`}
className="h-8 font-mono text-xs"
aria-label="Webhook delivery URL"
onFocus={(e) => e.currentTarget.select()}
/>
<Button
type="button"
variant="outline"
size="icon-sm"
onClick={copyURL}
disabled={!url}
aria-label="Copy webhook URL"
>
<Copy className="size-3.5" />
</Button>
<Button
type="button"
variant="outline"
size="icon-sm"
onClick={() => setRotateOpen(true)}
disabled={rotateWebhook.isPending}
aria-label="Rotate webhook URL"
title="Replace the URL — the old one stops working immediately"
>
<RefreshCw
className={`size-3.5 ${rotateWebhook.isPending ? "animate-spin" : ""}`}
/>
</Button>
</div>
<p className="text-muted-foreground text-xs">
Paste into Sonarr/Radarr → Settings → Connect → Webhook (On Import, On Rename, On File
Delete).
</p>
</>
) : (
<div className="space-y-1.5">
<Button
type="button"
variant="outline"
size="sm"
disabled={createWebhook.isPending}
onClick={() => createWebhook.mutate(source.id)}
>
<Webhook className="size-3.5" />
{createWebhook.isPending ? "Generating…" : "Generate webhook URL"}
</Button>
<p className="text-muted-foreground text-xs">
Creates the URL Sonarr/Radarr will POST import, rename, and delete events to.
</p>
</div>
)}
</div>
<div className="space-y-1.5">
<Label className="text-muted-foreground text-xs">Provider</Label>
<Select
value={provider}
onValueChange={(v) => onProviderChange(v as AutoscanWebhookProvider)}
disabled={isSaving}
>
<SelectTrigger className="w-[140px]" aria-label="Webhook payload provider">
<SelectValue />
</SelectTrigger>
<SelectContent>
<SelectItem value="auto">Auto</SelectItem>
<SelectItem value="sonarr">Sonarr</SelectItem>
<SelectItem value="radarr">Radarr</SelectItem>
</SelectContent>
</Select>
<p className="text-muted-foreground text-xs">
Auto infers Sonarr vs Radarr from each payload.
</p>
</div>
{/* Rotate confirmation */}
<AlertDialog open={rotateOpen} onOpenChange={setRotateOpen}>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>Rotate webhook URL?</AlertDialogTitle>
<AlertDialogDescription>
The current URL stops working immediately. Sonarr/Radarr keep sending to the old URL
until you paste the new one into their webhook settings.
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>Cancel</AlertDialogCancel>
<AlertDialogAction
onClick={() => {
rotateWebhook.mutate(source.id);
setRotateOpen(false);
}}
>
Rotate
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
</div>
);
}
// ---------------------------------------------------------------------------
// SourceRow helpers
// ---------------------------------------------------------------------------
@@ -814,13 +977,40 @@ function SourceRow({
});
}
function handleProviderChange(next: AutoscanWebhookProvider) {
const sourceConfig = { ...edit.sourceConfig, [WEBHOOK_PROVIDER_KEY]: next };
setEdit((ed) => ({ ...ed, sourceConfig }));
update.mutate({
id: source.id,
body: fullBody({ source_config: normalizeSourceConfig(sourceConfig) }),
});
}
// Whether this source has a bound connection (server-side or pending edit).
// Used to gate the Sync-from-server button, which needs a server to query.
const hasEffectiveConnection = Boolean(source.connection_id) || Boolean(edit.connectionId);
// Status column
const hasError = Boolean(source.last_error);
const hasRun = Boolean(source.last_run_at);
const isWebhook = isWebhookSource(source);
// Status column. Poll sources report last_run_at/last_error; webhook sources
// report their endpoint's delivery bookkeeping instead (deliveries do not
// stamp last_run_at).
const webhookReceivedMs = source.webhook_last_received_at
? new Date(source.webhook_last_received_at).getTime()
: 0;
const webhookErrorMs = source.webhook_last_error_at
? new Date(source.webhook_last_error_at).getTime()
: 0;
const hasError = isWebhook
? webhookErrorMs > 0 && webhookErrorMs >= webhookReceivedMs
: Boolean(source.last_error);
const hasRun = isWebhook ? webhookReceivedMs > 0 : Boolean(source.last_run_at);
const statusErrorMessage = isWebhook
? source.webhook_last_error_message || source.last_error || "Delivery failed"
: (source.last_error ?? "");
const statusTimestamp = isWebhook
? (source.webhook_last_received_at ?? null)
: source.last_run_at;
const connectionSelectClass = layout === "card" ? "!w-full min-w-0 max-w-full" : "w-[200px]";
const statusMessageClass =
layout === "card"
@@ -845,7 +1035,15 @@ function SourceRow({
const sourceIdentity = (
<div className="min-w-0 space-y-1">
<div className="min-w-0 space-y-0.5">
<p className="truncate leading-none font-medium">{resolvedLabel.name}</p>
<p className="flex items-center gap-1.5 truncate leading-none font-medium">
{resolvedLabel.name}
{isWebhook && (
<Badge variant="secondary" className="shrink-0 text-xs">
<Webhook className="size-3" />
Webhook
</Badge>
)}
</p>
<p className="text-muted-foreground text-xs">{resolvedLabel.detail}</p>
</div>
<Input
@@ -864,8 +1062,8 @@ function SourceRow({
<AlertTriangle className="text-destructive size-4 shrink-0" />
<div className="min-w-0 space-y-0.5">
<p className="text-destructive leading-none font-medium">Error</p>
<p className={statusMessageClass} title={source.last_error ?? ""}>
{source.last_error}
<p className={statusMessageClass} title={statusErrorMessage}>
{statusErrorMessage}
</p>
</div>
</div>
@@ -876,39 +1074,23 @@ function SourceRow({
<p className="leading-none font-medium">OK</p>
<p className="text-muted-foreground flex items-center gap-1 text-xs">
<Clock className="size-3" />
{formatRelativeTime(source.last_run_at)}
{formatRelativeTime(statusTimestamp)}
</p>
</div>
</div>
) : (
<span className="text-muted-foreground text-sm">Not run yet</span>
<span className="text-muted-foreground text-sm">
{isWebhook ? "No deliveries yet" : "Not run yet"}
</span>
);
const connectionControl =
source.connection_id === null && !edit.connectionId ? (
<div className="flex max-w-full min-w-0 flex-col gap-2 sm:flex-row sm:items-center">
<Select value="__none__" onValueChange={handleConnectionChange}>
<SelectTrigger
className={connectionSelectClass}
aria-label={`Connection for ${resolvedLabel.name}`}
>
<SelectValue placeholder="No connection" />
</SelectTrigger>
<SelectContent>
<SelectItem value="__none__">— No connection —</SelectItem>
{connectionOptions.map((c) => (
<SelectItem key={c.id} value={c.id}>
{c.name}
</SelectItem>
))}
</SelectContent>
</Select>
<Badge variant="outline" className="text-muted-foreground w-fit shrink-0">
No connection
</Badge>
</div>
) : (
<Select value={edit.connectionId || "__none__"} onValueChange={handleConnectionChange}>
const connectionControl = isWebhook ? (
<span className="text-muted-foreground text-xs">
Not needed — Sonarr/Radarr deliver directly
</span>
) : source.connection_id === null && !edit.connectionId ? (
<div className="flex max-w-full min-w-0 flex-col gap-2 sm:flex-row sm:items-center">
<Select value="__none__" onValueChange={handleConnectionChange}>
<SelectTrigger
className={connectionSelectClass}
aria-label={`Connection for ${resolvedLabel.name}`}
@@ -924,9 +1106,56 @@ function SourceRow({
))}
</SelectContent>
</Select>
);
<Badge variant="outline" className="text-muted-foreground w-fit shrink-0">
No connection
</Badge>
</div>
) : (
<Select value={edit.connectionId || "__none__"} onValueChange={handleConnectionChange}>
<SelectTrigger
className={connectionSelectClass}
aria-label={`Connection for ${resolvedLabel.name}`}
>
<SelectValue placeholder="No connection" />
</SelectTrigger>
<SelectContent>
<SelectItem value="__none__">— No connection —</SelectItem>
{connectionOptions.map((c) => (
<SelectItem key={c.id} value={c.id}>
{c.name}
</SelectItem>
))}
</SelectContent>
</Select>
);
const intervalSettings = (
// Webhook payload paths are arr-side paths and still need host-side
// rewrites in Docker/mount-mismatch setups, so the rewrite editor stays
// visible in both modes.
const rewriteEditor = (
<RewriteEditor
sourceId={source.id}
hasConnection={hasEffectiveConnection}
rewrites={edit.rewrites}
onChange={(next) => setEdit((ed) => ({ ...ed, rewrites: next }))}
onSave={handleRewriteSave}
isSaving={update.isPending}
/>
);
const intervalSettings = isWebhook ? (
<>
<WebhookEndpointSection
source={source}
provider={
(edit.sourceConfig[WEBHOOK_PROVIDER_KEY] as AutoscanWebhookProvider | undefined) ?? "auto"
}
onProviderChange={handleProviderChange}
isSaving={update.isPending}
/>
{rewriteEditor}
</>
) : (
<>
<div className="flex items-center gap-2">
<Input
@@ -947,14 +1176,7 @@ function SourceRow({
<p className="text-destructive mt-1 text-xs">Must be a positive integer.</p>
)}
<p className="text-muted-foreground mt-1 text-xs">{intervalHelp}</p>
<RewriteEditor
sourceId={source.id}
hasConnection={hasEffectiveConnection}
rewrites={edit.rewrites}
onChange={(next) => setEdit((ed) => ({ ...ed, rewrites: next }))}
onSave={handleRewriteSave}
isSaving={update.isPending}
/>
{rewriteEditor}
{isCephFSSource(source) && (
<CephFSConfigEditor
config={edit.sourceConfig}
@@ -983,16 +1205,18 @@ function SourceRow({
</div>
<div className="mt-4 grid min-w-0 gap-4">
<div className="grid min-w-0 gap-1.5">
<Label className="text-muted-foreground text-xs">Connection</Label>
{connectionControl}
</div>
{!isWebhook && (
<div className="grid min-w-0 gap-1.5">
<Label className="text-muted-foreground text-xs">Connection</Label>
{connectionControl}
</div>
)}
<div className="grid min-w-0 grid-cols-[minmax(0,1fr)_auto] items-center gap-3 rounded-md border px-3 py-2">
<div className="min-w-0">
<p className="text-sm font-medium">Enabled</p>
<p className="text-muted-foreground text-xs break-words">
Poll this source for changes
{isWebhook ? "Accept webhook deliveries" : "Poll this source for changes"}
</p>
</div>
<Switch
@@ -1004,12 +1228,16 @@ function SourceRow({
</div>
<div className="grid min-w-0 gap-1.5">
<Label className="text-muted-foreground text-xs">Last run</Label>
<Label className="text-muted-foreground text-xs">
{isWebhook ? "Last delivery" : "Last run"}
</Label>
<div className="min-w-0 overflow-hidden rounded-md border px-3 py-2">{statusNode}</div>
</div>
<div className="grid min-w-0 gap-1.5">
<Label className="text-muted-foreground text-xs">Interval &amp; settings</Label>
<Label className="text-muted-foreground text-xs">
{isWebhook ? "Webhook & settings" : "Interval & settings"}
</Label>
{intervalSettings}
</div>
</div>
@@ -1097,6 +1325,7 @@ function AddSourceDialog({
(p) => pluginKey(p.plugin_id, p.capability_id) === form.pluginKey,
);
const selectedIsCephFS = isCephFSPlugin(selectedPlugin);
const selectedIsWebhook = isArrWebhookPlugin(selectedPlugin);
function close() {
setForm(BLANK_ADD_SOURCE);
@@ -1105,6 +1334,20 @@ function AddSourceDialog({
function handleSubmit() {
if (!selectedPlugin) return;
if (selectedIsWebhook) {
createSource.mutate(
{
plugin_id: selectedPlugin.plugin_id,
capability_id: selectedPlugin.capability_id,
enabled: false,
delivery_mode: "webhook",
path_rewrites: [],
source_config: { [WEBHOOK_PROVIDER_KEY]: "auto" },
},
{ onSuccess: close },
);
return;
}
const connectionId =
form.connectionId && form.connectionId !== "__none__" ? form.connectionId : null;
const raw = form.intervalStr.trim();
@@ -1124,6 +1367,7 @@ function AddSourceDialog({
}
const intervalInvalid =
!selectedIsWebhook &&
form.intervalStr.trim() !== "" &&
(!Number.isInteger(Number(form.intervalStr)) || Number(form.intervalStr) < 1);
const canSubmit = Boolean(selectedPlugin) && !intervalInvalid && !createSource.isPending;
@@ -1184,50 +1428,67 @@ function AddSourceDialog({
</Select>
</div>
<div className="space-y-1.5">
<Label>Connection</Label>
<Select
value={form.connectionId || "__none__"}
onValueChange={(v) => setForm((f) => ({ ...f, connectionId: v }))}
>
<SelectTrigger className="w-full">
<SelectValue placeholder="No connection" />
</SelectTrigger>
<SelectContent>
<SelectItem value="__none__">— No connection —</SelectItem>
{connectionOptions.map((c) => (
<SelectItem key={c.id} value={c.id}>
{c.name}
</SelectItem>
))}
</SelectContent>
</Select>
<p className="text-muted-foreground text-xs">
Optional — bind one if the source needs to reach a server (e.g. Sonarr/Radarr).
Sources that read locally need no connection.
</p>
</div>
<div className="space-y-1.5">
<Label htmlFor="add-source-interval">Poll interval (seconds)</Label>
<div className="flex items-center gap-2">
<Input
id="add-source-interval"
className="w-32"
placeholder="Default"
value={form.intervalStr}
aria-invalid={intervalInvalid}
onChange={(e) => setForm((f) => ({ ...f, intervalStr: e.target.value }))}
/>
<span className="text-muted-foreground text-sm">sec</span>
{selectedIsWebhook && (
<div className="border-border rounded-md border p-3">
<p className="flex items-center gap-1.5 text-sm font-medium">
<Webhook className="size-3.5" />
Direct webhook delivery
</p>
<p className="text-muted-foreground mt-1 text-xs">
No arr API key or connection needed. After creating the source, generate its
webhook URL and paste it into Sonarr/Radarr → Settings → Connect → Webhook.
</p>
</div>
{intervalInvalid && (
<p className="text-destructive text-xs">Must be a positive integer.</p>
)}
<p className="text-muted-foreground text-xs">
Optional — leave blank to use the global default poll interval.
</p>
</div>
)}
{!selectedIsWebhook && (
<div className="space-y-1.5">
<Label>Connection</Label>
<Select
value={form.connectionId || "__none__"}
onValueChange={(v) => setForm((f) => ({ ...f, connectionId: v }))}
>
<SelectTrigger className="w-full">
<SelectValue placeholder="No connection" />
</SelectTrigger>
<SelectContent>
<SelectItem value="__none__">— No connection —</SelectItem>
{connectionOptions.map((c) => (
<SelectItem key={c.id} value={c.id}>
{c.name}
</SelectItem>
))}
</SelectContent>
</Select>
<p className="text-muted-foreground text-xs">
Optional — bind one if the source needs to reach a server (e.g. Sonarr/Radarr).
Sources that read locally need no connection.
</p>
</div>
)}
{!selectedIsWebhook && (
<div className="space-y-1.5">
<Label htmlFor="add-source-interval">Poll interval (seconds)</Label>
<div className="flex items-center gap-2">
<Input
id="add-source-interval"
className="w-32"
placeholder="Default"
value={form.intervalStr}
aria-invalid={intervalInvalid}
onChange={(e) => setForm((f) => ({ ...f, intervalStr: e.target.value }))}
/>
<span className="text-muted-foreground text-sm">sec</span>
</div>
{intervalInvalid && (
<p className="text-destructive text-xs">Must be a positive integer.</p>
)}
<p className="text-muted-foreground text-xs">
Optional — leave blank to use the global default poll interval.
</p>
</div>
)}
{selectedIsCephFS && (
<CephFSConfigEditor