haeaeae/silo-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(autoscan): pluggable scan-source autoscan category (Sonarr/Radarr) (#44)

Browse Source 
* docs: design spec for autoscan arr polling

Periodic poller over autoscan-enabled Radarr/Sonarr instances (reusing
request_integrations) that maps import paths to Silo media folders and
enqueues targeted scans via the existing scantrigger + scanqueue. Lean
single-service model: no cross-node fan-out guard or retry queue.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: implementation plan for autoscan arr polling

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): settings and sources schema

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(autoscan): core types

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(autoscan): path rewrite helper

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(autoscan): dedupe imported paths to parent folders

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(autoscan): arr import-history client

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(autoscan): settings + sources repository

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(autoscan): redis scan-suppression seam

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(autoscan): PollOnce poll cycle

* feat(autoscan): poll task and wiring

* feat(autoscan): admin API endpoints

* feat(autoscan): admin API endpoints

Adds ErrIntegrationNotFound sentinel (errors.Is) instead of string matching.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): autoscan types and hooks

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(web): autoscan admin tab

* fix(autoscan): release suppression claim on enqueue failure; reconfigure trigger on interval change; skip source on key-resolution error

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): update handler test for 3-arg NewAutoscanHandler

* fix(autoscan): per-path suppression key, bounded poll window + overlap, boundary-safe rewrites, GREATEST cursor guard, async trigger, quiet unresolved-path skip, FK->404

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): normalize Windows path separators, surface status errors, re-seed source editor on save

Addresses minor code-review findings: Windows backslash paths now normalized
before rewrite/dedupe; HandleStatus returns repository errors instead of 200;
the per-source editor re-seeds from server data after a save.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: design spec for autoscan rewrite-sync from arr root folders

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: implementation plan for autoscan rewrite-sync

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): suffix-match rewrite suggester

Add suggestRewrites / commonSuffixLen for Task 1 of the autoscan
arr-polling feature. Pure function: matches arr root-folder paths to
Silo media folder paths by longest common trailing segment count,
adjusted for depth-delta so coincidental same-named segments at
different structural levels don't inflate confidence. Categorises
each arr root as Proposed, Ambiguous, Unmatched, or Covered by an
existing PathRewrite rule. TDD: test file written first, verified
failing, then implementation added.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(autoscan): GetSource single-source lookup

* feat(autoscan): arr root-folder client + Silo folder lister

* feat(autoscan): Service.SuggestRewrites

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(autoscan): rewrite-suggestions endpoint

Add GET /autoscan/sources/{id}/rewrite-suggestions admin endpoint: extend
the autoscanTriggerer interface with SuggestRewrites, wire SetRewriteResolvers
in the router, and add handler + test.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(web): autoscan rewrite-suggestions types and hook

* feat(web): autoscan sync-rewrites preview

* fix(autoscan): normalize covered-rule paths, dedup roots/folders, skip no-op suggestions

Addresses final-review edge cases: coveredBy normalizes the existing rewrite's
From (so a stored Windows/dup-slash rule still covers a root); duplicate arr
roots and duplicate Silo folder paths are de-duplicated; an arr path that already
equals its Silo path is not proposed as a no-op rewrite.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(web): vitest 4 compatible fetch spy in recipes.test (unblocks build after vitest 4.1.0 bump)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): non-null suggestion slices + move Sync into rewrites card

- suggestRewrites initializes Proposed/Unmatched/Ambiguous/Covered to empty
  slices so the JSON response is [] not null — fixes the 'Something went wrong'
  crash when every root is already covered (frontend mapped over null).
- Move the sync button into the Path rewrites card beside 'Add rewrite' and
  rename it 'Sync rewrites'; guard the proposed map with ?? [].

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): long root-folder timeout + sync spinner + collapse rewrites on load

- Root-folder fetch for sync uses a 2-min timeout: Radarr/Sonarr compute
  unmappedFolders by scanning all roots, so a large library's /rootfolder takes
  20-30s+ and tripped arrclient's 30s default (Sonarr 502'd at exactly 30s).
- Spin the sync icon + show 'Syncing…' while the request is in flight.
- Path rewrites card starts collapsed on page load.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): rescan on Sonarr/Radarr file renames

History polling previously only tracked downloadFolderImported events. A
rename in Sonarr/Radarr (episodeFileRenamed / movieFileRenamed) moves a
file without an import event, leaving the library folder stale until the
next full scan.

Extend the history client to also surface renamed paths: both the new
path and the old sourcePath, since a rename can move a file between
folders and both parents may need rescanning. Delete events are still
skipped — upgrade-deletes are covered by the paired import, and standalone
deletes carry no file path in arr history.

Renames the interface method ImportedPaths -> ChangedPaths to reflect the
broader scope.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(autoscan): synchronize trigger test with detached PollOnce goroutine

HandleTrigger dispatches PollOnce on a detached goroutine and responds 202
immediately. The test read trig.called straight after the handler returned,
racing the goroutine (usually 'PollOnce was not invoked') and reading the
field without synchronization (a data race under -race).

Signal completion through a channel the fake sends on when PollOnce runs;
the test waits on it (bounded) before asserting. The channel send
happens-before the receive, so the subsequent read of called is race-free.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: design spec for autoscan as a pluggable scan-source category

Reframes autoscan from a Requests-coupled, arr-only feature into a
standalone Autoscan category. Change-detection providers become
out-of-process plugins via a new additive scan_source.v1 capability
(client-pull, opaque marker); Sonarr/Radarr is the first provider.
Host keeps a provider-agnostic resolve/suppress/enqueue engine; all
arr-specific logic (and path rewrites) move into the plugin.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: implementation plan for scan_source.v1 SDK capability

First of the per-repo plans from the autoscan-plugin-architecture spec.
Adds the additive scan_source.v1 capability to silo-plugin-sdk (proto +
codegen + capability allowlist + runtime wiring), TDD per task, tagged as
v0.5.0 so the host and arr-plugin plans can build against it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: implementation plan for autoscan host backend (part 1 of 2)

Backend for the standalone Autoscan category: scan_source.v1 plugin
plumbing (pluginhost client + plugins.Service resolver), generalized
engine driven by a provider seam, autoscan_connections + autoscan_sources
schema (decoupled from Requests), connection resolution (own or
Requests-linked), admin API. Depends on silo-plugin-sdk v0.5.0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: implementation plans for autoscan arr plugin + host UI

arr plugin: new installable scan_source.v1 plugin (history imports+renames,
rewrites, Silo-native paths), structured like silo-plugin-tmdb; ports the
arr-specific logic from the closed PR #43.
host UI (part 2 of 2): standalone Autoscan admin category (connections,
sources, settings) extracted out of Requests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* build(autoscan): replace silo-plugin-sdk with local scan_source.v1 checkout

Temporary dev replace so the host backend can build against the unreleased
scan_source.v1 capability (silo-plugin-sdk PR #2). Finalize to v0.5.0 once
the SDK is tagged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(pluginhost): scan_source.v1 capability client wrapper

Adds ScanSourceClient struct, the Client.ScanSource() accessor (mirrors
ScheduledTask pattern), and a PollChanges method. Also introduces
client_test.go with capability-gate tests for both scheduled_task.v1 and
scan_source.v1 using a lazy gRPC ClientConn.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test+fix(pluginhost): cover capability-id gate, dedicated scan_source timeout

Adds a "wrong id returns error" subtest to both capability-gate tests so the
capability-ID component is exercised independently of the type. Introduces
DefaultScanSourceTimeout (2m) for PollChanges, which polls an external arr API
that can be slow, instead of the generic 10s control timeout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(plugins): expose scan_source.v1 client resolver

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(migrations): autoscan v2 schema (connections + sources, no requests FK)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(autoscan): v2 types and repository

Replace the request_integrations-coupled model with the decoupled v2
schema (autoscan_settings + autoscan_connections + autoscan_sources).
Connection CRUD, source upsert/list/get, and AdvanceMarker/RecordError
for opaque marker bookkeeping. ErrIntegrationNotFound becomes ErrNotFound.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): resolve connections (own credentials or Requests-linked)

ConnectionResolver turns a stored Connection into concrete credentials,
reading a soft-linked Requests integration's live base URL/key when
RequestIntegrationID is set, then resolving the api-key ref to plaintext.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): scan-source provider seam over the plugin resolver

ScanSourceProvider lets the engine poll changed paths without a live
plugin; pluginProvider adapts plugins.Service.ScanSourceClient in
production.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): generic engine drives sources via scan_source provider

Rewrite PollOnce to iterate enabled sources, resolve each connection,
poll the provider for changed paths, and run the salvaged
resolve→suppress→enqueue loop (uniqueParentDirs, (folder,path)
suppression key, RequestError quiet-skip, release-claims-on-enqueue-fail)
verbatim. Store the opaque next marker via AdvanceMarker only after a
successful enqueue; RecordError + keep marker on provider failure.
Tests reworked onto a fakeProvider/fakeStore with an added
opaque-marker-verbatim assertion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): drop conflicting connection CHECK, add connection resolver tests

- migration 172: remove the autoscan_connections_source_present CHECK. It
  conflicted with request_integration_id ON DELETE SET NULL: deleting a
  Requests integration that a linked-only connection (base_url NULL) points
  at would null the FK and trip the CHECK, blocking the delete. The intended
  behavior is for the connection to survive as an orphaned 'needs attention'
  row. Creation-time validity is now enforced at the application layer.
  Verified on a throwaway DB: full chain applies and the delete-cascade
  leaves an orphaned (both-null) connection.
- connection.go: TrimSpace the api key ref + resolved secret before the
  empty-string checks, matching requests.resolveAPIKey parity.
- connection_test.go: fake-based tests for ConnectionResolver.Resolve
  (own creds, linked, linked-missing error, lookup error, trim/fallback).
- repository.go: bound RecordError's stored last_error to 2048 chars.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(api): autoscan v2 admin endpoints

Rewrite the autoscan admin HTTP handler against the v2 model: settings,
connection CRUD, source update, manual trigger (detached PollOnce), and
status. Connection/source responses omit api_key_ref and resolved keys
(has_api_key flag only); unknown connection/source ids map to 404 via
autoscan.ErrNotFound. Retire the host-side rewrite-suggestions endpoint
(now lives in the arr plugin).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): wire v2 service, routes, retire rewrite-suggestions

Export PollChangesClient/ScanSourceResolver from the autoscan provider so
the api package can declare a structurally-conformant plugin adapter (Go
has no return-type covariance, so the adapter must name the interface as
its return type). Add api.BuildAutoscanService with the requests-integration
lookup and plugin scan-source adapters, shared by the router (manual
trigger) and the background poll task. Re-wire router routes to the v2
connections/sources/settings/trigger/status surface and drop the
rewrite-suggestions route. Update cmd/silo to build the v2 poll task,
seeding its interval from Settings.DefaultPollIntervalSeconds.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(api): enforce connection requires own URL or a Requests link

Migration 172 dropped the DB CHECK that required an autoscan connection to
carry either its own base_url or a request_integration_id, delegating that
invariant to the application layer — but the enforcement was never added, so
HandleCreateConnection/HandleUpdateConnection accepted both-NULL orphans that
ConnectionResolver.Resolve would hand a plugin as an empty base URL. Add a
shared validateConnectionInput helper (whitespace-only request_integration_id
counts as absent) and reject both-empty payloads with HTTP 400 on both the
create and update paths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): deliver resolved connection to plugin

PollChanges now populates PollChangesRequest.Connection with the
resolved {base_url, api_key} instead of dropping the conn param on the
floor. Drops the stale doc comment claiming the connection was delivered
out-of-band at upsert time -- that mechanism never existed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): auto-discover sources from installed scan_source plugins

Auto-discovery seeds a disabled, connection-less source row per
installed scan_source.v1 capability before an operator binds a
connection, so connection_id is now nullable end to end:

- migration 172: connection_id drops NOT NULL (still ON DELETE RESTRICT)
- Source.ConnectionID becomes *string; repository scans/writes it as
  nullable and adds idempotent EnsureSource (INSERT ... ON CONFLICT DO
  NOTHING)
- new ScanSourceLister seam + Service.DiscoverSources, called at the
  start of PollOnce (errors logged, non-fatal); production adapter
  enumerates ListEnabled -> ListCapabilities filtered to scan_source.v1
- PollOnce skips an enabled source with no connection bound, recording
  'no connection bound' so the UI can surface it
- HandleUpdateSource rejects enabling a source with no effective
  connection (400); source DTOs expose connection_id as nullable
- BuildAutoscanService / NewService thread the installation store at
  both wiring sites (router + poll task)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): honor per-source poll interval

PollOnce now skips an enabled source that ran too recently: the floor is
source.PollIntervalSeconds when set, else
settings.DefaultPollIntervalSeconds. The global poll task fires at the
default cadence, so this makes the per-source interval a 'poll at most
every N seconds' floor.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(autoscan): reconcile spec + arr-plugin plan with credential-in-request + auto-discovery

The credential-delivery mechanism changed during execution: the host now
passes resolved {base_url, api_key} in PollChangesRequest.connection each
poll (not plugin runtime config). Also records source auto-discovery,
nullable connection_id, and the per-source interval floor decided at the
final integration review.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): autoscan v2 types and query hooks

Replace v1 autoscan types and hooks with v2 DTOs matching the backend
handler (autoscan.go): settings, connection (with has_api_key, no raw
key), source (installation_id/capability_id/connection_id), status.
Add connections CRUD hooks, useAutoscanStatus, update sources hook to
v2 input shape. Retain deprecated shims for AutoscanPathRewrite,
AutoscanRewriteSuggestions, and useAutoscanRewriteSuggestions so
AdminRequests.tsx continues to compile until Task 6 removes that tab.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): autoscan connections panel (reuse or own)

Card+Table listing connections with "Reused from Requests" / "Own" badges.
Add/edit dialog with two modes: reuse a Sonarr/Radarr Requests integration
or enter own name/URL/API-key credentials. Delete with alert-dialog confirm.
Never renders key material — only has_api_key is sent by the backend.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): autoscan sources panel

Table of auto-discovered scan sources (one row per installed scan_source
plugin capability). Operator can bind a connection via inline Select
(auto-saved on change), set a per-source poll interval (saved on blur),
and toggle enabled. Shows a "Needs connection" badge for unbound sources;
attempting to enable without a connection lets the backend 400 surface via
the existing toast in useUpdateAutoscanSource.onError. Status column shows
last_run_at relative time or last_error with icon.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): standalone Autoscan admin page

Tabs page (Sources | Connections | Settings) mirroring AdminRequests
header/layout. Settings tab exposes global enable switch, default poll
interval, and debounce — all auto-saved on blur or toggle. "Run now"
button calls useTriggerAutoscan and toasts "Autoscan triggered" on 202.
Route and sidebar nav are intentionally deferred to Task 5.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): route and sidebar nav for Autoscan category

Add /admin/autoscan route pointing to AdminAutoscan and a matching
"Autoscan" item in the Content group of the admin sidebar (with RefreshCw
icon), so the new standalone page is reachable from the nav.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(web): move Autoscan out of Requests into its own category

Remove the Autoscan tab, AutoscanTab/AutoscanSourceEditor component
definitions, and AutoscanSettingsFormState from AdminRequests.tsx.
Delete the Task-1 compatibility stubs: AutoscanPathRewrite and
AutoscanRewriteSuggestions types from api/types.ts, and the
useAutoscanRewriteSuggestions no-op shim from useAutoscan.ts. The
build confirms zero dangling references.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): allow unbinding a source connection (full-state source update)

Change the source-update input struct's connection_id from string to *string so
the UI can send null to unbind, a UUID to bind, or omit (null) to clear. Remove
the fall-back-to-existing logic; the handler now sets the source's ConnectionID
directly from the input. The enable-guard fires when the resulting connection is
nil regardless of cause. Frontend sends the complete triple (connection_id,
enabled, poll_interval_seconds) on every mutation site; selecting "— No
connection —" sends null for a real unbind. Adds aria-label to connection Select
and interval Input for accessibility. Backend tests cover bind, unbind, unbind
while enabled → 400, and enable without connection → 400.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(migrations): backfill autoscan v1 settings+connections instead of dropping

Migration 172 unconditionally DROPped the shipped v1 autoscan_settings/
autoscan_sources (migration 171), losing an upgraded operator's enable flag,
poll cadence, debounce, and arr server list — autoscan came back OFF.

Rewrite 172 up to be non-destructive of what can be carried: rename the v1
tables aside, create the v2 schema, backfill settings (poll minutes -> seconds)
and seed a reusable LINKED connection per distinct v1 source integration, then
drop the renamed v1 tables. v2 sources are keyed on a plugin
(installation_id, capability_id) that did not exist in v1, so they are left to
runtime discovery; path rewrites move to plugin config and are intentionally
not carried.

Verified against a throwaway DB: after 171 + v1 seed data, applying 172 yields
enabled=true, default_poll_interval_seconds=300, debounce_seconds=30, and one
autoscan_connections row linked to the v1 integration.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): preserve api key on metadata-only connection edit

UpdateConnection unconditionally wrote api_key_ref = nullable(c.APIKeyRef), so a
metadata-only edit (the UI omits the key when left blank — "leave blank to keep
existing") NULLed the stored key and broke the next poll. Mirror requests'
UpdateIntegration: api_key_ref = CASE WHEN $5 = '' THEN api_key_ref ELSE $5 END,
passing the raw trimmed string so a blank incoming ref keeps the existing value.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): skip orphaned sources + add source delete endpoint

An enabled source whose scan_source plugin was uninstalled/disabled kept its
autoscan_sources row, which errored every poll cycle, and there was no way to
remove it.

DiscoverSources now returns the set of currently-discovered
(installation_id, capability_id) pairs; PollOnce skips any enabled source not in
that set quietly (no RecordError), stopping the per-cycle error spam for
orphans. A nil set (no lister / discovery failed) disables pruning so a transient
discovery failure does not silence live sources.

Adds DELETE /admin/autoscan/sources/{id} -> HandleDeleteSource ->
repo.DeleteSource so an operator can clear orphans (unknown id -> 404).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): reject reused connection when Requests integration is disabled

RequestIntegrationLookup.Get returned a linked integration's base_url/api_key
even when the integration was disabled or had a blank base_url (the v1 poll gate
`WHERE ri.enabled = true` was dropped in v2). Now Get surfaces a disabled or
unconfigured linked integration as an error, which the engine turns into a
logged skip / RecordError instead of polling an unusable target. The gating is
extracted into a pure checkRequestIntegrationUsable helper so it is unit-testable
without a DB-backed repo.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): reschedule poll task on settings change

HandleUpdateSettings no longer rescheduled the poll task (the v1 triggerUpdater /
UpdateTriggers wiring was dropped in v2), so a default_poll_interval_seconds
change only applied after a restart.

Re-add an optional triggerUpdater (taskmanager.UpdateTriggers) on AutoscanHandler,
wired via SetTriggerUpdater from the router when a task manager is available. On a
successful settings update the handler recomputes the interval trigger from
default_poll_interval_seconds and calls UpdateTriggers("autoscan_poll", ...). The
dependency is optional: a nil updater skips rescheduling so tests need no task
manager, and a reschedule failure is non-fatal (the interval is persisted).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(web): disable enable toggle for unbound sources, add source delete + interval hint

- Disable the Enable switch when a source has no effective bound connection
  (connection_id null and no pending edit selection), re-enabling once bound.
- Add useDeleteAutoscanSource hook mirroring useDeleteAutoscanConnection pattern.
- Add per-row delete button (Trash2 icon → AlertDialog confirm) to let
  operators remove orphaned/unwanted source rows.
- Add interval floor helper text showing the global default poll interval
  so operators know values below it have no effect.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): consume source_paths from merged scan_source contract

The merged plugin SDK renamed PollChangesResponse.changed_paths to
source_paths and the plugin now returns RAW source-namespace paths.
pluginProvider.PollChanges reads GetSourcePaths(); the host applies
per-source path rewrites before resolving/enqueueing (separate commit).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(migrations): add path_rewrites to autoscan_sources

Add path_rewrites jsonb NOT NULL DEFAULT '[]' to the autoscan_sources
CREATE in migration 172 (unreleased/branch-only, so amended in place).
The host now owns per-source prefix rewrites. v1 path_rewrites cannot be
backfilled (v2 sources key on a plugin installation/capability with no v1
mapping); documented that operators must re-enter rewrites post-upgrade.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): host-owned per-source path rewrites

Rewrite ownership moved from the scan_source plugin to the host. The
plugin returns raw source-namespace paths; the host now normalizes
separators and applies the source's per-source prefix rewrites before
dedupe/resolve/enqueue.

- types: add PathRewrite{From,To} and Source.PathRewrites
- rewrite: re-add applyRewrites/normalizeSeparators; apply the
  MOST-SPECIFIC (longest From) match, not first-match, so a broad rule
  can't shadow a nested one regardless of ordering
- service.PollOnce: rewrite raw provider paths before resolveAndClaim
- repository: marshal/unmarshal path_rewrites jsonb in UpsertSource and
  all source scans (EnsureSource discovery rows take the DB default [])
- handlers: autoscanSourceInput/response + status DTO carry path_rewrites
  (full-state like connection_id); reject blank from/to with 400
- tests: rewrite unit tests, engine applies rewrites before enqueue,
  handler round-trips path_rewrites and 400s on a blank rewrite

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): discover installed scan_source plugins on sources-list view

A scan_source plugin installed via the normal /admin/plugins flow must show up
in the Autoscan component immediately, not only after a poll cycle (which runs
only when autoscan is enabled). HandleListSources now runs discovery (seeding a
disabled, connection-less source row per installed scan_source capability)
before listing. Best-effort: discovery failure does not block listing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): per-source path rewrites editor + plugins-page install hint

Add AutoscanPathRewrite type and path_rewrites fields to AutoscanSource/
AutoscanSourceInput. SourcesPanel gains an expandable rewrite editor per
source row (from→to pairs, Add/Remove/Save) threaded into the full-state
body so connection, interval, and rewrite changes always carry all fields.
Adds a Plugins-page install hint in both the empty state and above the
table for discoverability.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(autoscan): host-owned path rewrites + install/discovery flow

Reconcile the spec with the merged SDK decision (rewrites moved host-side;
PollChangesResponse.source_paths carries raw provider paths). Document that
scan-source plugins install via the normal /admin/plugins page and surface in
Autoscan via discovery (run on poll cycles and on sources-list view).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* build: depend on merged silo-plugin-sdk via pseudo-version (drop local replace)

PR #2 (scan_source.v1 + source_paths) is merged to silo-plugin-sdk main, so the
host can resolve the canonical module at the merged commit
(v0.4.1-0.20260603030807-807b07e785b2) instead of a local-path replace. The
branch now builds off-machine (CI/Docker). Bump to a clean v0.5.0 once tagged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor(migrations): single clean autoscan v2 migration (v1 never shipped)

The v1 in-process autoscan (migration 171) was never released to origin/main,
so no live system has v1 autoscan data to preserve. Collapse the v1-create +
v2-rename/backfill/drop dance into one clean 171 that creates the v2
connections-based schema directly. Removes 172 entirely.

The runner applies by version set-difference with no checksum validation, so
the already-migrated test instance (171+172 recorded) skips both and is
unaffected; fresh installs get the clean v2 schema in one step.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): allow many sources per plugin + add-source enumeration

Drop the one-source-per-(installation, capability) model. A single installed
scan_source plugin capability can now back many sources, each bound to a
different connection (e.g. one Sonarr plugin fronting four arr servers).

- migration 171: remove the autoscan_sources UNIQUE(installation_id,
  capability_id) constraint; sources are operator-created, not auto-seeded.
- repository: replace UpsertSource (relied on the unique conflict) with a plain
  CreateSource (fresh uuid) + a by-id UpdateSource; remove EnsureSource.
- discovery: replace auto-seeding (DiscoverSources/RefreshDiscovered) with
  ListAvailableScanSources (the Add-source picker list, enriched with plugin id
  + display name) and an installedScanSources set used only for orphan-skip.
- service: PollOnce stops seeding and instead fetches the installed-capability
  set for orphan detection; Store gains GetSource and drops EnsureSource.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): connection test endpoint (engine)

Add Service.TestConnection / TestConnectionByID: resolve a connection (ad-hoc
input or an existing stored connection) to concrete credentials and probe the
arr GET /api/v3/system/status with a short timeout. A reachable/authorized
target yields OK=true plus the reported version; an unreachable / 401 / non-200
target yields OK=false with a human-readable error (the probe failure is part of
the result payload, never an error from the method itself).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): host-side rewrite suggester + admin API for new endpoints

Port the path-rewrite suggester back host-side (it had moved into the plugin):
suggestRewrites suffix-matches arr root folders against Silo media folders to
propose path rewrites, reporting proposed / unmatched / ambiguous / covered.
Service.SuggestRewrites resolves the source's bound connection, lists arr roots
(GET /api/v3/rootfolder) and Silo folder paths, and runs the matcher; a source
with no bound connection returns ErrNoConnection (400).

Admin API (all admin-gated):
- POST   /admin/autoscan/sources                       create a source
- GET    /admin/autoscan/scan-source-plugins           Add-source picker list
- POST   /admin/autoscan/connections/test              probe a connection
- GET    /admin/autoscan/sources/{id}/rewrite-suggestions  sync rewrites
HandleListSources no longer auto-seeds; create validates the capability is
currently installed and that enabling requires a connection.

Wiring threads the arr root-folder/status client and the catalog folder lister
through BuildAutoscanService; the lister now surfaces plugin id + display name.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): autoscan hooks + types for sources, connection test, rewrites

Add types and React Query hooks backing the autoscan admin UI batch:
- AutoscanAvailableSource / useAvailableScanSources (scan-source plugins)
- AutoscanSourceCreateInput / useCreateAutoscanSource (POST sources)
- AutoscanConnectionTestResult / useTestAutoscanConnection (advisory test)
- AutoscanRewriteSuggestions / useAutoscanRewriteSuggestions (on-demand)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): add-source dialog + sync-from-arr rewrites in SourcesPanel

Add a "+ Add source" header action opening a dialog that creates a scan
source from any installed scan-source plugin bound to an arr connection,
so operators can add one source per connection (e.g. four arr instances).
Empty state links to /admin/plugins when no plugins are installed.

Add a "Sync from arr" button to each source's rewrite editor that fetches
root-folder rewrite suggestions and renders a preview: checkbox-selectable
Proposed rewrites plus collapsed Unmatched / Ambiguous / Already-mapped
sections. "Apply selected" merges the checked rewrites (dedupe by `from`)
and persists via the normal full-state source PUT. Sync is disabled until
the source has a bound connection.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): test-connection button in autoscan ConnectionsPanel dialog

Add an advisory "Test connection" button to the add/edit connection
dialog. It probes the current dialog input — connection_id when editing,
request_integration_id in reuse mode, or base_url/api_key_ref for own
credentials — and renders the result inline: green "Connected (vX.Y)" on
success, red error on failure. Never blocks save; stale results clear when
credential fields change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): autoscan page polish + global enable toggle in header

Surface a global Autoscan enable toggle and an enabled/disabled status
badge next to the page title, alongside the existing "Run now" header
action so primary controls are reachable without opening a tab. Remove the
now-redundant enable switch from the Settings tab (it points at the header
toggle instead). Tighten header layout for wrap on narrow widths.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(web): right-align autoscan enable toggle + Run now in the page header

Drop the redundant nested justify-between wrapper so the header actions sit
directly under .page-header (space-between + bottom-align), matching the
/admin/libraries header layout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): hold poll marker when paths return but none resolve

A freshly-enabled source whose path_rewrites aren't configured yet returns
provider paths that resolve to zero library folders. PollOnce previously
advanced the marker unconditionally on any successful poll, permanently
skipping those imports. Now the marker advances only when there is nothing to
do (zero paths) or at least one path resolved+enqueued; when paths come back
but none resolve, the marker is held and an explaining error recorded so the
operator can fix the rewrites and a later poll re-reads the same window.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): don't prune sources of disabled-but-installed plugins

PluginScanSourceLister used the installation store's ListEnabled, so a
temporarily-disabled plugin dropped out of the discovered set and PollOnce
treated its sources as orphaned, skipping them with no last_error (silent
vanish). Switch to List so only a fully-uninstalled plugin counts as orphaned;
a disabled-but-installed plugin's sources are still attempted and surface a
visible RecordError when the client fails to load. The Add-source picker shares
the same all-installed set.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): treat empty request_integration_id as no link

ConnectionResolver.Resolve gated the linked-integration path on a non-nil
RequestIntegrationID pointer, so a pointer-to-empty-string (from a both-NULL
orphan or a stripped link) called requests.Get(""). Guard on a non-empty
trimmed value so it falls back to the connection's own fields instead.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): align startup poll interval with reschedule computation

Startup seeded the poll task by integer-dividing default_poll_interval_seconds
by 60 (minutes), while HandleUpdateSettings reschedules with seconds*1000 ms;
the two diverged for sub-minute and non-60-multiple intervals. NewAutoscanPollTask
now takes the interval in milliseconds and main.go seeds it as seconds*1000,
matching the reschedule path so both agree.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): normalize stored rewrite From at poll time

applyRewrites matched the stored From after only TrimSpace/TrimSuffix, while
suggest.go coveredBy normalizes via normalizePath (backslash->slash, collapse
'//'). A Windows-style or dup-slash stored rewrite was thus reported 'covered'
at suggest time yet never matched at poll time. applyRewrites now normalizes
From through normalizePath so poll-time and suggest-time agree.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(web): don't corrupt source poll interval on enable/connection change

Add a `parseInterval` helper that maps empty input to null (use global
default), valid positive integers to the integer, and any other
mid-edit-invalid value to the source's currently-persisted
`poll_interval_seconds` — so toggling the enable switch or changing the
connection cannot silently overwrite the interval with 0 or NaN.

Wire the helper through `fullBody()` (the single source of truth for PUT
payloads) and remove the two inline duplications in `handleConnectionChange`
and `handleRewriteSave` that both previously used raw `Number()`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): make the source connection optional (provider-agnostic)

A host connection is the credential/endpoint for server-based providers
(Sonarr/Radarr); other scan_source providers (e.g. a CephFS/filesystem watcher
that reads ceph.dir.r* xattrs) need none. PollOnce now polls connection-less
sources, passing an empty ResolvedConnection the plugin may ignore; a plugin
that requires credentials surfaces the error at poll time. Drops the
enable-requires-connection 400s. Provider-specific config lives in the plugin's
own global_config_schema, not a host connection.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(web): provider-agnostic autoscan copy + optional source connection

Replace arr-hardcoded framing in AdminAutoscan, SourcesPanel, and
ConnectionsPanel with neutral scan-source language. Remove the
connection-required gate on the source enable toggle so connectionless
providers (e.g. filesystem watchers) can be enabled; soften the badge
from "Needs connection" to "No connection". Sync-from-server button
remains gated on a bound connection (it needs a server to query).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: repo-relative paths in autoscan plans

Replace local absolute filesystem paths (/opt/silo, sibling checkouts,
/tmp/go/bin) in docs/superpowers/plans with repository-relative wording
per CLAUDE.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): rune-safe last_error truncation

Truncate RecordError messages on a UTF-8 rune boundary so a byte-bounded
cut can't split a multi-byte rune and store invalid UTF-8.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): advance marker when resolved-but-suppressed (not unresolved)

resolveAndClaim now reports resolvedAny (whether any path mapped to a
Silo library folder, independent of suppression). PollOnce gates the
"none matched a Silo library folder" hold+RecordError on !resolvedAny
instead of len(targets)==0, so a poll whose paths resolved but were all
debounced/suppressed advances the marker instead of being treated as a
misconfiguration. Adds a regression test for the suppressed case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(api): normalize request_integration_id

Trim whitespace and collapse empty-after-trim request_integration_id to
nil on connection create and update, so a pointer-to-"" or "  " is never
persisted as a bogus Requests link. Also corrects a stale migration-172
comment to 171 (the collapsed migration number).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore(autoscan): provider-agnostic poll-task copy

Rename the poll task to "Autoscan poll" with a provider-agnostic
description and progress message; drop Sonarr/Radarr/arr wording. Key()
(autoscan_poll) is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(autoscan): fix typo in connectionless-source test name

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): add scan source management

* chore(deps): bump silo-plugin-sdk for structured scan source changes

Pins silo-plugin-sdk to 0d78651, which adds source_config on
PollChangesRequest plus the structured changes / ScanSourceChangeScope
fields on PollChangesResponse that internal/autoscan/provider.go already
consumes. Without this the branch fails to compile against the prior
pin (807b07e).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): label scan sources by connection name in admin UI

arr-plugin sources fan out one-per-connection under a single generic
"arr" capability, so every row in the Sources and Activity panels
rendered an identical "arr (plugin #N)" label. Lead with the bound
connection name (Radarr/Sonarr/...) instead, demoting capability +
plugin to a subtitle. Sources without a connection (e.g. cephfs) keep
the capability fallback.

Activity threads a source_id -> connection name lookup (built from the
existing sources + connections queries) through the scan/poll tables the
same way librariesByID is threaded.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(autoscan): spec for generic + operator-editable source labels

Design for a shared label-resolution helper (operator label -> connection
name -> manifest display_name -> capability_id) consumed by the Sources and
Activity panels, plus an operator-editable per-source label backed by a new
autoscan_sources.label column.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(autoscan): implementation plan for source labels

Task-by-task TDD plan: migration 174 (label column), Go domain/repo/handler
wiring with server-side normalization, shared frontend label helper, and
Sources/Activity panel integration.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): migration for source label column

* feat(autoscan): source label domain field + normalizer

* feat(autoscan): persist source label in repository

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(autoscan): accept, normalize, and return source label

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): add label to source API types

* feat(autoscan): shared source-label resolution helper

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(autoscan): polish source-label helper per review

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(autoscan): label sources via shared helper + operator label input

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(autoscan): clarify source label naming per review

* feat(autoscan): resolve activity source labels via shared helper

Replace the sourceNames Map plumbing in ActivityPanel with SourceLabelLookups
and delegate both name functions to resolveEventSourceName from @/lib/autoscanLabels,
enabling the full label chain (operator label → connection name → manifest display_name
→ capability_id) for all Scan History and Poll log rows.

* fix(autoscan): carry label on status source + guard poll label

Final-review follow-ups: add the label field to the autoscanStatusSource
response (and AutoscanStatusSource type) so the status view matches the
source response per spec, and give pollSourceName a non-empty fallback for
symmetry with scanSourceName.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(autoscan): resolve source aria-labels through the label chain

Replace the legacy capability-only sourceLabel() helper with resolveSourceName()
(operator label -> connection -> display_name -> capability). Row controls now
announce the row's resolvedLabel (reflecting in-progress edits) and the delete
dialog announces the resolved name, so screen readers hear "4K Movies" instead
of "arr (plugin #4)".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(autoscan): paginate queue + history with a shared table pager

Replace the card/table hybrid and 200-row "Load more" cap on the autoscan
Activity panel with proper tables and real pagination.

Backend: add offset + total-count to the scans/events list endpoints so
history pages through the full set instead of a capped window. Extract
shared event/scan WHERE-clause builders so list and count filter
identically, and add CountEvents / CountAutoscanScans.

Frontend: add a reusable TablePagination component (rows-per-page,
"showing X-Y of Z", numbered window with ellipses, responsive) and reuse
it for the server-paginated history (scans + polls) and the
client-paginated live queue. Unify all three tables behind one DataTable
shell so they read as one family.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
This commit is contained in:
fluxis
2026-06-05 22:19:38 -04:00
committed by GitHub
co-authored by Claude Opus 4.8 Quick
parent efd377fbdc
commit 177cfdc485
81 changed files with 16236 additions and 138 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cmd/silo/main.go
+25
View File
@@ -35,6 +35,7 @@ import (
"github.com/Silo-Server/silo-server/internal/api"
"github.com/Silo-Server/silo-server/internal/api/handlers"
"github.com/Silo-Server/silo-server/internal/auth"
"github.com/Silo-Server/silo-server/internal/autoscan"
"github.com/Silo-Server/silo-server/internal/cache"
"github.com/Silo-Server/silo-server/internal/catalog"
"github.com/Silo-Server/silo-server/internal/catalogseed"
@@ -1395,6 +1396,30 @@ func main() {
requestReconcileSvc.SetEntitlementResolver(mediarequests.NewAccessEntitlements(reconcileResolver))
}
taskMgr.Register(tasks.NewReconcileRequestsTask(requestReconcileSvc, 100))
if deps.FolderRepo != nil && deps.LibraryScanQueue != nil && pluginService != nil && pluginInstallationStore != nil {
autoscanRepo := autoscan.NewRepository(deps.DB)
autoscanSvc := api.BuildAutoscanService(
autoscanRepo,
pluginService,
pluginInstallationStore,
mediarequests.NewRepository(deps.DB),
settingsRepo,
deps.FolderRepo,
deps.LibraryScanQueue,
deps.RedisClient,
)
// The poll task's default interval seeds the schedule from the stored
// settings (DefaultPollIntervalSeconds); per-cycle gating still runs
// off the live settings inside PollOnce. Seed in MILLISECONDS as
// seconds*1000 — the SAME computation HandleUpdateSettings uses to
// reschedule — so startup and reschedule agree for sub-minute and
// non-60-multiple intervals (the old seconds/60 minutes path diverged).
var intervalMs int64 = 10 * 60 * 1000
if settings, serr := autoscanRepo.GetSettings(appCtx); serr == nil && settings.DefaultPollIntervalSeconds > 0 {
intervalMs = int64(settings.DefaultPollIntervalSeconds) * 1000
}
taskMgr.Register(tasks.NewAutoscanPollTask(autoscanSvc, intervalMs))
}
reconcileProviderIDRepo := catalog.NewProviderIDRepository(deps.DB)
reconcileEpisodeRepo := catalog.NewEpisodeRepository(deps.DB)
historyResolver := watchstate.NewStableIdentityResolver(nil, reconcileEpisodeRepo, reconcileProviderIDRepo)
docs/superpowers/plans/2026-06-02-autoscan-arr-polling.md
+1198
View File
File diff suppressed because it is too large Load Diff
docs/superpowers/plans/2026-06-02-autoscan-host-backend.md
+779
View File
@@ -0,0 +1,779 @@
# Autoscan Host Backend Implementation Plan (Part 1 of 2)
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Build the silo-server backend for the standalone Autoscan category: poll `scan_source.v1` plugins on a timer, resolve their Silo-native paths to library folders, and enqueue rescans — with connections decoupled from Requests.
**Architecture:** Mirror the existing `scheduled_task.v1` host dispatch (`pluginhost.Client` capability wrapper + `plugins.Service` resolver + a `taskmanager` task). Generalize the closed PR #43 autoscan engine (resolve → suppress → enqueue) so its input is "Silo-native paths from a scan-source plugin" instead of "arr history." Replace the old `request_integrations`-coupled schema with `autoscan_connections` + `autoscan_sources`.
**Tech Stack:** Go, pgx/v5, the `internal/plugins`/`internal/pluginhost` runtime, `internal/scantrigger`/`internal/scanqueue`, `internal/taskmanager`, numbered SQL migrations, `github.com/Silo-Server/silo-plugin-sdk` (the new `scan_source.v1` capability).
Commands assume the silo-server repository root is the cwd. This is **Part 1 (backend)**; the Autoscan admin UI is Part 2.
---
## Prerequisites
1. **SDK dependency.** This plan needs the SDK's `scan_source.v1` symbols (`runtime.Client.ScanSource()`, `pluginv1.ScanSourceClient`, `pluginv1.PollChangesRequest/Response`, `capability.ScanSource`). They are in `silo-plugin-sdk` PR #2 but **not yet released**. Until `v0.5.0` is tagged, add a local replace directive so the host builds against the working checkout:
```bash
# from the repository root (cwd)
go mod edit -replace github.com/Silo-Server/silo-plugin-sdk=../silo-plugin-sdk
go mod tidy
```
This points the replace at the `silo-plugin-sdk` checkout (a sibling of this repo).
The final task removes the replace and bumps to the tagged version.
2. **Closed PR #43 is on `main`.** The in-process arr autoscan (`internal/autoscan/*`, `internal/api/handlers/autoscan.go`, `internal/taskmanager/tasks/autoscan_poll.go`, router routes, migration `171_autoscan`, the `AdminRequests.tsx` tab) is the salvage source. This plan **generalizes the engine and replaces the schema/wiring**; it does not preserve the arr-specific history client (that moves to the arr plugin in a separate plan).
3. **Tests that touch `internal/api/handlers` or `internal/api` link libvips via CGO.** Run the suite in the libvips-equipped container (see project memory `silo-test-libvips-gotcha`), not the bare host, or those packages report `[build failed]` and skip silently.
---
## Pattern references (read before starting)
- Host capability client wrapper: `internal/pluginhost/client.go` — `ScheduledTaskClient` type (l.42), `Client.ScheduledTask` accessor (l.111), `requireCapability` (l.157), `Run` (l.226).
- Host resolver: `internal/plugins/service.go:413` `ScheduledTaskClient(ctx, installationID, capabilityID)` → `ensureClient` → `client.ScheduledTask(capabilityID)`.
- Timer task over plugin capabilities: `internal/plugins/task_registry.go` (iterate enabled installations, filter capability type, build a `taskmanager.Task`).
- Salvageable engine (generalize, don't rewrite): `internal/autoscan/service.go` (resolve/suppress/enqueue loop), `dedupe.go` (`uniqueParentDirs`), `suppress.go` (`Suppressor`). Discard `history.go`, `rewrite.go`, `suggest*.go` (arr-specific → arr plugin).
- Migration mechanics: paired `migrations/NNN_*.up.sql`/`.down.sql`, embedded via `migrations/embed.go`, auto-applied on boot.
- Credential decryption for the reuse-from-Requests case: `internal/requests/repository.go` (`integrationColumns` has `base_url, api_key_ref`); Fernet resolution via `catalog.ServerSettingsRepo.Get` (the same path PR #43's `SecretResolver` used).
---
## File Structure
| File | Responsibility | Action |
|---|---|---|
| `internal/pluginhost/client.go` | `ScanSourceClient` wrapper + `Client.ScanSource()` accessor | Modify |
| `internal/pluginhost/client_test.go` | `requireCapability` gate for `scan_source.v1` | Modify |
| `internal/plugins/service.go` | `ScanSourceClient(ctx, installationID, capabilityID)` resolver | Modify |
| `migrations/172_autoscan_v2.up.sql` / `.down.sql` | `autoscan_connections` + `autoscan_sources`; drop the `171` tables | Create |
| `internal/autoscan/types.go` | `Source`, `Connection`, `Settings`, `PollChanges` provider seam | Rewrite |
| `internal/autoscan/repository.go` | CRUD for connections + sources; `AdvanceMarker` | Rewrite |
| `internal/autoscan/provider.go` | `ScanSourceProvider` interface + adapter over the plugins resolver | Create |
| `internal/autoscan/connection.go` | Resolve a `Connection` (own or Requests-linked) → `{base_url, api_key}` | Create |
| `internal/autoscan/service.go` | Generic engine: per source → `PollChanges(marker)` → resolve → suppress → enqueue → store marker | Rewrite |
| `internal/autoscan/dedupe.go`, `suppress.go` | Unchanged salvage | Keep |
| `internal/autoscan/history.go`, `rewrite.go`, `suggest*.go` | Arr-specific | Delete |
| `internal/taskmanager/tasks/autoscan_poll.go` | Poll task → `Service.PollOnce` | Keep (retarget) |
| `internal/api/handlers/autoscan.go` | Admin API (settings/connections/sources/trigger/status) | Rewrite |
| `internal/api/router.go`, `cmd/silo/main.go` | Wire the rebuilt service + connection resolver; drop request-coupled wiring | Modify |
| `go.mod` | SDK replace (dev) → `v0.5.0` (final) | Modify |
---
## Task 1: Host `pluginhost` client wrapper for `scan_source.v1`
**Files:**
- Modify: `internal/pluginhost/client.go`
- Modify: `internal/pluginhost/client_test.go`
- [ ] **Step 1: Write the failing test**
In `internal/pluginhost/client_test.go`, add a test mirroring the existing `scheduled_task` capability-gate test (find it first; it constructs a `Client` with a manifest declaring a capability and asserts the accessor errors when the capability is absent and succeeds when present):
```go
func TestClientScanSourceRequiresCapability(t *testing.T) {
// Client whose manifest does NOT declare scan_source.v1
c := newTestClientWithCapabilities(t, "metadata_provider.v1")
if _, err := c.ScanSource("missing"); err == nil {
t.Fatal("ScanSource: expected error when capability absent")
}
// Client WHOSE manifest declares scan_source.v1 with id "arr"
c2 := newTestClientWithCapabilities(t, "scan_source.v1:arr")
if _, err := c2.ScanSource("arr"); err != nil {
t.Fatalf("ScanSource: unexpected error when capability present: %v", err)
}
}
```
Use whatever helper the sibling `scheduled_task` test uses to construct a `Client` with declared capabilities; mirror it exactly (do not invent a helper that does not exist — read `client_test.go` and copy the construction).
- [ ] **Step 2: Run test to verify it fails**
Run: `go test ./internal/pluginhost/ -run TestClientScanSourceRequiresCapability`
Expected: FAIL — `c.ScanSource undefined`.
- [ ] **Step 3: Add the wrapper type and accessor**
In `internal/pluginhost/client.go`, after the `ScheduledTaskClient` type (l.42), add:
```go
type ScanSourceClient struct {
client pluginv1.ScanSourceClient
timeout time.Duration
}
```
After the `Client.ScheduledTask` accessor (l.117), add:
```go
func (c *Client) ScanSource(capabilityID string) (*ScanSourceClient, error) {
if err := c.requireCapability("scan_source.v1", capabilityID); err != nil {
return nil, err
}
return &ScanSourceClient{
client: c.rpc.ScanSource(),
timeout: DefaultControlTimeout,
}, nil
}
func (c *ScanSourceClient) PollChanges(ctx context.Context, req *pluginv1.PollChangesRequest) (*pluginv1.PollChangesResponse, error) {
callCtx, cancel := ensureDeadline(ctx, c.timeout)
defer cancel()
return c.client.PollChanges(callCtx, req)
}
```
- [ ] **Step 4: Run test to verify it passes**
Run: `go test ./internal/pluginhost/ -run TestClientScanSourceRequiresCapability`
Expected: PASS.
- [ ] **Step 5: Commit**
```bash
git add internal/pluginhost/client.go internal/pluginhost/client_test.go
git commit -m "feat(pluginhost): scan_source.v1 capability client wrapper"
```
---
## Task 2: `plugins.Service` resolver for `scan_source.v1`
**Files:**
- Modify: `internal/plugins/service.go`
- [ ] **Step 1: Add the resolver method**
After `ScheduledTaskClient` (l.413–423), add the identical shape for scan source:
```go
func (s *Service) ScanSourceClient(
ctx context.Context,
installationID int,
capabilityID string,
) (*pluginhost.ScanSourceClient, error) {
client, err := s.ensureClient(ctx, installationID)
if err != nil {
return nil, err
}
return client.ScanSource(capabilityID)
}
```
- [ ] **Step 2: Build to verify it compiles**
Run: `go build ./internal/plugins/`
Expected: exits 0.
- [ ] **Step 3: Commit**
```bash
git add internal/plugins/service.go
git commit -m "feat(plugins): expose scan_source.v1 client resolver"
```
---
## Task 3: Schema — `autoscan_connections` + `autoscan_sources`
**Files:**
- Create: `migrations/172_autoscan_v2.up.sql`, `migrations/172_autoscan_v2.down.sql`
- [ ] **Step 1: Write the up migration**
Create `migrations/172_autoscan_v2.up.sql`. It drops PR #43's `171` tables and creates the decoupled schema (no FK into `request_integrations`; the optional reuse link is `ON DELETE SET NULL`):
```sql
DROP TABLE IF EXISTS public.autoscan_sources;
DROP TABLE IF EXISTS public.autoscan_settings;
CREATE TABLE public.autoscan_settings (
id boolean PRIMARY KEY DEFAULT true,
enabled boolean NOT NULL DEFAULT false,
default_poll_interval_seconds integer NOT NULL DEFAULT 600,
debounce_seconds integer NOT NULL DEFAULT 60,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
CONSTRAINT autoscan_settings_singleton CHECK (id),
CONSTRAINT autoscan_settings_interval_pos CHECK (default_poll_interval_seconds > 0),
CONSTRAINT autoscan_settings_debounce_nonneg CHECK (debounce_seconds >= 0)
);
INSERT INTO public.autoscan_settings (id) VALUES (true) ON CONFLICT (id) DO NOTHING;
CREATE TABLE public.autoscan_connections (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
name text NOT NULL,
kind text NOT NULL,
-- Own credentials (used when request_integration_id IS NULL):
base_url text,
api_key_ref text,
-- Optional soft reuse of a Requests arr server (live link; SET NULL on delete):
request_integration_id text REFERENCES public.request_integrations(id) ON DELETE SET NULL,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
CONSTRAINT autoscan_connections_source_present
CHECK (request_integration_id IS NOT NULL OR base_url IS NOT NULL)
);
CREATE TABLE public.autoscan_sources (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
installation_id integer NOT NULL,
capability_id text NOT NULL,
connection_id uuid NOT NULL REFERENCES public.autoscan_connections(id) ON DELETE RESTRICT,
enabled boolean NOT NULL DEFAULT false,
poll_interval_seconds integer,
marker text,
last_run_at timestamptz,
last_error text,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
CONSTRAINT autoscan_sources_interval_pos CHECK (poll_interval_seconds IS NULL OR poll_interval_seconds > 0),
CONSTRAINT autoscan_sources_capability_unique UNIQUE (installation_id, capability_id)
);
```
- [ ] **Step 2: Write the down migration**
Create `migrations/172_autoscan_v2.down.sql`:
```sql
DROP TABLE IF EXISTS public.autoscan_sources;
DROP TABLE IF EXISTS public.autoscan_connections;
DROP TABLE IF EXISTS public.autoscan_settings;
```
(Recreating the `171` tables on down-migration is unnecessary: `171` is unreleased and superseded; the down simply removes `172`'s tables.)
- [ ] **Step 3: Apply and verify against a disposable DB**
Run the migrations against a throwaway Postgres (the project's disposable-DB pattern) and confirm boot applies cleanly:
```bash
docker compose up -d postgres
go run ./cmd/silo --migrate-only 2>&1 | grep -i "migrations applied" || echo "check migration output"
```
Expected: migrations apply without error; `\d autoscan_connections` shows the soft FK `request_integration_id`.
- [ ] **Step 4: Commit**
```bash
git add migrations/172_autoscan_v2.up.sql migrations/172_autoscan_v2.down.sql
git commit -m "feat(migrations): autoscan v2 schema (connections + sources, no requests FK)"
```
---
## Task 4: Types and repository
**Files:**
- Rewrite: `internal/autoscan/types.go`
- Rewrite: `internal/autoscan/repository.go`
- [ ] **Step 1: Define the types**
Replace `internal/autoscan/types.go` with the v2 model:
```go
package autoscan
import "time"
type Settings struct {
Enabled bool
DefaultPollIntervalSeconds int
DebounceSeconds int
}
// Connection is an arr server the host can reach: either own credentials, or a
// live reference to a Requests integration (RequestIntegrationID set).
type Connection struct {
ID string
Name string
Kind string
BaseURL string // own; empty when linked
APIKeyRef string // own; empty when linked
RequestIntegrationID *string
}
// Source ties a scan_source plugin capability instance to a connection plus the
// host-owned scheduling/bookkeeping state.
type Source struct {
ID string
InstallationID int
CapabilityID string
ConnectionID string
Enabled bool
PollIntervalSeconds *int // nil => use settings default
Marker *string // opaque; nil on first run
LastRunAt *time.Time
LastError *string
}
```
- [ ] **Step 2: Write a failing repository test**
Add `internal/autoscan/repository_test.go` exercising round-trip + `AdvanceMarker` against a disposable DB (gate with the project's DB-test build tag/helper if one exists; otherwise a `//go:build integration` tag matching sibling DB tests). Minimal:
```go
func TestRepositoryMarkerRoundTrip(t *testing.T) {
repo := newTestRepo(t) // disposable-DB helper used by sibling repo tests
connID := repo.mustCreateConnection(t, Connection{Name: "Sonarr", Kind: "sonarr", BaseURL: "http://x", APIKeyRef: "k"})
srcID := repo.mustCreateSource(t, Source{InstallationID: 1, CapabilityID: "arr", ConnectionID: connID, Enabled: true})
if err := repo.AdvanceMarker(ctx, srcID, "2026-06-02T14:10:00Z"); err != nil {
t.Fatalf("AdvanceMarker: %v", err)
}
got, _ := repo.ListEnabledSources(ctx)
if len(got) != 1 || got[0].Marker == nil || *got[0].Marker != "2026-06-02T14:10:00Z" {
t.Fatalf("marker not persisted: %+v", got)
}
}
```
- [ ] **Step 3: Run to verify it fails**
Run: `go test ./internal/autoscan/ -run TestRepositoryMarkerRoundTrip` (with the DB helper available)
Expected: FAIL — undefined repository methods.
- [ ] **Step 4: Implement the repository**
Rewrite `internal/autoscan/repository.go` with: `GetSettings`, `UpdateSettings`, connection CRUD (`CreateConnection`, `UpdateConnection`, `DeleteConnection`, `ListConnections`, `GetConnection`), source CRUD (`UpsertSource`, `ListEnabledSources`, `ListSources`, `GetSource`), and `AdvanceMarker(ctx, sourceID, marker string)` (sets `marker`, `last_run_at = now()`, clears `last_error`). Use pgx; follow the column/scan style of the existing (pre-rewrite) `repository.go` and `internal/requests/repository.go`.
- [ ] **Step 5: Run to verify it passes**
Run: `go test ./internal/autoscan/ -run TestRepositoryMarkerRoundTrip`
Expected: PASS.
- [ ] **Step 6: Commit**
```bash
git add internal/autoscan/types.go internal/autoscan/repository.go internal/autoscan/repository_test.go
git commit -m "feat(autoscan): v2 types and repository (connections, sources, markers)"
```
---
## Task 5: Connection resolution (own or Requests-linked)
**Files:**
- Create: `internal/autoscan/connection.go`
- Create: `internal/autoscan/connection_test.go`
- [ ] **Step 1: Write the failing test**
```go
func TestResolveConnectionOwnVsLinked(t *testing.T) {
reqLookup := fakeRequestIntegrationLookup{"req-1": creds{BaseURL: "http://req:7878", APIKeyRef: "rk"}}
secrets := fakeSecrets{"rk": "REQKEY", "ownref": "OWNKEY"}
r := NewConnectionResolver(reqLookup, secrets)
own, err := r.Resolve(ctx, Connection{BaseURL: "http://own:8989", APIKeyRef: "ownref"})
if err != nil || own.BaseURL != "http://own:8989" || own.APIKey != "OWNKEY" {
t.Fatalf("own resolve = %+v, err=%v", own, err)
}
id := "req-1"
linked, err := r.Resolve(ctx, Connection{RequestIntegrationID: &id})
if err != nil || linked.BaseURL != "http://req:7878" || linked.APIKey != "REQKEY" {
t.Fatalf("linked resolve = %+v, err=%v", linked, err)
}
}
func TestResolveConnectionLinkedMissingFallsBack(t *testing.T) {
r := NewConnectionResolver(fakeRequestIntegrationLookup{}, fakeSecrets{})
id := "gone"
if _, err := r.Resolve(ctx, Connection{RequestIntegrationID: &id}); err == nil {
t.Fatal("expected error when linked Requests integration is missing")
}
}
```
- [ ] **Step 2: Run to verify it fails**
Run: `go test ./internal/autoscan/ -run TestResolveConnection`
Expected: FAIL — undefined `NewConnectionResolver`.
- [ ] **Step 3: Implement**
Create `internal/autoscan/connection.go`:
```go
package autoscan
import (
"context"
"fmt"
)
// ResolvedConnection is concrete credentials handed to the plugin.
type ResolvedConnection struct {
BaseURL string
APIKey string
}
type RequestIntegrationLookup interface {
// Returns base URL + encrypted api key ref for a Requests integration.
Get(ctx context.Context, integrationID string) (baseURL, apiKeyRef string, err error)
}
type SecretResolver interface {
Get(ctx context.Context, ref string) (string, error)
}
type ConnectionResolver struct {
requests RequestIntegrationLookup
secrets SecretResolver
}
func NewConnectionResolver(r RequestIntegrationLookup, s SecretResolver) *ConnectionResolver {
return &ConnectionResolver{requests: r, secrets: s}
}
func (cr *ConnectionResolver) Resolve(ctx context.Context, c Connection) (ResolvedConnection, error) {
baseURL, apiKeyRef := c.BaseURL, c.APIKeyRef
if c.RequestIntegrationID != nil {
u, ref, err := cr.requests.Get(ctx, *c.RequestIntegrationID)
if err != nil {
return ResolvedConnection{}, fmt.Errorf("autoscan: linked requests integration %q: %w", *c.RequestIntegrationID, err)
}
baseURL, apiKeyRef = u, ref
}
apiKey := apiKeyRef
if cr.secrets != nil && apiKeyRef != "" {
resolved, err := cr.secrets.Get(ctx, apiKeyRef)
if err != nil {
return ResolvedConnection{}, fmt.Errorf("autoscan: resolve api key: %w", err)
}
if resolved != "" {
apiKey = resolved
}
}
return ResolvedConnection{BaseURL: baseURL, APIKey: apiKey}, nil
}
```
- [ ] **Step 4: Run to verify it passes**
Run: `go test ./internal/autoscan/ -run TestResolveConnection`
Expected: PASS.
- [ ] **Step 5: Commit**
```bash
git add internal/autoscan/connection.go internal/autoscan/connection_test.go
git commit -m "feat(autoscan): resolve connections (own credentials or Requests-linked)"
```
---
## Task 6: Provider seam over the plugin resolver
**Files:**
- Create: `internal/autoscan/provider.go`
- [ ] **Step 1: Define the seam and adapter**
The engine must be testable without a live plugin, so it depends on a narrow interface, with a production adapter over `plugins.Service.ScanSourceClient`. Create `internal/autoscan/provider.go`:
```go
package autoscan
import (
"context"
pluginv1 "github.com/Silo-Server/silo-plugin-sdk/pkg/pluginproto/silo/plugin/v1"
)
// ScanSourceProvider yields changed paths for one source. The engine calls
// PollChanges; production wraps the plugins.Service scan_source resolver.
type ScanSourceProvider interface {
PollChanges(ctx context.Context, installationID int, capabilityID, marker string, conn ResolvedConnection) (paths []string, nextMarker string, err error)
}
// pluginScanSourceClient is the slice of *pluginhost.ScanSourceClient used here.
type pluginScanSourceClient interface {
PollChanges(ctx context.Context, req *pluginv1.PollChangesRequest) (*pluginv1.PollChangesResponse, error)
}
type scanSourceResolver interface {
ScanSourceClient(ctx context.Context, installationID int, capabilityID string) (pluginScanSourceClient, error)
}
type pluginProvider struct{ resolver scanSourceResolver }
func NewPluginProvider(resolver scanSourceResolver) ScanSourceProvider {
return &pluginProvider{resolver: resolver}
}
func (p *pluginProvider) PollChanges(ctx context.Context, installationID int, capabilityID, marker string, conn ResolvedConnection) ([]string, string, error) {
client, err := p.resolver.ScanSourceClient(ctx, installationID, capabilityID)
if err != nil {
return nil, "", err
}
resp, err := client.PollChanges(ctx, &pluginv1.PollChangesRequest{CapabilityId: capabilityID, Marker: marker})
if err != nil {
return nil, "", err
}
return resp.GetChangedPaths(), resp.GetNextMarker(), nil
}
```
Note: the connection (`conn`) is delivered to the plugin out-of-band as runtime config when the source is configured (the plugin reads its own `{base_url, api_key}`); it is passed here so a future provider variant could inject per-call. For v1 the production path configures the plugin instance with the resolved connection at upsert time; document this in `NewPluginProvider`'s comment.
- [ ] **Step 2: Build**
Run: `go build ./internal/autoscan/`
Expected: exits 0.
- [ ] **Step 3: Commit**
```bash
git add internal/autoscan/provider.go
git commit -m "feat(autoscan): scan-source provider seam over the plugin resolver"
```
---
## Task 7: Generalize the engine (`Service.PollOnce`)
**Files:**
- Rewrite: `internal/autoscan/service.go`
- Modify: `internal/autoscan/service_test.go` (generalize the salvaged tests)
- [ ] **Step 1: Adapt the salvaged tests to the provider seam**
The PR #43 `service_test.go` used a `fakeHistory`; replace it with a `fakeProvider` implementing `ScanSourceProvider`, keeping the same assertions (dedupe to one folder; distinct paths same folder → two scans; disabled no-op; provider error keeps marker; enqueue failure releases claims + keeps marker). Add an opaque-marker assertion: the provider's returned `nextMarker` is stored verbatim via `AdvanceMarker`.
```go
type fakeProvider struct {
paths map[string][]string // key: capabilityID
nextMarker string
err error
}
func (f *fakeProvider) PollChanges(_ context.Context, _ int, capabilityID, _ string, _ ResolvedConnection) ([]string, string, error) {
if f.err != nil {
return nil, "", f.err
}
return f.paths[capabilityID], f.nextMarker, nil
}
```
- [ ] **Step 2: Run to verify the tests fail**
Run: `go test ./internal/autoscan/ -run TestPollOnce`
Expected: FAIL — `Service` signature/`fakeProvider` not yet wired.
- [ ] **Step 3: Rewrite the engine**
Rewrite `internal/autoscan/service.go`. Keep the salvaged resolve→suppress→enqueue loop verbatim (it is provider-agnostic already), but drive it from sources + the provider seam:
```go
func (s *Service) PollOnce(ctx context.Context) error {
settings, err := s.store.GetSettings(ctx)
if err != nil {
return err
}
if !settings.Enabled {
return nil
}
sources, err := s.store.ListEnabledSources(ctx)
if err != nil {
return err
}
ttl := time.Duration(settings.DebounceSeconds) * time.Second
for _, src := range sources {
conn, cerr := s.resolveConnection(ctx, src.ConnectionID)
if cerr != nil {
slog.WarnContext(ctx, "autoscan: resolve connection failed", "source_id", src.ID, "err", cerr)
continue
}
marker := ""
if src.Marker != nil {
marker = *src.Marker
}
paths, next, perr := s.provider.PollChanges(ctx, src.InstallationID, src.CapabilityID, marker, conn)
if perr != nil {
slog.WarnContext(ctx, "autoscan: poll changes failed", "source_id", src.ID, "err", perr)
_ = s.store.RecordError(ctx, src.ID, perr.Error())
continue
}
targets, claimed := s.resolveAndClaim(ctx, paths, ttl) // salvaged dedupe→resolve→suppress
if len(targets) > 0 {
if eerr := s.queue.EnqueueScans(ctx, targets); eerr != nil {
s.releaseClaims(ctx, claimed)
slog.WarnContext(ctx, "autoscan: enqueue failed", "source_id", src.ID, "err", eerr)
continue // do NOT advance marker
}
}
if aerr := s.store.AdvanceMarker(ctx, src.ID, next); aerr != nil {
slog.WarnContext(ctx, "autoscan: advance marker failed", "source_id", src.ID, "err", aerr)
}
}
return nil
}
```
Extract the salvaged loop body into `resolveAndClaim`/`releaseClaims` helpers (lifted verbatim from PR #43's `service.go` lines that build `targets`/`claimed`). Keep `uniqueParentDirs`, `Suppressor`, the `fmt.Sprintf("%d|%s", folderID, target.Path)` suppression key, and the `scantrigger.RequestError` quiet-skip.
- [ ] **Step 4: Run to verify the tests pass**
Run: `go test ./internal/autoscan/ -run TestPollOnce`
Expected: PASS.
- [ ] **Step 5: Commit**
```bash
git add internal/autoscan/service.go internal/autoscan/service_test.go
git commit -m "feat(autoscan): generic engine drives sources via scan_source provider"
```
---
## Task 8: Delete arr-specific salvage residue
**Files:**
- Delete: `internal/autoscan/history.go`, `history_test.go`, `rewrite.go`, `rewrite_test.go`, `suggest.go`, `suggest_deps.go`, `suggest_test.go`
- [ ] **Step 1: Remove the files**
```bash
git rm internal/autoscan/history.go internal/autoscan/history_test.go \
internal/autoscan/rewrite.go internal/autoscan/rewrite_test.go \
internal/autoscan/suggest.go internal/autoscan/suggest_deps.go internal/autoscan/suggest_test.go
```
- [ ] **Step 2: Build to confirm no remaining references**
Run: `go build ./internal/autoscan/...`
Expected: exits 0 (the engine no longer references the deleted arr logic). If it fails, a reference was missed — remove it.
- [ ] **Step 3: Commit**
```bash
git commit -m "refactor(autoscan): drop arr-specific history/rewrite/suggest (moved to plugin)"
```
---
## Task 9: Admin API rewrite
**Files:**
- Rewrite: `internal/api/handlers/autoscan.go`
- Modify: `internal/api/handlers/autoscan_test.go`
- [ ] **Step 1: Define the endpoints**
Rewrite the handler to back the new model. Endpoints (admin-gated, mounted in Task 10):
- `GET/PUT /admin/autoscan/settings`
- `GET/POST /admin/autoscan/connections`, `PUT/DELETE /admin/autoscan/connections/{id}`
- `GET /admin/autoscan/sources`, `PUT /admin/autoscan/sources/{id}` (enable, interval, connection binding)
- `POST /admin/autoscan/trigger` (async `PollOnce`, 202 — keep the detached-goroutine + channel-tested pattern from the closed PR's `autoscan_test.go`)
- `GET /admin/autoscan/status`
Strip credentials from responses (never emit `api_key_ref`/resolved keys), matching the `_SENSITIVE_METADATA_KEYS` defense-in-depth posture.
- [ ] **Step 2: Port + extend the handler tests**
Carry over the trigger test (the `done`-channel synchronization fix from `d3ee373`), the FK→404 test (now: unknown source/connection → 404), and a secrets-not-leaked test for the connections list.
- [ ] **Step 3: Run the handler tests (libvips container)**
Run (in the libvips-equipped container per Prerequisites #3):
`go test ./internal/api/handlers/ -run Autoscan`
Expected: PASS.
- [ ] **Step 4: Commit**
```bash
git add internal/api/handlers/autoscan.go internal/api/handlers/autoscan_test.go
git commit -m "feat(api): autoscan v2 admin endpoints (settings, connections, sources)"
```
---
## Task 10: Wiring + rate limits + retire old coupling
**Files:**
- Modify: `internal/api/router.go`
- Modify: `cmd/silo/main.go`
- [ ] **Step 1: Rebuild the service wiring**
In `internal/api/router.go` and `cmd/silo/main.go`, construct the v2 service:
```go
autoscanRepo := autoscan.NewRepository(deps.DB)
provider := autoscan.NewPluginProvider(pluginServiceScanSourceAdapter{pluginService})
connResolver := autoscan.NewConnectionResolver(requestIntegrationLookup{requestsRepo}, serverSettingsSecretResolver{settingsRepo})
autoscanSvc := autoscan.NewService(autoscanRepo, provider, connResolver, deps.Resolver, deps.ScanQueue, autoscan.NewRedisSuppressor(deps.RedisClient))
autoscanHandler = handlers.NewAutoscanHandler(autoscanRepo, autoscanSvc)
```
Replace the old route block with the Task 9 routes. Keep the `catalog_*`/admin rate-limit conventions consistent with the security standards (per-user/admin caps on the new endpoints; mirror the request-integrations admin tier).
- [ ] **Step 2: Retarget the poll task**
`internal/taskmanager/tasks/autoscan_poll.go` calls `Service.PollOnce` — unchanged behavior; confirm it compiles against the rebuilt service and uses `Settings.DefaultPollIntervalSeconds` for the interval (per-source overrides are honored inside `PollOnce`).
- [ ] **Step 3: Build the whole tree**
Run (libvips container): `go build ./...`
Expected: exits 0.
- [ ] **Step 4: Commit**
```bash
git add internal/api/router.go cmd/silo/main.go internal/taskmanager/tasks/autoscan_poll.go
git commit -m "feat(autoscan): wire v2 service, routes, and poll task"
```
---
## Task 11: Full sweep + finalize SDK dependency
**Files:**
- Modify: `go.mod`, `go.sum`
- [ ] **Step 1: Full build + test (libvips container)**
Run in the libvips-equipped container (Prerequisites #3):
```bash
go build ./... && go test ./... && go vet ./...
```
Expected: build OK; all packages `ok`; vet clean. `gofmt -l internal migrations` prints nothing.
- [ ] **Step 2: Finalize the SDK version (after `v0.5.0` is tagged upstream)**
Once `silo-plugin-sdk` PR #2 is merged and `v0.5.0` tagged:
```bash
go mod edit -dropreplace github.com/Silo-Server/silo-plugin-sdk
go get github.com/Silo-Server/silo-plugin-sdk@v0.5.0
go mod tidy
go build ./...
```
Expected: builds against the released tag with no replace directive.
If `v0.5.0` is not yet tagged when this plan is otherwise complete, leave the replace directive in place, commit it, and note in the PR that the dependency must be finalized before merge (do not merge a `replace` pointing at a local path).
- [ ] **Step 3: Commit**
```bash
git add go.mod go.sum
git commit -m "build: depend on silo-plugin-sdk v0.5.0 (scan_source.v1)"
```
---
## Self-Review notes
- **Spec coverage:** §7 engine (Tasks 6–7), §8 connections incl. reuse-or-own + soft Requests link (Tasks 3,5), §9 data model decoupled from Requests + Autoscan-owned state (Tasks 3–4,9–10), §5/§11 host consumes `scan_source.v1` (Tasks 1–2). §10 (arr plugin) and §9 UI are separate plans.
- **Salvage map honored (§13):** engine/dedupe/suppress kept and generalized (Tasks 6–8); arr history/rewrite/suggest deleted (Task 8); `last_poll_at` → opaque `marker` (Tasks 3–4,7).
- **Out of scope here:** the Autoscan admin **UI** category (Part 2 plan) and the **arr plugin** (separate repo plan). This backend is testable on its own via the repository/engine/handler tests.
- **Risk carried from spec §14:** plugin multiplicity (one installation per arr server vs. many connections per installation) is modeled as one `autoscan_sources` row per `(installation_id, capability_id)`; if the runtime supports multiple capability instances per installation this maps cleanly, otherwise it implies one installation per server. Confirm during Task 1–2 by inspecting `ensureClient`/installation handling.
- **Type consistency:** `ResolvedConnection{BaseURL,APIKey}`, `Connection{BaseURL,APIKeyRef,RequestIntegrationID}`, `Source{InstallationID,CapabilityID,ConnectionID,Marker}`, `ScanSourceProvider.PollChanges(...)`, `Service.PollOnce` used consistently across Tasks 4–10.
docs/superpowers/plans/2026-06-02-autoscan-rewrite-sync.md
+694
View File
@@ -0,0 +1,694 @@
# Autoscan Rewrite-Sync Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** A "Sync rewrites from arr" action that suggests autoscan `path_rewrites` for an instance by matching its arr root folders to Silo media folders on shared trailing path segments, previewed and committed by the admin.
**Architecture:** A pure `suggestRewrites` matcher in `internal/autoscan`, a `Service.SuggestRewrites` method backed by two new injected deps (an arr root-folder client and a Silo folder lister), one read-only admin endpoint returning a preview, and a "Sync from arr" button in the existing Autoscan tab that merges proposals into the editor (admin saves via the existing per-source PUT). Manual rewrites stay the source of truth.
**Tech Stack:** Go (standard `testing`), React/TypeScript.
**Spec:** `docs/superpowers/specs/2026-06-02-autoscan-rewrite-sync-design.md`
**Commands assume the repository root is the cwd.** Go tests: `go test ./internal/autoscan/...`. Frontend: `cd web && pnpm exec tsc -b && pnpm run lint`. Ensure the Go toolchain is on `PATH` (prepend its `bin` directory if `go` is missing). `internal/api`/`internal/api/handlers` cannot be compiled without libvips; verify those via `gofmt` + the Docker build.
---
## Phase 0 — Branch
- [ ] **Step 0.1: Confirm branch**
```bash
git rev-parse --abbrev-ref HEAD # expect: feat/autoscan-arr-polling
```
This feature extends the autoscan branch. If you're elsewhere: `git checkout feat/autoscan-arr-polling`.
---
## Phase 1 — Core matcher (pure, TDD)
### Task 1: suggestRewrites + types
**Files:**
- Create: `internal/autoscan/suggest.go`
- Create: `internal/autoscan/suggest_test.go`
- [ ] **Step 1.1: Write the failing test**
`internal/autoscan/suggest_test.go`:
```go
package autoscan
import (
"reflect"
"testing"
)
func TestSuggestRewrites(t *testing.T) {
silo := []string{
"/mnt/media/happy/storage2/tvshows1",
"/mnt/media/happy4k/4ktv7",
"/mnt/media/storage/Anime/Subs",
"/mnt/media/storage/Anime2/Subs",
"/library/Films",
"/tank/television/Show",
}
t.Run("multi-segment unique match", func(t *testing.T) {
got := suggestRewrites([]string{"/mnt/happy/storage2/tvshows1"}, silo, nil)
if len(got.Proposed) != 1 || got.Proposed[0].To != "/mnt/media/happy/storage2/tvshows1" || got.Proposed[0].MatchDepth != 2 {
t.Fatalf("proposed=%+v", got.Proposed)
}
})
t.Run("single-segment unique match (different parents)", func(t *testing.T) {
got := suggestRewrites([]string{"/mnt/kodama/storage2/4ktv7"}, silo, nil)
if len(got.Proposed) != 1 || got.Proposed[0].To != "/mnt/media/happy4k/4ktv7" || got.Proposed[0].MatchDepth != 1 {
t.Fatalf("proposed=%+v", got.Proposed)
}
})
t.Run("longest-suffix disambiguation", func(t *testing.T) {
got := suggestRewrites([]string{"/mnt/kodama/storage1/Anime/Subs"}, silo, nil)
if len(got.Proposed) != 1 || got.Proposed[0].To != "/mnt/media/storage/Anime/Subs" || got.Proposed[0].MatchDepth != 2 {
t.Fatalf("expected Anime/Subs (depth 2), got %+v", got.Proposed)
}
if len(got.Ambiguous) != 0 {
t.Fatalf("should not be ambiguous: %+v", got.Ambiguous)
}
})
t.Run("unmatched when no shared segment", func(t *testing.T) {
got := suggestRewrites([]string{"/data/Movies"}, silo, nil)
if len(got.Unmatched) != 1 || got.Unmatched[0] != "/data/Movies" || len(got.Proposed) != 0 {
t.Fatalf("got=%+v", got)
}
})
t.Run("leaf match across unlike layouts", func(t *testing.T) {
got := suggestRewrites([]string{"/srv/tv/Show"}, silo, nil)
if len(got.Proposed) != 1 || got.Proposed[0].To != "/tank/television/Show" {
t.Fatalf("got=%+v", got)
}
})
t.Run("ambiguous tie", func(t *testing.T) {
// a root whose only shared segment is "Subs" ties two folders
got := suggestRewrites([]string{"/foo/bar/Subs"}, silo, nil)
if len(got.Ambiguous) != 1 || len(got.Ambiguous[0].Candidates) != 2 {
t.Fatalf("expected ambiguous with 2 candidates, got %+v", got.Ambiguous)
}
})
t.Run("covered by existing rule", func(t *testing.T) {
existing := []PathRewrite{{From: "/mnt/happy", To: "/mnt/media/happy"}}
got := suggestRewrites([]string{"/mnt/happy/storage2/tvshows1"}, silo, existing)
if len(got.Covered) != 1 || len(got.Proposed) != 0 {
t.Fatalf("expected covered, got %+v", got)
}
})
t.Run("normalization: trailing slash, backslashes, dup slashes", func(t *testing.T) {
got := suggestRewrites([]string{`\mnt\happy\\storage2\tvshows1\`}, silo, nil)
if len(got.Proposed) != 1 || got.Proposed[0].From != "/mnt/happy/storage2/tvshows1" || got.Proposed[0].To != "/mnt/media/happy/storage2/tvshows1" {
t.Fatalf("normalization failed: %+v", got.Proposed)
}
})
}
func TestCommonSuffixLen(t *testing.T) {
cases := []struct {
a, b []string
want int
}{
{[]string{"a", "b", "c"}, []string{"x", "b", "c"}, 2},
{[]string{"4ktv7"}, []string{"happy4k", "4ktv7"}, 1},
{[]string{"4ktv7"}, []string{"4ktv70"}, 0}, // full-segment, not substring
{[]string{"a"}, []string{"b"}, 0},
}
for _, tc := range cases {
if got := commonSuffixLen(tc.a, tc.b); got != tc.want {
t.Fatalf("commonSuffixLen(%v,%v)=%d want %d", tc.a, tc.b, got, tc.want)
}
}
}
```
- [ ] **Step 1.2: Run it — expect failure**
```bash
go test ./internal/autoscan/ -run 'TestSuggestRewrites|TestCommonSuffixLen' -v
```
Expected: FAIL (`suggestRewrites` undefined).
- [ ] **Step 1.3: Implement** `internal/autoscan/suggest.go`:
```go
package autoscan
import "strings"
// RewriteSuggestions is the result of matching arr root folders to Silo folders.
type RewriteSuggestions struct {
Proposed []ProposedRewrite `json:"proposed"`
Unmatched []string `json:"unmatched"`
Ambiguous []AmbiguousRoot `json:"ambiguous"`
Covered []string `json:"covered"`
}
// ProposedRewrite is a suggested rewrite plus its confidence (shared trailing segments).
type ProposedRewrite struct {
From string `json:"from"`
To string `json:"to"`
MatchDepth int `json:"match_depth"`
}
// AmbiguousRoot is an arr root that tied across multiple Silo folders.
type AmbiguousRoot struct {
Root string `json:"root"`
Candidates []string `json:"candidates"`
}
// normalizePath makes a path comparable: backslashes -> '/', collapse duplicate
// slashes, strip a trailing slash (but keep a bare "/").
func normalizePath(p string) string {
p = normalizeSeparators(strings.TrimSpace(p))
for strings.Contains(p, "//") {
p = strings.ReplaceAll(p, "//", "/")
}
if len(p) > 1 {
p = strings.TrimRight(p, "/")
}
return p
}
func segments(p string) []string {
p = strings.Trim(p, "/")
if p == "" {
return nil
}
return strings.Split(p, "/")
}
// commonSuffixLen counts equal trailing segments (full-segment equality).
func commonSuffixLen(a, b []string) int {
i, j, n := len(a)-1, len(b)-1, 0
for i >= 0 && j >= 0 && a[i] == b[j] {
n++
i--
j--
}
return n
}
// coveredBy reports whether an existing rewrite already matches root (same
// boundary rule as applyRewrites).
func coveredBy(root string, existing []PathRewrite) bool {
for _, rw := range existing {
from := strings.TrimRight(strings.TrimSpace(rw.From), "/")
if from == "" {
continue
}
if root == from || strings.HasPrefix(root, from+"/") {
return true
}
}
return false
}
// suggestRewrites matches each arr root to the Silo folder sharing the most
// trailing path segments (unique winner). Roots already handled by an existing
// rewrite are reported as Covered; roots with no shared segment as Unmatched;
// ties as Ambiguous. Pure: no I/O, no deployment constants.
func suggestRewrites(arrRoots, siloFolderPaths []string, existing []PathRewrite) RewriteSuggestions {
siloNorm := make([]string, 0, len(siloFolderPaths))
siloSegs := make([][]string, 0, len(siloFolderPaths))
for _, p := range siloFolderPaths {
n := normalizePath(p)
if n == "" {
continue
}
siloNorm = append(siloNorm, n)
siloSegs = append(siloSegs, segments(n))
}
var out RewriteSuggestions
for _, raw := range arrRoots {
root := normalizePath(raw)
if root == "" {
continue
}
if coveredBy(root, existing) {
out.Covered = append(out.Covered, root)
continue
}
rootSegs := segments(root)
best := 0
var winners []string
for i, segs := range siloSegs {
n := commonSuffixLen(rootSegs, segs)
if n == 0 {
continue
}
if n > best {
best, winners = n, []string{siloNorm[i]}
} else if n == best {
winners = append(winners, siloNorm[i])
}
}
switch {
case best == 0:
out.Unmatched = append(out.Unmatched, root)
case len(winners) == 1:
out.Proposed = append(out.Proposed, ProposedRewrite{From: root, To: winners[0], MatchDepth: best})
default:
out.Ambiguous = append(out.Ambiguous, AmbiguousRoot{Root: root, Candidates: winners})
}
}
return out
}
```
- [ ] **Step 1.4: Run it — expect pass; commit**
```bash
go test ./internal/autoscan/ -run 'TestSuggestRewrites|TestCommonSuffixLen' -v
gofmt -l internal/autoscan/suggest.go internal/autoscan/suggest_test.go
git add internal/autoscan/suggest.go internal/autoscan/suggest_test.go
git commit -m "feat(autoscan): suffix-match rewrite suggester"
```
---
## Phase 2 — Service support
### Task 2: Repository.GetSource
**Files:**
- Modify: `internal/autoscan/repository.go`
- [ ] **Step 2.1: Implement** `GetSource` (single-source join, includes disabled). Add after `ListEnabledSources`:
```go
// GetSource returns one instance's autoscan state joined with its
// request_integrations row (kind/base_url/api_key_ref), regardless of enabled.
func (r *Repository) GetSource(ctx context.Context, integrationID string) (*Source, error) {
row := r.pool.QueryRow(ctx, sourceSelect+` WHERE ri.id = $1`, integrationID)
src, err := scanSource(row)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return nil, fmt.Errorf("%w: %s", ErrIntegrationNotFound, integrationID)
}
return nil, fmt.Errorf("get autoscan source: %w", err)
}
return &src, nil
}
```
(`errors`, `pgx`, `fmt`, `ErrIntegrationNotFound`, and `sourceSelect`/`scanSource` already exist in the file.)
- [ ] **Step 2.2: Build + commit**
```bash
go build ./internal/autoscan/... && go test ./internal/autoscan/...
git add internal/autoscan/repository.go
git commit -m "feat(autoscan): GetSource single-source lookup"
```
### Task 3: RootFolderClient + FolderLister adapters
**Files:**
- Create: `internal/autoscan/suggest_deps.go`
- [ ] **Step 3.1: Implement the two adapters + interfaces**
```go
package autoscan
import (
"context"
"net/http"
"github.com/Silo-Server/silo-server/internal/catalog"
"github.com/Silo-Server/silo-server/internal/requests/arrclient"
)
// RootFolderClient lists a Radarr/Sonarr instance's configured root folder paths.
type RootFolderClient interface {
RootFolders(ctx context.Context, baseURL, apiKey string) ([]string, error)
}
// FolderLister lists every Silo media-folder path.
type FolderLister interface {
ListFolderPaths(ctx context.Context) ([]string, error)
}
type arrRootFolderClient struct{ httpClient *http.Client }
// NewArrRootFolderClient returns a RootFolderClient backed by the shared arrclient.
func NewArrRootFolderClient(httpClient *http.Client) RootFolderClient {
return &arrRootFolderClient{httpClient: httpClient}
}
func (c *arrRootFolderClient) RootFolders(ctx context.Context, baseURL, apiKey string) ([]string, error) {
client := arrclient.New(baseURL, apiKey, c.httpClient)
folders, err := arrclient.ListRootFolders(ctx, client)
if err != nil {
return nil, err
}
paths := make([]string, 0, len(folders))
for _, f := range folders {
if f.Path != "" {
paths = append(paths, f.Path)
}
}
return paths, nil
}
type catalogFolderLister struct{ repo *catalog.FolderRepository }
// NewCatalogFolderLister adapts catalog.FolderRepository to FolderLister.
func NewCatalogFolderLister(repo *catalog.FolderRepository) FolderLister {
return &catalogFolderLister{repo: repo}
}
func (l *catalogFolderLister) ListFolderPaths(ctx context.Context) ([]string, error) {
folders, err := l.repo.List(ctx)
if err != nil {
return nil, err
}
var paths []string
for _, f := range folders {
paths = append(paths, f.Paths...)
}
return paths, nil
}
```
Note: `autoscan` already depends on `catalog` (via `scantrigger`) and on `requests/arrclient` (via `history.go`), so these imports add no new module-level coupling and no cycle (`catalog` does not import `autoscan`).
- [ ] **Step 3.2: Build + commit**
```bash
go build ./internal/autoscan/...
git add internal/autoscan/suggest_deps.go
git commit -m "feat(autoscan): arr root-folder client + Silo folder lister"
```
### Task 4: Service.SuggestRewrites
**Files:**
- Modify: `internal/autoscan/service.go`
- Modify: `internal/autoscan/service_test.go`
- [ ] **Step 4.1: Write the failing test** (append to `service_test.go`):
```go
type fakeRootFolders struct {
paths []string
err error
}
func (f fakeRootFolders) RootFolders(context.Context, string, string) ([]string, error) {
return f.paths, f.err
}
type fakeFolderLister struct{ paths []string }
func (f fakeFolderLister) ListFolderPaths(context.Context) ([]string, error) { return f.paths, nil }
type sourceGetterStore struct {
fakeStore
src *Source
}
func (s *sourceGetterStore) GetSource(context.Context, string) (*Source, error) { return s.src, nil }
func TestSuggestRewritesService(t *testing.T) {
store := &sourceGetterStore{src: &Source{IntegrationID: "i1", Kind: "sonarr", BaseURL: "http://x", APIKeyRef: "k"}}
svc := NewService(store, &fakeHistory{}, fakeResolver{}, &recordingQueuer{}, allowSuppressor{}, nil)
svc.SetRewriteResolvers(
fakeRootFolders{paths: []string{"/mnt/happy/storage2/tvshows1", "/data/Movies"}},
fakeFolderLister{paths: []string{"/mnt/media/happy/storage2/tvshows1"}},
)
got, err := svc.SuggestRewrites(context.Background(), "i1")
if err != nil {
t.Fatalf("SuggestRewrites: %v", err)
}
if len(got.Proposed) != 1 || got.Proposed[0].To != "/mnt/media/happy/storage2/tvshows1" {
t.Fatalf("proposed=%+v", got.Proposed)
}
if len(got.Unmatched) != 1 || got.Unmatched[0] != "/data/Movies" {
t.Fatalf("unmatched=%+v", got.Unmatched)
}
}
```
> The `Store` interface needs a `GetSource` method so the service can call it. Add `GetSource(ctx context.Context, integrationID string) (*Source, error)` to the `Store` interface in `service.go`. `*Repository` already implements it (Task 2). The test's `sourceGetterStore` embeds the existing `fakeStore` and adds `GetSource`. If `fakeStore` does not satisfy the widened interface on its own, that's fine — only `sourceGetterStore` is used here; but ensure the package still compiles (other tests construct `fakeStore` directly and pass it where `Store` is required — add a `GetSource` method to `fakeStore` returning `(nil, nil)` so it still satisfies `Store`).
- [ ] **Step 4.2: Run it — expect failure**
```bash
go test ./internal/autoscan/ -run TestSuggestRewritesService -v
```
Expected: FAIL (`SetRewriteResolvers`/`SuggestRewrites` undefined).
- [ ] **Step 4.3: Implement** in `service.go`. Add fields + setter + method, and add `GetSource` to `Store`:
```go
// add to the Store interface:
// GetSource(ctx context.Context, integrationID string) (*Source, error)
// add fields to Service struct:
// rootFolders RootFolderClient
// folders FolderLister
// SetRewriteResolvers wires the deps used by SuggestRewrites (optional; only the
// admin-facing service needs them).
func (s *Service) SetRewriteResolvers(rootFolders RootFolderClient, folders FolderLister) {
s.rootFolders = rootFolders
s.folders = folders
}
// SuggestRewrites matches an instance's arr root folders to Silo media folders.
func (s *Service) SuggestRewrites(ctx context.Context, integrationID string) (RewriteSuggestions, error) {
if s.rootFolders == nil || s.folders == nil {
return RewriteSuggestions{}, fmt.Errorf("autoscan: rewrite suggestion not configured")
}
src, err := s.store.GetSource(ctx, integrationID)
if err != nil {
return RewriteSuggestions{}, err
}
apiKey := src.APIKeyRef
if s.secrets != nil && apiKey != "" {
if resolved, rerr := s.secrets.Get(ctx, apiKey); rerr == nil && resolved != "" {
apiKey = resolved
}
}
arrRoots, err := s.rootFolders.RootFolders(ctx, src.BaseURL, apiKey)
if err != nil {
return RewriteSuggestions{}, fmt.Errorf("autoscan: list arr root folders: %w", err)
}
siloPaths, err := s.folders.ListFolderPaths(ctx)
if err != nil {
return RewriteSuggestions{}, fmt.Errorf("autoscan: list silo folders: %w", err)
}
return suggestRewrites(arrRoots, siloPaths, src.PathRewrites), nil
}
```
Add `GetSource` to `fakeStore` in `service_test.go` (`func (f *fakeStore) GetSource(context.Context, string) (*Source, error) { return nil, nil }`).
- [ ] **Step 4.4: Run tests — expect pass; commit**
```bash
go test ./internal/autoscan/...
gofmt -l internal/autoscan/service.go internal/autoscan/service_test.go
git add internal/autoscan/service.go internal/autoscan/service_test.go
git commit -m "feat(autoscan): Service.SuggestRewrites"
```
---
## Phase 3 — API endpoint
### Task 5: Handler + route
**Files:**
- Modify: `internal/api/handlers/autoscan.go`
- Modify: `internal/api/router.go`
- Modify: `internal/api/handlers/autoscan_test.go`
- [ ] **Step 5.1: Extend the handler's service interface + add the handler method**
In `autoscan.go`, add `SuggestRewrites` to the `autoscanTriggerer` interface:
```go
type autoscanTriggerer interface {
PollOnce(ctx context.Context) error
SuggestRewrites(ctx context.Context, integrationID string) (autoscan.RewriteSuggestions, error)
}
```
Add the handler:
```go
func (h *AutoscanHandler) HandleRewriteSuggestions(w http.ResponseWriter, r *http.Request) {
id := strings.TrimSpace(chi.URLParam(r, "id"))
suggestions, err := h.svc.SuggestRewrites(r.Context(), id)
if err != nil {
if errors.Is(err, autoscan.ErrIntegrationNotFound) {
writeError(w, http.StatusNotFound, "not_found", "Autoscan source not found")
return
}
// arr unreachable / bad key / folder list failure
writeError(w, http.StatusBadGateway, "arr_unreachable", "Could not load root folders from the arr instance")
return
}
writeJSON(w, http.StatusOK, suggestions)
}
```
(`errors`, `autoscan`, `chi`, `strings`, `writeError`, `writeJSON` already imported.)
- [ ] **Step 5.2: Mount the route** in `router.go`, inside the autoscan route block:
```go
r.Get("/autoscan/sources/{id}/rewrite-suggestions", autoscanHandler.HandleRewriteSuggestions)
```
- [ ] **Step 5.3: Wire the service deps** in `router.go` where `autoscanSvc` is built (after `autoscan.NewService(...)`):
```go
autoscanSvc.SetRewriteResolvers(
autoscan.NewArrRootFolderClient(nil),
autoscan.NewCatalogFolderLister(deps.FolderRepo),
)
```
(This is inside the existing `if deps.FolderRepo != nil && deps.LibraryScanQueue != nil` block, so `deps.FolderRepo` is non-nil here.)
- [ ] **Step 5.4: Update the handler test fake**
In `autoscan_test.go`, the fake satisfying `autoscanTriggerer` (e.g. `fakeAutoscanTriggerer`) gains:
```go
func (f *fakeAutoscanTriggerer) SuggestRewrites(context.Context, string) (autoscan.RewriteSuggestions, error) {
return autoscan.RewriteSuggestions{}, nil
}
```
Add a test asserting `HandleRewriteSuggestions` returns 200 with JSON for a known id (mirror the existing handler tests).
- [ ] **Step 5.5: Verify (limited) + commit**
```bash
# ensure the Go toolchain is on PATH
go vet ./internal/autoscan/...
gofmt -l internal/api/handlers/autoscan.go internal/api/router.go internal/api/handlers/autoscan_test.go
# internal/api cannot compile here (libvips) — Docker verifies later.
git add internal/api/handlers/autoscan.go internal/api/router.go internal/api/handlers/autoscan_test.go
git commit -m "feat(autoscan): rewrite-suggestions endpoint"
```
---
## Phase 4 — Frontend
### Task 6: Types + hook
**Files:**
- Modify: `web/src/api/types.ts`
- Modify: `web/src/hooks/queries/useAutoscan.ts`
- [ ] **Step 6.1: Types** — add to `types.ts`:
```ts
export interface AutoscanProposedRewrite {
from: string;
to: string;
match_depth: number;
}
export interface AutoscanAmbiguousRoot {
root: string;
candidates: string[];
}
export interface AutoscanRewriteSuggestions {
proposed: AutoscanProposedRewrite[];
unmatched: string[];
ambiguous: AutoscanAmbiguousRoot[];
covered: string[];
}
```
- [ ] **Step 6.2: Hook** — add to `useAutoscan.ts` a mutation that fetches suggestions for an id:
```ts
export function useAutoscanRewriteSuggestions() {
return useMutation({
mutationFn: (id: string) =>
api<AutoscanRewriteSuggestions>(`/admin/autoscan/sources/${id}/rewrite-suggestions`),
onError: (err) =>
toast.error(err instanceof Error ? err.message : "Could not sync rewrites from the arr instance"),
});
}
```
Match the import/`api`/`toast`/`useMutation` conventions already in the file (it imports `api` and `toast` already; reuse them; import the new type).
- [ ] **Step 6.3: Lint + commit**
```bash
cd web && pnpm exec tsc -b && pnpm run lint && pnpm exec prettier --check src/api/types.ts src/hooks/queries/useAutoscan.ts
git add web/src/api/types.ts web/src/hooks/queries/useAutoscan.ts
git commit -m "feat(web): autoscan rewrite-suggestions types and hook"
```
### Task 7: Sync button + preview in AutoscanSourceEditor
**Files:**
- Modify: `web/src/pages/AdminRequests.tsx`
- [ ] **Step 7.1: Add the button + preview**
In `AutoscanSourceEditor`:
- Call `const suggest = useAutoscanRewriteSuggestions();` and add local state `const [preview, setPreview] = useState<AutoscanRewriteSuggestions | null>(null);` and `const [selected, setSelected] = useState<Set<string>>(new Set());`.
- Add a **"Sync from arr"** `Button` beside Save: `onClick={async () => { const s = await suggest.mutateAsync(source.integration_id); setPreview(s); setSelected(new Set(s.proposed.map((p) => p.from))); }}` (disabled while `suggest.isPending`).
- When `preview` is set, render a panel (reuse the file's `Dialog` or an inline bordered panel) with three sections:
- **Proposed:** each `proposed[]` row — a checkbox bound to `selected.has(p.from)` (toggle updates the set), `from → to`, and a `Badge` showing confidence: `match_depth >= 2 ? "${match_depth} segments" : "1 segment — weak"` (destructive/secondary variant for weak).
- **Unmatched:** muted list of `unmatched[]` ("no Silo match — add manually if needed").
- **Ambiguous:** list of `ambiguous[]` (`root` + its `candidates` joined) — info only.
- An **"Add selected to rewrites"** button that merges the checked proposals into the editor's existing rewrites state, deduped by `from` (skip any whose `from` already exists), then `setPreview(null)`. Do NOT auto-save — the admin then clicks the existing **Save**.
- Keep the existing manual path-rewrite editor unchanged; sync only appends rows to it.
Reuse existing `Button`/`Badge`/`Field`/`Dialog` primitives and the editor's existing rewrites-state setter.
- [ ] **Step 7.2: Lint + commit**
```bash
cd web && pnpm exec tsc -b && pnpm run lint && pnpm exec prettier --check src/pages/AdminRequests.tsx
git add web/src/pages/AdminRequests.tsx
git commit -m "feat(web): autoscan sync-rewrites preview"
```
---
## Phase 5 — Verification
### Task 8: Full verification
- [ ] **Step 8.1: Go**
```bash
# ensure the Go toolchain is on PATH
go test ./internal/autoscan/... 2>&1 | tail
go vet ./internal/autoscan/...
gofmt -l internal/autoscan internal/api/handlers/autoscan.go internal/api/router.go
```
Expected: all pass; vet/gofmt clean.
- [ ] **Step 8.2: Frontend**
```bash
cd web && pnpm exec tsc -b && pnpm run lint && pnpm run format:check
```
- [ ] **Step 8.3: Docker build (closes the libvips gap)**
```bash
docker build --build-arg BUILD_REVISION=$(git rev-parse HEAD) --build-arg BUILD_DIRTY=false -t silo-server:autoscan .
```
Expected: exit 0 (handlers + router + frontend compile).
- [ ] **Step 8.4: Smoke (optional)**
On a deploy with arr instances configured: open an instance in the Autoscan tab, click **Sync from arr**, confirm the preview shows proposed (with confidence) + unmatched + ambiguous, "Add selected" appends deduped rows, and Save persists them.
---
## Self-review notes (resolved)
- **Spec §1 matcher** → Task 1 (incl. normalization, covered-first, suffix depth, unlike-deployment fixtures). **§2 endpoint/service** → Tasks 2–5 (`GetSource`, `RootFolderClient`/`FolderLister`, `SuggestRewrites`, endpoint, 404/502 mapping). **§3 frontend** → Tasks 6–7. **§4 testing** → Tasks 1,4,5 + Task 8.
- **Manual-preserving**: sync only *appends* deduped rows to the editor; the admin saves via the existing `PUT` (Task 7). Never auto-writes.
- **Generic**: `suggestRewrites` takes plain string slices, no deployment constants; tests include `/data/Movies` and `/srv/tv` fixtures (Task 1).
- **Optional deps**: `SetRewriteResolvers` keeps `main.go`'s poll service untouched; `SuggestRewrites` errors if unset (Task 4).
- **Type consistency**: `RewriteSuggestions`/`ProposedRewrite`/`AmbiguousRoot` (Go) ↔ `AutoscanRewriteSuggestions`/`AutoscanProposedRewrite`/`AutoscanAmbiguousRoot` (TS); `match_depth` JSON tag matches.
docs/superpowers/plans/2026-06-02-autoscan-scan-source-sdk-capability.md
+375
View File
@@ -0,0 +1,375 @@
# `scan_source.v1` SDK Capability — Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add a new, additive `scan_source.v1` plugin capability to `silo-plugin-sdk` so the Silo host can pull changed paths from a scan-source plugin on a timer.
**Architecture:** Mirror the existing `scheduled_task.v1` capability exactly — a single-RPC, request/response gRPC service defined in a `.proto`, generated into `pkg/pluginproto`, then registered in three Go helper sites (`capability.KnownTypes`, the `runtime.CapabilityServers` struct, and the `runtime.GRPCPlugin` server registration). Nothing existing is modified in shape; only new symbols are added. Tag a new SDK release so the host and the arr plugin can build against it.
**Tech Stack:** Go, Protocol Buffers (proto3), `buf` codegen (`buf.gen.yaml`), `google.golang.org/protobuf`, `google.golang.org/grpc`, the `hashicorp/go-plugin` runtime the SDK wraps.
---
## Prerequisites (read before starting)
This plan targets the **`silo-plugin-sdk`** repository (Go module `github.com/Silo-Server/silo-plugin-sdk`), which is **not currently checked out** in this workspace — only `silo-server` and `silo-plugin-tmdb` are. Before executing:
1. Clone `silo-plugin-sdk` as a writable working copy beside the other repos. Commands below assume **the silo-plugin-sdk repository root is the cwd**.
2. Ensure the proto toolchain is available: the `make proto` target auto-installs `buf`, `protoc-gen-go@v1.36.11`, and `protoc-gen-go-grpc@v1.6.1` into `$(GO_BIN)`, but it requires the **`protoc`** binary to already be on `PATH` (it errors `protoc is required` otherwise). Install `protoc` if absent.
3. This SDK is the **long pole**: the silo-server host plan and the arr-plugin plan both depend on the version tagged at the end of this plan (Task 6). Do not start those until this is tagged.
**Pattern reference (current `scheduled_task.v1`, the template this mirrors):**
- Proto: `proto/silo/plugin/v1/scheduled_task.proto`
- Generated: `pkg/pluginproto/silo/plugin/v1/scheduled_task.pb.go` + `scheduled_task_grpc.pb.go`
- Capability const + allowlist: `pkg/pluginsdk/capability/capability.go`
- Runtime wiring: `pkg/pluginsdk/runtime/runtime.go` (struct field, `Client` accessor, `GRPCServer` registration)
- Manifest acceptance test pattern: `pkg/pluginsdk/manifest/manifest_test.go`
---
## File Structure
| File | Responsibility | Action |
|---|---|---|
| `proto/silo/plugin/v1/scan_source.proto` | The `ScanSource` service + `PollChanges` request/response messages | Create |
| `pkg/pluginproto/silo/plugin/v1/scan_source.pb.go` | Generated message types | Create (codegen) |
| `pkg/pluginproto/silo/plugin/v1/scan_source_grpc.pb.go` | Generated client/server stubs | Create (codegen) |
| `pkg/pluginsdk/capability/capability.go` | `ScanSource` const + add to `KnownTypes` allowlist | Modify |
| `pkg/pluginsdk/capability/capability_test.go` | Assert `ScanSource` is a known type | Create or Modify |
| `pkg/pluginsdk/runtime/runtime.go` | `CapabilityServers.ScanSource` field, `Client.ScanSource()` accessor, `GRPCServer` registration | Modify |
| `pkg/pluginsdk/runtime/runtime_test.go` | Assert a `ScanSource` server registers without error | Create or Modify |
| `pkg/pluginsdk/manifest/manifest_test.go` | Assert a manifest declaring `scan_source.v1` loads | Modify |
---
## Task 1: Define and generate the `scan_source.v1` proto
**Files:**
- Create: `proto/silo/plugin/v1/scan_source.proto`
- Create (via codegen): `pkg/pluginproto/silo/plugin/v1/scan_source.pb.go`, `pkg/pluginproto/silo/plugin/v1/scan_source_grpc.pb.go`
- [ ] **Step 1: Write the proto**
Create `proto/silo/plugin/v1/scan_source.proto` (the `go_package` option must match the sibling protos exactly so generated code lands in package `pluginv1`):
```proto
syntax = "proto3";
package silo.plugin.v1;
option go_package = "github.com/Silo-Server/silo-plugin-sdk/pkg/pluginproto/silo/plugin/v1;pluginv1";
// ScanSource lets the host pull changed filesystem paths from a provider
// (e.g. Sonarr/Radarr, inotify, Ceph). Pull only: the host owns the timer and
// calls PollChanges; the plugin never initiates.
service ScanSource {
rpc PollChanges(PollChangesRequest) returns (PollChangesResponse);
}
message PollChangesRequest {
// Which configured scan_source capability instance is being polled.
string capability_id = 1;
// Opaque continuation token from the previous PollChanges. Empty on first
// run, which the provider treats as "start from now" (do not replay history).
string marker = 2;
}
message PollChangesResponse {
// Absolute paths already translated into Silo's filesystem namespace (the
// plugin has applied its own path rewrites). Files or directories.
repeated string changed_paths = 1;
// Opaque continuation token. The host stores it verbatim and echoes it back
// on the next PollChanges; the host never parses it.
string next_marker = 2;
}
```
- [ ] **Step 2: Generate the Go code**
Run: `make proto`
Expected: exits 0; creates `pkg/pluginproto/silo/plugin/v1/scan_source.pb.go` and `scan_source_grpc.pb.go`. (If it prints `protoc is required`, install `protoc` and re-run.)
- [ ] **Step 3: Verify the generated symbols exist and compile**
Run: `go build ./... && grep -l "func NewScanSourceClient" pkg/pluginproto/silo/plugin/v1/scan_source_grpc.pb.go && grep -l "RegisterScanSourceServer" pkg/pluginproto/silo/plugin/v1/scan_source_grpc.pb.go`
Expected: build succeeds; both `grep`s print the filename (confirming `ScanSourceClient`, `ScanSourceServer`, `NewScanSourceClient`, `RegisterScanSourceServer` were generated — `require_unimplemented_servers=false` in `buf.gen.yaml` means no forced `UnimplementedScanSourceServer` embedding).
- [ ] **Step 4: Commit**
```bash
git add proto/silo/plugin/v1/scan_source.proto pkg/pluginproto/silo/plugin/v1/scan_source.pb.go pkg/pluginproto/silo/plugin/v1/scan_source_grpc.pb.go
git commit -m "feat(proto): add scan_source.v1 capability service"
```
---
## Task 2: Register the capability type in the allowlist
The manifest loader rejects capability types not in `capability.KnownTypes` (see `TestLoadRejectsUnknownCapabilityType`). A manifest declaring `scan_source.v1` cannot load until the type is registered here.
**Files:**
- Modify: `pkg/pluginsdk/capability/capability.go`
- Create or Modify: `pkg/pluginsdk/capability/capability_test.go`
- [ ] **Step 1: Write the failing test**
Append to `pkg/pluginsdk/capability/capability_test.go` (create the file with this `package capability` header if it does not exist):
```go
package capability
import "testing"
func TestScanSourceIsKnownType(t *testing.T) {
if ScanSource != "scan_source.v1" {
t.Fatalf("ScanSource const = %q, want %q", ScanSource, "scan_source.v1")
}
found := false
for _, k := range KnownTypes {
if k == ScanSource {
found = true
break
}
}
if !found {
t.Fatalf("ScanSource (%q) missing from KnownTypes %v", ScanSource, KnownTypes)
}
}
```
- [ ] **Step 2: Run test to verify it fails**
Run: `go test ./pkg/pluginsdk/capability/ -run TestScanSourceIsKnownType -v`
Expected: FAIL — `undefined: ScanSource` (the const does not exist yet).
- [ ] **Step 3: Add the const and allowlist entry**
In `pkg/pluginsdk/capability/capability.go`, add the const inside the existing `const (...)` block (after `EbookBackend`):
```go
EbookBackend = "ebook_backend.v1"
ScanSource = "scan_source.v1"
```
and add it to the `KnownTypes` slice (append `ScanSource` to the existing list):
```go
ScanSource,
```
- [ ] **Step 4: Run test to verify it passes**
Run: `go test ./pkg/pluginsdk/capability/ -run TestScanSourceIsKnownType -v`
Expected: PASS.
- [ ] **Step 5: Commit**
```bash
git add pkg/pluginsdk/capability/capability.go pkg/pluginsdk/capability/capability_test.go
git commit -m "feat(capability): register scan_source.v1 as a known capability type"
```
---
## Task 3: Wire the capability into the runtime
Add the `ScanSource` server to `CapabilityServers`, a `Client.ScanSource()` accessor, and the conditional registration in `GRPCServer` — mirroring `ScheduledTask` in `pkg/pluginsdk/runtime/runtime.go`.
**Files:**
- Modify: `pkg/pluginsdk/runtime/runtime.go`
- Create or Modify: `pkg/pluginsdk/runtime/runtime_test.go`
- [ ] **Step 1: Write the failing test**
Append to `pkg/pluginsdk/runtime/runtime_test.go` (create with the `package runtime` header if absent). This test constructs a real `*grpc.Server`, supplies a minimal `ScanSource` server, and asserts `GRPCServer` registers it without error:
```go
package runtime
import (
"context"
"testing"
pluginv1 "github.com/Silo-Server/silo-plugin-sdk/pkg/pluginproto/silo/plugin/v1"
"google.golang.org/grpc"
)
type stubScanSource struct{}
func (stubScanSource) PollChanges(context.Context, *pluginv1.PollChangesRequest) (*pluginv1.PollChangesResponse, error) {
return &pluginv1.PollChangesResponse{}, nil
}
func TestGRPCServerRegistersScanSource(t *testing.T) {
p := &GRPCPlugin{Servers: CapabilityServers{
Runtime: stubRuntime{}, // existing stub used by sibling runtime tests
ScanSource: stubScanSource{},
}}
srv := grpc.NewServer()
if err := p.GRPCServer(nil, srv); err != nil {
t.Fatalf("GRPCServer with ScanSource = %v, want nil", err)
}
if _, ok := srv.GetServiceInfo()["silo.plugin.v1.ScanSource"]; !ok {
t.Fatalf("ScanSource service not registered; got %v", srv.GetServiceInfo())
}
}
```
Note: `GRPCServer` requires `Servers.Runtime` to be non-nil (it errors otherwise). Reuse the existing runtime stub the sibling tests use. If no such stub exists in `runtime_test.go`, add a minimal one:
```go
type stubRuntime struct{ pluginv1.RuntimeServer }
```
- [ ] **Step 2: Run test to verify it fails**
Run: `go test ./pkg/pluginsdk/runtime/ -run TestGRPCServerRegistersScanSource -v`
Expected: FAIL — `unknown field 'ScanSource' in struct literal of type CapabilityServers` (the field does not exist yet).
- [ ] **Step 3: Add the struct field**
In `pkg/pluginsdk/runtime/runtime.go`, add to the `CapabilityServers` struct (after the `ScheduledTask` field):
```go
ScheduledTask pluginv1.ScheduledTaskServer
ScanSource pluginv1.ScanSourceServer
```
- [ ] **Step 4: Add the `Client` accessor**
After the existing `func (c *Client) ScheduledTask() ...` method, add:
```go
func (c *Client) ScanSource() pluginv1.ScanSourceClient {
return pluginv1.NewScanSourceClient(c.conn)
}
```
- [ ] **Step 5: Add the conditional registration in `GRPCServer`**
In `func (p *GRPCPlugin) GRPCServer(...)`, after the `ScheduledTask` registration block, add:
```go
if p.Servers.ScheduledTask != nil {
pluginv1.RegisterScheduledTaskServer(server, p.Servers.ScheduledTask)
}
if p.Servers.ScanSource != nil {
pluginv1.RegisterScanSourceServer(server, p.Servers.ScanSource)
}
```
- [ ] **Step 6: Run test to verify it passes**
Run: `go test ./pkg/pluginsdk/runtime/ -run TestGRPCServerRegistersScanSource -v`
Expected: PASS.
- [ ] **Step 7: Commit**
```bash
git add pkg/pluginsdk/runtime/runtime.go pkg/pluginsdk/runtime/runtime_test.go
git commit -m "feat(runtime): serve and dial scan_source.v1 capability"
```
---
## Task 4: Prove a `scan_source.v1` manifest loads
Guards the end-to-end path: a plugin manifest declaring the new capability must pass the loader (it would not before Task 2's allowlist entry).
**Files:**
- Modify: `pkg/pluginsdk/manifest/manifest_test.go`
- [ ] **Step 1: Write the failing test**
The test file is package `manifest_test` (external) and loads from a JSON literal via `manifest.Load([]byte)`. Append this test, mirroring `TestLoadAcceptsRequestRouterCapability` exactly with the type swapped to `scan_source.v1`:
```go
func TestLoadAcceptsScanSourceCapability(t *testing.T) {
raw := []byte(`{
"plugin_id": "silo.example",
"version": "1.0.0",
"silo_api_version": "v1",
"capabilities": [
{"type": "scan_source.v1", "id": "arr", "display_name": "X", "description": "Y"}
]
}`)
m, err := manifest.Load(raw)
if err != nil {
t.Fatalf("Load returned unexpected error: %v", err)
}
if len(m.GetCapabilities()) != 1 {
t.Fatalf("expected 1 capability, got %d", len(m.GetCapabilities()))
}
if got := m.GetCapabilities()[0].GetType(); got != "scan_source.v1" {
t.Fatalf("capability type = %q, want scan_source.v1", got)
}
}
```
- [ ] **Step 2: Run test to verify it fails (before this task's dependency was in place) / passes now**
Run: `go test ./pkg/pluginsdk/manifest/ -run TestLoadAcceptsScanSourceCapability -v`
Expected: PASS (Task 2 already added `scan_source.v1` to `KnownTypes`). If it FAILS with an unknown-capability-type error, Task 2 was not completed correctly — fix that first.
- [ ] **Step 3: Commit**
```bash
git add pkg/pluginsdk/manifest/manifest_test.go
git commit -m "test(manifest): accept scan_source.v1 capability in manifests"
```
---
## Task 5: Full build + test sweep
**Files:** none (verification only)
- [ ] **Step 1: Build everything**
Run: `go build ./...`
Expected: exits 0.
- [ ] **Step 2: Run the full SDK test suite**
Run: `go test ./...`
Expected: all packages `ok`. Confirms the additive change broke no existing capability, manifest, or runtime test.
- [ ] **Step 3: Vet + format**
Run: `go vet ./... && gofmt -l pkg proto 2>/dev/null; test -z "$(gofmt -l pkg)"`
Expected: vet clean; `gofmt -l pkg` prints nothing.
---
## Task 6: Tag the release
Downstream repos (silo-server host, arr plugin) consume the SDK as a tagged module version.
**Files:** none (release action)
- [ ] **Step 1: Confirm the version to tag**
Check the latest tag: `git tag --list 'v*' | sort -V | tail -3`. The current consumed version is `v0.4.0`; this additive feature warrants a minor bump to **`v0.5.0`** (no breaking changes).
- [ ] **Step 2: Tag and push**
```bash
git tag v0.5.0
git push origin v0.5.0
```
- [ ] **Step 3: Record the contract for downstream plans**
Note in the host and arr-plugin plans that they must require `github.com/Silo-Server/silo-plugin-sdk v0.5.0` and use:
- `capability.ScanSource` (`"scan_source.v1"`)
- `runtime.CapabilityServers{ ScanSource: <impl> }` (plugin side)
- `client.ScanSource()` → `pluginv1.ScanSourceClient.PollChanges(ctx, &pluginv1.PollChangesRequest{CapabilityId, Marker})` (host side)
- response fields `PollChangesResponse.ChangedPaths []string`, `PollChangesResponse.NextMarker string`
---
## Self-Review notes
- **Spec coverage:** Implements spec §5 (the `scan_source.v1` contract: `PollChanges`, opaque marker, Silo-native paths) and §11 (additive-only; registered in `KnownTypes`; no existing capability modified). Spec §12 build-order step 1 (tag the SDK) is Task 6.
- **Out of scope here:** the host engine (spec §7), connections (§8), the Autoscan category (§9), and the arr plugin (§10) are separate per-repo plans built against the `v0.5.0` tag produced by Task 6.
- **Marker is opaque end-to-end:** the proto carries `marker`/`next_marker` as plain strings with no server-side interpretation, satisfying the "any provider's bookmark" requirement.
docs/superpowers/plans/2026-06-03-autoscan-arr-plugin.md
+265
View File
@@ -0,0 +1,265 @@
# Sonarr/Radarr Scan-Source Plugin Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Build a new installable Silo plugin that implements `scan_source.v1` for Sonarr/Radarr: when the host pulls, it polls arr `/history`, extracts imported + renamed paths, applies its own path rewrites, and returns Silo-native paths + a marker.
**Architecture:** A standalone Go module structured like `silo-plugin-tmdb` — a `manifest.json` declaring `scan_source.v1`, a `main.go` serving `runtime.CapabilityServers{Runtime, ScanSource}` via `runtime.Serve`, and the arr-specific logic ported from the closed silo-server PR #43 (`internal/autoscan/history.go`, `rewrite.go`, `suggest.go`).
**Tech Stack:** Go, `github.com/Silo-Server/silo-plugin-sdk` (`v0.5.0`, the `scan_source.v1` capability), `hashicorp/go-plugin` runtime (via the SDK), arr HTTP API (`X-Api-Key`).
Commands assume the **new plugin repository root** is the cwd (the `silo-plugin-autoscan-arr` checkout, a sibling of this repo like the `silo-plugin-sdk` checkout).
---
## Prerequisites
1. **SDK `v0.5.0`.** Needs `scan_source.v1` from `silo-plugin-sdk` PR #2. Until tagged, depend on the local checkout via replace:
```bash
go mod edit -replace github.com/Silo-Server/silo-plugin-sdk=../silo-plugin-sdk
```
This points at the `silo-plugin-sdk` checkout (a sibling of this repo). Task 8 finalizes to `v0.5.0`.
2. **Template:** the `silo-plugin-tmdb` checkout (a sibling of this repo) — copy its `main.go` runtime scaffold, `Makefile`, and `manifest.json` shape.
3. **Salvage source:** the arr logic is on silo-server `main` at `internal/autoscan/history.go` (imports + renames extraction, bounded window), `rewrite.go` (`applyRewrites`, `normalizeSeparators`), `suggest.go`/`suggest_deps.go` (root-folder suffix-match suggester). Port these verbatim where possible.
4. **New repo:** create `Silo-Server/silo-plugin-autoscan-arr` (or fork pattern). The plugin is registered in the `silo-plugins` catalog in a separate follow-up (Task 9 notes it).
---
## File Structure
| File | Responsibility | Action |
|---|---|---|
| `go.mod` | module `github.com/Silo-Server/silo-plugin-autoscan-arr`; SDK dep | Create |
| `manifest.json` | declares `scan_source.v1` capability + `global_config_schema` (path rewrites only; connection arrives per request) | Create |
| `main.go` | `runtimeServer` (GetManifest/Configure) + `scanSourceServer` (PollChanges); `runtime.Serve` | Create |
| `internal/arr/history.go` | poll `/history`, imports + renames, bounded window | Create (port) |
| `internal/arr/rewrite.go` | `applyRewrites`, `normalizeSeparators` | Create (port) |
| `internal/arr/client.go` | arr HTTP client (`X-Api-Key`, response cap, timeout) | Create |
| `internal/config/config.go` | parse Configure payload → `{rewrites}` (connection comes per request, not config) | Create |
| `Makefile` | build (cross-platform), like tmdb | Create |
---
## Task 1: Scaffold the plugin (serves an empty `scan_source.v1`)
**Files:** Create `go.mod`, `manifest.json`, `main.go`, `Makefile`.
- [ ] **Step 1: Module + SDK dep**
```bash
go mod init github.com/Silo-Server/silo-plugin-autoscan-arr
go mod edit -replace github.com/Silo-Server/silo-plugin-sdk=../silo-plugin-sdk # the silo-plugin-sdk checkout, a sibling of this repo
go get github.com/Silo-Server/silo-plugin-sdk@v0.4.0 # replace points at the scan_source branch checkout
```
- [ ] **Step 2: manifest.json**
Mirror tmdb's shape, declaring the scan source:
```json
{
"plugin_id": "silo.autoscan.arr",
"version": "0.1.0",
"checksum": "__CHECKSUM__",
"silo_api_version": "v1",
"supported_platforms": [
{"os": "linux", "arch": "amd64"},
{"os": "linux", "arch": "arm64"},
{"os": "darwin", "arch": "arm64"}
],
"capabilities": [
{
"type": "scan_source.v1",
"id": "arr",
"display_name": "Sonarr / Radarr",
"description": "Triggers Silo rescans from Sonarr/Radarr import and rename history."
}
],
"global_config_schema": [
{"key": "path_rewrites", "type": "json", "required": false, "display_name": "Path rewrites"}
]
}
```
> **Connection is NOT plugin config.** The arr base URL + API key are delivered by the host in each `PollChangesRequest.connection` (resolved from the operator's Autoscan connection — own creds or a reused Requests link). The plugin reads them from the request and stores no credentials. Only path rewrites (and other provider-specific behaviour) live in plugin config.
(Confirm the exact `global_config_schema` entry shape against `silo-plugin-sdk` `pkg/pluginsdk/config` — adapt field names to the SDK's config schema type.)
- [ ] **Step 3: main.go scaffold**
Copy tmdb's `runtimeServer` (GetManifest from embedded manifest; Configure stores config) and add a `scanSourceServer` returning empty for now:
```go
type scanSourceServer struct{ cfg *config.Config }
func (s *scanSourceServer) PollChanges(ctx context.Context, req *pluginv1.PollChangesRequest) (*pluginv1.PollChangesResponse, error) {
return &pluginv1.PollChangesResponse{}, nil
}
func main() {
manifest := mustLoadManifest()
rt := &runtimeServer{manifest: manifest}
runtime.Serve(runtime.ServeConfig{
Servers: runtime.CapabilityServers{
Runtime: rt,
ScanSource: &scanSourceServer{cfg: rt.cfg},
},
})
}
```
- [ ] **Step 4: Build**
Run: `go build ./...`
Expected: exits 0.
- [ ] **Step 5: Commit**
```bash
git init && git add -A
git commit -m "feat: scaffold autoscan-arr scan_source.v1 plugin"
```
---
## Task 2: Config parsing (Configure → path rewrites only)
**Files:** Create `internal/config/config.go`, `internal/config/config_test.go`.
> Connection (base_url/api_key) is NOT config — it arrives per poll in `PollChangesRequest.connection` (Task 5). Plugin config carries only path rewrites.
- [ ] **Step 1: Failing test**
```go
func TestParseConfig(t *testing.T) {
cfg, err := Parse(map[string]string{
"path_rewrites": `[{"from":"/mnt/arr/tv","to":"/mnt/media/tv"}]`,
})
if err != nil { t.Fatalf("Parse: %v", err) }
if len(cfg.Rewrites) != 1 || cfg.Rewrites[0].From != "/mnt/arr/tv" { t.Fatalf("bad rewrites: %+v", cfg.Rewrites) }
}
```
- [ ] **Step 2: Run → fail.** `go test ./internal/config/` → undefined `Parse`.
- [ ] **Step 3: Implement** `Parse(map[string]string) (*Config, error)` with `Config{Rewrites []Rewrite}` where `Rewrite{From, To string}`; unmarshal `path_rewrites` JSON. Wire `runtimeServer.Configure` to store the parsed config so `scanSourceServer` reads the rewrites.
- [ ] **Step 4: Run → pass.**
- [ ] **Step 5: Commit** `feat(config): parse arr connection and path rewrites`.
---
## Task 3: arr history client (imports + renames, bounded window)
**Files:** Create `internal/arr/client.go`, `internal/arr/history.go`, `internal/arr/history_test.go`.
Port from silo-server `main:internal/autoscan/history.go` (the `ChangedPaths` logic: `downloadFolderImported.importedPath`; `episodeFileRenamed`/`movieFileRenamed` → `path` + `sourcePath`; deletes ignored) and the client (`X-Api-Key`, 1 MiB cap, timeout). The marker is an RFC3339 timestamp; empty marker ⇒ "now".
- [ ] **Step 1: Failing test** — copy the salvaged `history_test.go` (`TestArrHistoryChangedPaths`: imports + both rename paths returned, `grabbed`/`episodeFileDeleted` ignored), renamed to this package, asserting the returned `(paths, nextMarker)`.
- [ ] **Step 2: Run → fail.**
- [ ] **Step 3: Implement** `ChangedPaths(ctx, baseURL, apiKey, since time.Time) (paths []string, newest time.Time, err error)` — port `history.go`, additionally returning the newest history timestamp as the next marker. Apply the bounded window (24h max-lookback floor + overlap buffer) here.
- [ ] **Step 4: Run → pass.**
- [ ] **Step 5: Commit** `feat(arr): poll history for imports and renames`.
---
## Task 4: Path rewrites (→ Silo-native paths)
**Files:** Create `internal/arr/rewrite.go`, `internal/arr/rewrite_test.go`.
Port `applyRewrites` (boundary-safe prefix: `path==trimmed || HasPrefix(path, trimmed+"/")`) and `normalizeSeparators` from silo-server `main:internal/autoscan/rewrite.go`.
- [ ] **Step 1: Failing test** — copy salvaged `rewrite_test.go` cases.
- [ ] **Step 2: Run → fail.**
- [ ] **Step 3: Implement** the port.
- [ ] **Step 4: Run → pass.**
- [ ] **Step 5: Commit** `feat(arr): boundary-safe path rewrites`.
---
## Task 5: Wire `PollChanges` end-to-end
**Files:** Modify `main.go` (`scanSourceServer.PollChanges`), add `main_test.go`.
- [ ] **Step 1: Failing test** — a `scanSourceServer` (rewrites configured) called with a `PollChangesRequest` whose `Connection` points at a stub arr server (httptest); assert `PollChanges` with empty marker returns rewritten Silo-native paths and a non-empty `next_marker`, and that the request actually hit the stub at `connection.base_url` with `X-Api-Key: connection.api_key`. A second call with the returned marker queries arr with the right `since`. Also assert a request with a nil/empty `Connection` returns an error (no credentials to poll with).
- [ ] **Step 2: Run → fail.**
- [ ] **Step 3: Implement** — read the connection from the request, not from config:
```go
func (s *scanSourceServer) PollChanges(ctx context.Context, req *pluginv1.PollChangesRequest) (*pluginv1.PollChangesResponse, error) {
conn := req.GetConnection()
if conn == nil || conn.GetBaseUrl() == "" {
return nil, fmt.Errorf("scan_source: no connection supplied")
}
since := time.Time{} // empty marker => now (history client floors to now)
if m := req.GetMarker(); m != "" {
if t, err := time.Parse(time.RFC3339, m); err == nil { since = t }
}
raw, newest, err := arr.ChangedPaths(ctx, conn.GetBaseUrl(), conn.GetApiKey(), since)
if err != nil { return nil, err }
out := make([]string, 0, len(raw))
for _, p := range raw {
out = append(out, arr.ApplyRewrites(arr.NormalizeSeparators(p), s.cfg.Rewrites))
}
return &pluginv1.PollChangesResponse{ChangedPaths: out, NextMarker: newest.UTC().Format(time.RFC3339)}, nil
}
```
- [ ] **Step 4: Run → pass.**
- [ ] **Step 5: Commit** `feat: PollChanges returns Silo-native paths from arr history`.
---
## Task 6: Rewrite suggester ("Sync from arr") — root folders
**Files:** Create `internal/arr/suggest.go`, `suggest_test.go`.
Port `suggestRewrites` (suffix-match arr root folders → Silo folders) from `main:internal/autoscan/suggest.go`. **Open question (spec §14 risk #2):** how the operator triggers "Sync" and how the plugin reaches Silo's media-folder list.
- [ ] **Step 1:** Decide the delivery mechanism — recommended: the plugin reads Silo's folder list via the host library-listing service (`pluginhost` library lister, reachable through the SDK `runtime.Host()` client). Confirm that path exists in `v0.5.0`; if not, scope "Sync" to a follow-up and ship Tasks 1–5 (manual rewrites only) first.
- [ ] **Step 2:** Port the suffix-match suggester + its tests (unique/ambiguous/covered/no-op/normalization cases from `suggest_test.go`).
- [ ] **Step 3:** Expose it — either via an `http_routes.v1` capability the admin UI calls, or as part of the plugin config flow. Document the chosen mechanism.
- [ ] **Step 4:** Tests pass; commit `feat(arr): suffix-match rewrite suggester`.
> If §14 risk #2 is unresolved at execution time, **skip this task**, ship manual rewrites (Tasks 1–5), and file the suggester as a follow-up. The plugin is fully functional without it.
---
## Task 7: Cross-repo smoke test against the host
**Files:** none (verification).
- [ ] **Step 1:** Build the plugin binary; install it into a dev silo-server (with the Part-1 backend) via the plugin installer.
- [ ] **Step 2:** Configure a source pointing at a real/stub arr; trigger an autoscan poll; confirm the host enqueues a scan for an imported path. (This exercises the full SDK contract end-to-end.)
- [ ] **Step 3:** Note results; no commit unless fixes are needed.
---
## Task 8: Finalize SDK dependency
- [ ] After `silo-plugin-sdk v0.5.0` is tagged: `go mod edit -dropreplace ...; go get .../silo-plugin-sdk@v0.5.0; go mod tidy; go build ./...`.
- [ ] Commit `build: depend on silo-plugin-sdk v0.5.0`.
- [ ] Do not publish/tag the plugin with a `replace` directive in `go.mod`.
---
## Task 9: Catalog registration (follow-up, separate repo)
- [ ] Add `silo.autoscan.arr` to the `silo-plugins` catalog manifest so it is installable. This is a separate PR in the `silo-plugins` repo (not this module) — note it; out of scope for this plan's repo.
---
## Self-Review notes
- **Spec coverage:** §10 (arr plugin: history imports + renames, bounded window, rewrites, Silo-native paths) Tasks 3–5; §5 contract (`PollChanges`, opaque marker = RFC3339 timestamp, first-run "now") Task 5; "Sync from arr" §10/§14-risk-2 Task 6 (with an explicit skip path).
- **Salvage:** history/rewrite/suggest ported from the closed PR (Tasks 3,4,6) — exactly the files the host backend plan deletes (Part-1 Task 8).
- **Risk:** §14 risk #1 (one plugin installation per arr server vs. many) — this plugin holds one connection per installation/config; multiple arr servers ⇒ multiple installs unless the runtime supports multiple capability instances. §14 risk #2 (Sync UI) gated in Task 6 with a clean skip.
- **Type consistency:** `Config{Rewrites}` (no creds — connection arrives in `PollChangesRequest.connection`), `Rewrite{From,To}`, `arr.ChangedPaths(ctx, baseURL, apiKey, since) ([]string, time.Time, error)`, `arr.ApplyRewrites`/`NormalizeSeparators`, `scanSourceServer.PollChanges` consistent across Tasks 2–6.
docs/superpowers/plans/2026-06-03-autoscan-host-ui.md
+209
View File
@@ -0,0 +1,209 @@
# Autoscan Admin UI Implementation Plan (Part 2 of 2)
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Give Autoscan its own admin category (page + sidebar nav), separate from Requests: manage Connections (reuse a Requests arr server or enter own), Sources (per scan-source plugin: enable, interval, connection), and global settings — backed by the Part-1 v2 API.
**Architecture:** Extract the Autoscan tab out of `AdminRequests.tsx` into a standalone `AdminAutoscan.tsx` page; register a route and a sidebar item; rebuild `useAutoscan.ts` hooks + `types.ts` against the v2 endpoints (`/admin/autoscan/{settings,connections,sources,trigger,status}`).
**Tech Stack:** React + TypeScript, React Query (`@tanstack/react-query`), the project's `ui/` component library (`Tabs`, `Table`, `Card`, etc.), `react-router` routes in `App.tsx`, 2-space indent / double quotes / 100-col (`web/.prettierrc`).
Commands assume the silo-server repository root is the cwd. **Depends on Part 1 (backend)** being implemented (the v2 endpoints must exist).
---
## Prerequisites
1. **Part-1 backend merged/available** — the v2 endpoints and response shapes (connections with `request_integration_id`, sources with `marker`/`last_error`, settings with `default_poll_interval_seconds`).
2. **Reference the existing autoscan tab** at `web/src/pages/AdminRequests.tsx` (the `autoscan` `TabsContent`, the `AutoscanTab`/`AutoscanSourceEditor` components) and hooks at `web/src/hooks/queries/useAutoscan.ts` — these are the starting material; they are reshaped, not discarded.
3. **Run before MR:** `cd web && pnpm run lint && pnpm run format:check`. Note the 20 pre-existing failing frontend tests are unrelated (project memory); do not be alarmed if they appear — only newly-introduced failures matter.
---
## File Structure
| File | Responsibility | Action |
|---|---|---|
| `web/src/api/types.ts` | `AutoscanConnection`, `AutoscanSource`, `AutoscanSettings` (v2 shapes) | Modify |
| `web/src/hooks/queries/useAutoscan.ts` | hooks for settings/connections/sources/trigger/status | Rewrite |
| `web/src/hooks/queries/keys.ts` | query keys for connections/sources | Modify |
| `web/src/pages/AdminAutoscan.tsx` | the new standalone Autoscan page (Connections + Sources + Settings tabs) | Create |
| `web/src/pages/admin/autoscan/ConnectionsPanel.tsx` | list/add/edit/delete connections (own or Requests-linked) | Create |
| `web/src/pages/admin/autoscan/SourcesPanel.tsx` | per-source enable, interval, connection binding, status | Create |
| `web/src/pages/AdminRequests.tsx` | remove the Autoscan tab | Modify |
| `web/src/App.tsx` | add `<Route path="autoscan" element={<AdminAutoscan />} />` | Modify |
| `web/src/components/AdminSidebar.tsx` | add the Autoscan nav item under "Content" | Modify |
---
## Task 1: v2 types + hooks
**Files:** Modify `web/src/api/types.ts`, `web/src/hooks/queries/keys.ts`; rewrite `web/src/hooks/queries/useAutoscan.ts`.
- [ ] **Step 1: Types**
Add to `web/src/api/types.ts`:
```ts
export interface AutoscanSettings {
enabled: boolean;
default_poll_interval_seconds: number;
debounce_seconds: number;
}
export interface AutoscanConnection {
id: string;
name: string;
kind: string;
// present when the connection has its own credentials:
base_url?: string;
// present when linked to a Requests integration (live reuse):
request_integration_id?: string | null;
// NOTE: api_key_ref / resolved keys are never sent by the backend.
}
export interface AutoscanSource {
id: string;
installation_id: number;
capability_id: string;
connection_id: string;
enabled: boolean;
poll_interval_seconds: number | null;
last_run_at: string | null;
last_error: string | null;
}
```
- [ ] **Step 2: Hooks**
Rewrite `useAutoscan.ts` with: `useAutoscanSettings`/`useUpdateAutoscanSettings`, `useAutoscanConnections`/`useCreateAutoscanConnection`/`useUpdateAutoscanConnection`/`useDeleteAutoscanConnection`, `useAutoscanSources`/`useUpdateAutoscanSource`, `useAutoscanStatus`, `useTriggerAutoscan`. Follow the existing hook style (the current `useAutoscan.ts` is the template for query/mutation + invalidation). Add query keys in `keys.ts`.
- [ ] **Step 3: Typecheck**
Run: `cd web && pnpm exec tsc --noEmit`
Expected: no type errors in the autoscan files.
- [ ] **Step 4: Commit**
```bash
git add web/src/api/types.ts web/src/hooks/queries/useAutoscan.ts web/src/hooks/queries/keys.ts
git commit -m "feat(web): autoscan v2 types and query hooks"
```
---
## Task 2: Connections panel
**Files:** Create `web/src/pages/admin/autoscan/ConnectionsPanel.tsx`.
- [ ] **Step 1: Build the panel**
A `Card` + `Table` listing connections (name, kind, and a "Reused from Requests" / "Own" badge derived from `request_integration_id`). An "Add connection" dialog offers two modes:
- **Reuse from Requests** — a select populated from the existing request-integrations hook (filter to arr kinds), storing `request_integration_id`.
- **Enter own** — name + URL + API key fields, posting own credentials.
Edit + delete actions per row. Never render any key material (the backend doesn't send it).
- [ ] **Step 2: Lint/format/typecheck**
Run: `cd web && pnpm run lint && pnpm exec tsc --noEmit`
Expected: clean.
- [ ] **Step 3: Commit** `feat(web): autoscan connections panel (reuse or own)`.
---
## Task 3: Sources panel
**Files:** Create `web/src/pages/admin/autoscan/SourcesPanel.tsx`.
- [ ] **Step 1: Build the panel**
A `Table` of sources, one row per installed `scan_source` capability instance: plugin/capability label, a connection `<select>` (bound to `connection_id`), an enable toggle, a poll-interval input (blank = settings default), and a status column (`last_run_at`, `last_error`). Reuse the spinner/"Syncing…" affordances from the old `AutoscanSourceEditor` where relevant. Wire to `useAutoscanSources`/`useUpdateAutoscanSource`.
- [ ] **Step 2: Lint/format/typecheck** — clean.
- [ ] **Step 3: Commit** `feat(web): autoscan sources panel`.
---
## Task 4: The Autoscan page + global settings + manual trigger
**Files:** Create `web/src/pages/AdminAutoscan.tsx`.
- [ ] **Step 1: Compose the page**
A page with a `Tabs` (or sections): **Sources**, **Connections**, **Settings**. Settings = global enable toggle, default poll interval, debounce. A "Run now" button calling `useTriggerAutoscan` (202 → toast "Autoscan triggered"). Mirror the header/layout of `AdminRequests.tsx` for visual consistency.
- [ ] **Step 2: Lint/format/typecheck** — clean.
- [ ] **Step 3: Commit** `feat(web): standalone Autoscan admin page`.
---
## Task 5: Route + sidebar nav
**Files:** Modify `web/src/App.tsx`, `web/src/components/AdminSidebar.tsx`.
- [ ] **Step 1: Route**
In `web/src/App.tsx`, next to `<Route path="requests" element={<AdminRequests />} />` (l.374), add:
```tsx
<Route path="autoscan" element={<AdminAutoscan />} />
```
with the matching `import AdminAutoscan from "@/pages/AdminAutoscan";`.
- [ ] **Step 2: Sidebar item**
In `web/src/components/AdminSidebar.tsx`, under the **Content** group (after the Requests item, ~l.112), add:
```tsx
{
label: "Autoscan",
icon: <RefreshCw className="h-[18px] w-[18px]" />,
href: "/admin/autoscan",
},
```
(Import an appropriate icon from the icon set already used in the file.)
- [ ] **Step 3: Verify navigation**
Run: `cd web && pnpm run build`
Expected: build succeeds. Manually confirm `/admin/autoscan` renders and the sidebar item highlights.
- [ ] **Step 4: Commit** `feat(web): route and sidebar nav for Autoscan category`.
---
## Task 6: Remove the Autoscan tab from Requests
**Files:** Modify `web/src/pages/AdminRequests.tsx`.
- [ ] **Step 1: Excise the tab**
Remove the `<TabsTrigger value="autoscan">` (l.146) and its `<TabsContent value="autoscan">`, the `AutoscanTab`/`AutoscanSourceEditor` definitions (now superseded by the new panels), and `autoscan` from `ADMIN_REQUEST_TABS`. Remove now-unused imports.
- [ ] **Step 2: Lint/format/typecheck/build**
Run: `cd web && pnpm run lint && pnpm run format:check && pnpm run build`
Expected: clean; build succeeds; no dangling references to removed components.
- [ ] **Step 3: Commit** `refactor(web): move Autoscan out of Requests into its own category`.
---
## Task 7: Final sweep
- [ ] **Step 1:** `cd web && pnpm run lint && pnpm run format:check && pnpm run build` — all clean.
- [ ] **Step 2:** `cd web && pnpm test` — confirm no **newly** failing tests (the 20 pre-existing failures, per project memory, are unrelated; compare against baseline).
- [ ] **Step 3:** Manual pass: add a connection (both modes), bind a source, toggle enable, "Run now", confirm status updates.
---
## Self-Review notes
- **Spec coverage:** §9 (Autoscan as its own category, decoupled from Requests; Connections reuse-or-own; per-source enable/interval/status) — Tasks 1–6. The reuse-from-Requests UI (Task 2) surfaces the soft link from spec §8.
- **Salvage:** the old `AutoscanTab`/`AutoscanSourceEditor` and `useAutoscan.ts` are reshaped (Tasks 1–3), then the Requests tab is removed (Task 6).
- **Out of scope:** plugin-side config UI (the path-rewrite table lives on the plugin's own settings screen per spec §10 — that is the arr-plugin plan / spec §14 risk #2, not this page).
- **Type consistency:** `AutoscanConnection{request_integration_id?}`, `AutoscanSource{connection_id,poll_interval_seconds,last_error}`, `AutoscanSettings{default_poll_interval_seconds}` used consistently across hooks (Task 1) and panels (Tasks 2–4).
docs/superpowers/plans/2026-06-05-autoscan-source-labels.md
+1031
View File
File diff suppressed because it is too large Load Diff
docs/superpowers/specs/2026-06-02-autoscan-arr-polling-design.md
+222
View File
@@ -0,0 +1,222 @@
# Autoscan: Polling Radarr/Sonarr to Trigger Targeted Library Scans
**Status:** Design approved, ready for implementation plan
**Date:** 2026-06-02
**Area:** new `internal/autoscan`, `internal/taskmanager/tasks`, `migrations/`, `internal/api`, `web/src/pages/AdminRequests.tsx`
## Summary
Silo currently learns about new media via scheduled/manual library scans. When
Radarr/Sonarr import a download, the new file only appears in Silo on the next
scan. This adds an **autoscan poller**: a periodic background task that polls the
configured Radarr/Sonarr instances for newly-imported files and enqueues
**targeted folder scans** into Silo's existing scan queue, so requested (and
non-requested) content shows up promptly.
The design follows the established autoscan pattern — poll arr import history,
translate the imported paths, and trigger targeted scans — but kept deliberately
simple. That pattern is normally built to orchestrate scans across many separate
media-server boxes, which needs a bounded fan-out guard and a retry queue. Silo
is a single service scanning *itself*, so the cross-node fan-out machinery is
unnecessary — imported paths become folder jobs in Silo's own worker-bounded
`scanqueue`.
## Goals
- Poll autoscan-enabled Radarr/Sonarr instances on an interval for import events.
- Translate arr import paths into Silo media folders and enqueue targeted scans.
- Reuse the Radarr/Sonarr instances already configured for requests
(`request_integrations`) — no duplicate URLs/keys.
- Coalesce bursts (a season's episodes) into minimal redundant scans.
- Admin-configurable: global enable, interval, debounce, per-instance toggle,
optional path rewrites, on-demand "poll now".
## Non-Goals
- No cross-node scan fan-out or bounded-concurrency semaphore (single service;
the `scanqueue` workers are the bound).
- No separate retry queue (a failed source poll simply does not advance its
high-water mark, so the next cycle retries the window).
- No arr webhook/"Connect" push integration (polling only, as requested; webhook
push is a possible future add-on).
- No separate autoscan source credential store — sources are existing
`request_integrations`.
## Background
- **`internal/scantrigger`** already maps a filesystem path to a Silo
`MediaFolder` and scan path: `Resolver.Resolve(Request{Path}) -> Target{Folder,
Mode, Path}` (via `LongestMatchingRoot` / `MatchFolderForPath`).
- **`internal/scanqueue`** executes scans through a worker-bounded queue:
`Queuer.EnqueueScan(ctx, folderID, mode, path, trigger)` /
`EnqueueScans(targets)`.
- **`internal/requests/arrclient`** is a thin Radarr/Sonarr HTTP client
(`New(baseURL, apiKey, httpClient)`, `GetJSON`, `DoJSON`) — reused for the new
history call.
- **`request_integrations`** holds each instance's `id`, `kind`, `base_url`, and
Fernet-encrypted `api_key_ref`, resolved via the same secret resolver the
request service uses.
- The **task manager** runs periodic tasks via the `taskmanager.Task` interface
with interval `DefaultTriggers()` (e.g. `ReconcileRequestsTask`).
## Design
### 1. Data model & config
A new `internal/autoscan` package owns its own schema; `request_integrations`
stays focused on fulfillment.
**`autoscan_settings`** (singleton, mirrors `request_settings`):
| column | type / default | purpose |
|---|---|---|
| `id` | bool PK default true, singleton CHECK | |
| `enabled` | bool, default false | master on/off |
| `poll_interval_minutes` | int, default 10 | poll task interval |
| `debounce_seconds` | int, default 60 | folder re-scan suppression window |
| `created_at`, `updated_at` | timestamptz | |
**`autoscan_sources`** (one row per autoscan-enabled instance):
| column | type | purpose |
|---|---|---|
| `integration_id` | text PK → `request_integrations(id)` ON DELETE CASCADE | the source instance |
| `enabled` | bool, default false | per-instance autoscan toggle |
| `path_rewrites` | jsonb, default `[]` | `[{from, to}]` prefix overrides |
| `last_poll_at` | timestamptz, null | `history/since` high-water mark |
| `created_at`, `updated_at` | timestamptz | |
**Reused, not duplicated:** `base_url` / `api_key_ref` come from
`request_integrations`. Debounce/suppression state lives in **Redis**
(`autoscan:scanned:{folder_id}`, TTL = `debounce_seconds`), not Postgres.
### 2. Poll cycle
`autoscan.Service.PollOnce(ctx)` (invoked by the periodic task):
1. Load `autoscan_settings`; if `!enabled`, return immediately (no-op).
2. Load `autoscan_sources WHERE enabled`, joined to `request_integrations` for
`kind` / `base_url` / `api_key_ref`.
3. For each source (sequential — a handful of instances):
1. Decrypt the API key via the secret resolver.
2. Record `cycleStart := now`.
3. `arrclient.GetJSON("/api/v3/history/since?date={last_poll_at}&eventType=downloadFolderImported")`
— both Radarr and Sonarr expose this. `last_poll_at` null → a sane floor
(do not scan all history on first enable; floor at `cycleStart`).
4. Extract the imported **file path** per event (Radarr: movie file path;
Sonarr: episode file `outputPath`); take its **parent directory**.
5. On success, advance that source's `last_poll_at = cycleStart`. On failure
(arr unreachable, decode error), log and **do not** advance — next cycle
retries the window.
4. For each extracted directory: apply path rewrites (§3), resolve via
`scantrigger.Resolver` → `Target{Folder, Mode, Path}`. Unresolvable paths
(outside Silo's media folders) are logged and skipped.
5. **Dedupe to unique folders** for this cycle; for each, check the Redis
suppression key — if absent, `EnqueueScan(..., trigger="autoscan")` and set
the key with TTL `debounce_seconds`; if present, skip (already scanned this
window).
**Debounce rationale:** because the poller batches a whole interval of imports
per `history/since` call, within-cycle folder dedup does most of the work. The
Redis suppression key only prevents re-enqueuing the same folder on back-to-back
cycles. Silo's scanner is idempotent, so a season importing across two cycles
simply gets a second (cheap) scan that picks up the rest. This is simpler than a
"collect-expired" debounce plus a retry queue, which the polling + idempotent
re-scan model makes unnecessary.
### 3. Path resolution & scan enqueue
A small pipeline in front of the existing scan plumbing — no new scan logic:
1. **Rewrite** (per source, optional): apply the first matching `path_rewrites`
entry as a prefix replacement (e.g. `/data/media -> /mnt/media`). No rewrites
→ path used as-is (shared-mount common case).
2. **Resolve** via `scantrigger.Resolver.Resolve(Request{Path: rewrittenDir})`
→ `Target{Folder, Mode, Path}`. No matching media folder → log + skip.
3. **Enqueue** deduped targets via `scanqueue` `EnqueueScans` with
`trigger="autoscan"` (attributable in scan history, flows through the same
worker-bounded queue).
### 4. Scheduling & concurrency
- **`internal/taskmanager/tasks/autoscan_poll.go`** implements `taskmanager.Task`
(`Key: "autoscan_poll"`, name/description/category, `DefaultTriggers` = interval
trigger seeded from `poll_interval_minutes`). Its run body calls
`autoscan.Service.PollOnce(ctx)`.
- **Disabled = cheap no-op:** the task stays registered; `PollOnce` returns early
when `enabled` is false. Toggling autoscan never (de)registers tasks.
- **Interval changes:** updating `poll_interval_minutes` reconfigures the task's
interval trigger via the task manager's existing trigger-config path; if a live
update is unavailable it applies on next start.
- **No overlap, no semaphore:** the task manager serializes a task's executions;
sources are polled sequentially; scan execution is bounded by `scanqueue`. The
single-service simplification: no bounded-fan-out semaphore is needed.
- **Primary/integrated server only** (same as the request reconcile task), not on
standalone transcode/proxy worker nodes.
### 5. Admin API & UI
**API** (admin-gated, `/api/v1/admin/autoscan`, mirroring the request-settings
handlers; never echoes API keys):
- `GET/PUT /autoscan/settings` — `{enabled, poll_interval_minutes, debounce_seconds}`
- `GET /autoscan/sources` — per autoscan-eligible instance (joined from
`request_integrations`): `{integration_id, name, kind, enabled, path_rewrites,
last_poll_at}`
- `PUT /autoscan/sources/{integration_id}` — set `{enabled, path_rewrites}`
- `POST /autoscan/trigger` — run `PollOnce` once on demand; tight per-admin rate
limit so it cannot hammer the arr instances
- `GET /autoscan/status` — `{enabled, per-source last_poll_at, last scan summary}`
**UI:** a new **"Autoscan" tab** in `web/src/pages/AdminRequests.tsx` (alongside
Queue / Settings / Integrations / User Overrides):
- Global controls: enable switch, poll interval, debounce window.
- Per-source list — one row per Radarr/Sonarr instance with an autoscan toggle,
an optional `from → to` path-rewrite editor, and a read-only "last polled".
- A **"Poll now"** button → `POST /autoscan/trigger`.
Rate limits and admin-auth follow Silo's existing request-admin conventions.
### 6. Testing
- **Path rewrite:** prefix replacement, first-match ordering, no-match
passthrough, trailing-slash handling.
- **arr history client:** table-driven `httptest` fixtures for both Radarr and
Sonarr import-event JSON shapes; asserts the correct file path is extracted and
its parent folder taken.
- **Dedupe + suppression:** many paths in one folder → one target; folder with a
live suppression key skipped; absent/expired key enqueues.
- **`PollOnce` (integration seam):** fakes for source store, arr history client,
`scantrigger.Resolver`, `scanqueue.Queuer`, Redis — asserts deduped folder
scans enqueued with `trigger="autoscan"`, `last_poll_at` advanced on success,
**not** advanced on per-source failure, clean no-op when disabled.
- **Unresolvable path:** logged + skipped, never enqueued or fatal.
- **Repository / migration:** `autoscan_settings` singleton + `autoscan_sources`
CRUD and the `request_integrations` join; migration applies + rolls back
(sidecar table, `ON DELETE CASCADE`).
- **Frontend:** Autoscan tab renders, per-source toggle / path-rewrite editor
round-trips, "Poll now" calls the trigger endpoint.
## Risks & open considerations
- **First-enable floor:** `last_poll_at` null must floor at "now" (not epoch) so
enabling autoscan doesn't enqueue a scan for every historical import.
- **history/since shape differences:** Radarr vs Sonarr import-event payloads
differ; the path extraction is per-kind and covered by fixtures.
- **Path rewrites vs scantrigger roots:** if neither rewrites nor Silo's media
folders match the arr path, the import is silently skipped (logged) — this is
the expected "content outside Silo's libraries" behavior, but a misconfigured
rewrite will look like "autoscan does nothing"; the `last_poll_at` + status
endpoint and skip logs are the debugging surface.
- **Idempotent re-scan:** relies on Silo's scanner being safe to run repeatedly
on a folder (it is — re-scan finds new files).
## References
- Existing Silo plumbing reused by this design: `internal/scantrigger/scantrigger.go`,
`internal/scanqueue/`, `internal/requests/arrclient/`,
`internal/taskmanager/tasks/reconcile_requests.go`
- Prior art: the general autoscan pattern (poll arr import history → translate
paths → trigger targeted scans). Silo's variant drops the cross-node fan-out
guard and retry queue, which only a multi-box orchestrator needs.
docs/superpowers/specs/2026-06-02-autoscan-plugin-architecture-design.md
+262
View File
@@ -0,0 +1,262 @@
# Autoscan as a Pluggable Scan-Source Category — Design Spec
**Date:** 2026-06-02
**Status:** Approved design, pre-implementation
**Goal:** Turn autoscan from a Requests-coupled, arr-only feature into a standalone **Autoscan** category whose change-detection providers are out-of-process Silo plugins, with Sonarr/Radarr as the first provider.
**Architecture (one sentence):** Silo's host keeps a thin, provider-agnostic engine that resolves changed paths to library folders and enqueues rescans; each *way of noticing change* (arr, later inotify/ceph) is an installable plugin that Silo pulls on a timer through a new additive `scan_source.v1` capability.
**Tech stack:** Go (silo-server host + plugins), `silo-plugin-sdk` (protobuf capability contract, consumed as a tagged module), existing `internal/plugins` + `internal/pluginhost` runtime, existing `scantrigger`/`scanqueue` scan pipeline, React/TS admin UI.
Commands assume the repository root is the cwd.
---
## 1. Background & motivation
The current autoscan implementation (delivered in the now-closed PR #43, still on local `main`) is in-process Go that:
- polls Sonarr/Radarr `/api/v3/history/since` on a host timer,
- rewrites arr-side paths to Silo paths, dedups to parent dirs,
- resolves each to a `MediaFolder` and enqueues a targeted scan,
- is configured against `request_integrations` rows (hard FK, `ON DELETE CASCADE`), living as a tab inside the **Requests** admin page.
Two structural problems:
1. **It is arr-only and not extensible.** Adding inotify, Ceph notifications, or any other change source means more bespoke host code.
2. **It is subordinate to Requests.** A source cannot exist without a Requests integration row; deleting the row destroys the source.
Silo already ships a full out-of-process plugin runtime (`internal/plugins`, `internal/pluginhost`): manifests, an installer, typed capability descriptors (`metadata_provider.v1`, `request_router.v1`, `media_analyzer.v1`, `scheduled_task.v1`, `event_consumer.v1`, …), a host-driven scheduled-task mechanism, and a rate-limited plugin→host event path. This design uses that runtime to make change-detection pluggable.
## 2. Goals / non-goals
**Goals**
- A new **Autoscan** admin category, independent of Requests.
- A new additive `scan_source.v1` plugin capability — **breaks no existing plugin**.
- A **client-pull** contract: Silo asks each source "what changed since `<marker>`?" on a timer.
- Sonarr/Radarr re-implemented as the first scan-source plugin; the closed PR's arr logic relocates into it.
- A provider-agnostic host engine reusable by every future scan source.
- Optional reuse of Requests' arr connection details, *or* standalone connection entry.
**Non-goals (this design)**
- inotify / Ceph / other providers (the contract must *accommodate* them; we do not build them).
- A push (plugin-initiated) delivery mode. Pull only; push sources buffer internally and answer on pull (see §6).
- Refactoring the Requests subsystem itself.
## 3. Architecture overview
```
┌──────────────────────────── Silo (host) ────────────────────────────┐
│ Autoscan category (new; not under Requests) │
│ • Sources: which scan_source plugin instance, on/off, interval │
│ • Connections: reuse a Requests arr server, or enter own │
│ • Per-source state: opaque marker, last-run status │
│ │ │
│ ▼ │
│ AutoscanStore — generic engine (provider-agnostic) │
│ per tick, per enabled source: │
│ call plugin PollChanges(marker) │
│ → resolve each Silo-native path → MediaFolder │
│ → suppress duplicates → enqueue scan (existing scanner) │
│ → store returned next_marker │
│ │ scan_source.v1 (pull) │
└──────────────────────────────┼───────────────────────────────────────┘
▼
┌──────── Sonarr/Radarr scan-source plugin ────────┐
│ (new installable repo, like silo-plugin-tmdb) │
│ receives resolved arr address + key │
│ polls /history (imports + renames) │
│ returns RAW source paths + next marker │
│ (host applies the source's rewrites) │
└──────────────────────────────────────────────────┘
```
**Five components**
1. **`scan_source.v1`** (in `silo-plugin-sdk`) — the pull contract. §5.
2. **AutoscanStore** (host) — generic resolve→suppress→enqueue→marker engine. §7.
3. **Connection resolution** (host) — reference-or-inline → concrete credentials. §8.
4. **Autoscan category + data model** (host) — sources, connections, decoupled from Requests. §9.
5. **arr scan-source plugin** (new repo) — first provider. §10.
## 4. Responsibility split (the dividing line)
| Concern | Owner | Rationale |
|---|---|---|
| Talk to arr; poll `/history`; imports + renames; bounded window; first-run | **Plugin** | Provider-specific I/O. |
| **Path rewrites** (per-source config + application) | **Host** | _(Revised post-merge: the SDK landed host-owned rewrites.)_ The plugin returns RAW source-namespace paths in `source_paths`; the host applies each source's rewrite rules (longest-`From`-first) before resolving. Keeps every provider dumb and rewrite behavior uniform. Rewrites are edited per-source in the Autoscan UI and stored in `autoscan_sources.path_rewrites`. |
| Resolve Silo-native path → `MediaFolder`; dedupe; **enqueue rescan** | **Host** | Touches Silo internals (scan queue, DB, library map) that plugins cannot and must not reach. |
| Opaque per-source marker; poll interval; enabled; last-run status | **Host** | Scheduling and bookkeeping are the engine's job. |
| Connection selection (reuse-from-Requests vs own) and credential resolution | **Host** | Only the host knows about Requests; the plugin receives resolved creds. |
The plugin's responsibility ends at **"here are the changed paths, in Silo's terms."** Everything past that is the host engine, which is provider-agnostic.
## 5. The `scan_source.v1` contract (centerpiece)
A new, additive capability in `silo-plugin-sdk`. Existing plugins never declare it and are unaffected (§11).
### Protobuf (as implemented in silo-plugin-sdk)
```proto
// scan_source.v1 — host pulls changed paths from a provider on a timer.
service ScanSource {
// Host calls this on each poll tick for a configured scan_source capability.
rpc PollChanges(PollChangesRequest) returns (PollChangesResponse);
}
message PollChangesRequest {
string capability_id = 1; // which configured scan_source instance
string marker = 2; // opaque; empty string on first run
ResolvedConnection connection = 3; // resolved upstream credentials (see §8)
}
// Concrete upstream credentials the host resolves and hands to the plugin each
// poll, so the plugin stays credential-source-agnostic and stores no secrets.
message ResolvedConnection {
string base_url = 1;
string api_key = 2;
}
message PollChangesResponse {
// Absolute changed paths in the provider/source namespace (RAW — NOT yet
// rewritten). Files or directories. The host applies the source's rewrite
// rules and validates before enqueueing.
repeated string source_paths = 1;
// Opaque continuation token. Host stores verbatim and echoes it back
// on the next PollChanges. The plugin never assumes the host parses it.
string next_marker = 2;
}
```
### Contract rules
- **Pull only.** The host owns the timer and calls `PollChanges`. The plugin never initiates.
- **Opaque marker.** The host stores and echoes `next_marker` without interpreting it. arr uses an RFC3339 timestamp; a future provider may use a sequence number or event id — no host change required.
- **First run.** `marker == ""` ⇒ the plugin starts from "now" (do not replay full history).
- **Idempotent, at-least-once.** The host may re-issue the same `marker` if it failed to persist `next_marker`; providers must tolerate returning overlapping paths. The host's suppression layer (§7) absorbs duplicates.
- **Error semantics.** A failed `PollChanges` (e.g. arr unreachable) ⇒ the host keeps the *old* marker and retries next tick; nothing is skipped or advanced.
- **Raw source paths.** `source_paths` are in the provider's namespace, NOT rewritten. The host applies each source's path-rewrite rules (host-owned, per-source) before resolving to a library folder.
- **Credentials per call.** The host resolves the source's connection (§8) and passes the concrete `{base_url, api_key}` in `PollChangesRequest.connection` on every poll. The plugin reads them from the request — it stores no credentials and never knows whether they came from its own config or a reused Requests link. Delivered over the local host↔plugin gRPC channel only.
The capability needs **only** `PollChanges` for v1. A `TestConnection`/validate RPC is explicitly deferred (YAGNI; connection validity surfaces as a failed poll).
## 6. Marker & delivery semantics
- One opaque marker stored per source, host-side.
- Pull-only does **not** exclude push-style providers: an inotify/Ceph plugin maintains its own internal buffer of observed changes and flushes it (and advances its marker) on each `PollChanges`. Cost: up to one poll-interval of latency. This keeps a single uniform contract.
## 7. Host: AutoscanStore engine
Provider-agnostic. Per enabled source, per tick:
1. `paths, next := plugin.PollChanges(marker)`; on error, log + keep marker + continue.
2. For each `dir` in `uniqueParentDirs(paths)`:
- `target := scantrigger.Resolve(dir)`. Unresolvable (outside Silo's media folders) ⇒ quiet skip.
- Suppression key `fmt.Sprintf("%d|%s", folderID, target.Path)` with debounce TTL; already-claimed ⇒ skip.
- Accumulate `target`; record claimed key.
3. `scanqueue.EnqueueScans(targets)`. On failure, release claims and **do not** advance the marker.
4. On success, persist `next_marker`.
This is the salvaged core of PR #43 (`internal/autoscan/service.go` resolve/suppress/enqueue, `dedupe.go`, `suppress.go`), generalized so its input is "paths from a provider" rather than "paths from arr." Path-rewrite helpers (`rewrite.go`, `suggest.go`, `history.go`, the arr history client) leave the host and move into the plugin (§10).
## 8. Host: Connections (reuse or own)
A **connection** is "an arr server Silo can reach," created two ways:
- **Own:** name + base URL + API key, entered in the Autoscan category; the key is Fernet-encrypted at rest (same mechanism as existing service credentials — never plaintext in JSONB).
- **Linked (reuse):** a live reference to an existing Requests `request_integrations` arr server. Resolved at use-time, so edits in Requests propagate.
**Resolution:** on each poll the host resolves the chosen connection (own or linked) into concrete `{base_url, api_key}` and passes them in `PollChangesRequest.connection` (§5). The plugin is credential-source-agnostic — it only ever sees resolved values, fresh each poll, so a change to a reused Requests credential propagates immediately with no re-push. This "reuse or own" behaviour is therefore free for every future provider. (Resolving per-call rather than caching in plugin config also means no decrypted secret is ever stored at rest in the plugin.)
**Decoupling from Requests:** the link is **soft and optional** — `SET NULL` / surfaced-as-"needs attention", never cascading. Requests has no knowledge of Autoscan. Autoscan and Requests are peers; Autoscan merely *offers* to borrow a Requests connection.
## 9. Host: Autoscan category & data model
New top-level **Autoscan** admin section (its own nav entry, not a Requests tab). Manages **Connections** (§8) and **Sources**.
A **source** corresponds to one configured `scan_source.v1` capability instance and carries:
- **Host-owned per-source config:** path rewrites (`autoscan_sources.path_rewrites`, edited in the Autoscan Sources UI) and any provider-specific settings. _(Revised post-merge: rewrites moved host-side; the plugin has no config.)_
- **Host-owned per-source state:** the chosen connection, enabled on/off, poll interval (per-source, not one global timer), the opaque marker, and last-run status/error.
**Install flow + source auto-discovery (resolves §14 risk #1).** A scan-source plugin (Sonarr/Radarr, etc.) is installed through the **normal `/admin/plugins`** page — there is no separate install path; the installer doesn't special-case the capability type. Source rows are then never created by hand: the host enumerates every installed `scan_source.v1` capability (via the plugin installation store) and *seeds a disabled, connection-less row* per capability (`INSERT … ON CONFLICT (installation_id, capability_id) DO NOTHING`, never disturbing existing config). Discovery runs **both on each poll cycle AND when the Autoscan sources list is viewed** (so a freshly-installed plugin appears in Autoscan immediately, even when autoscan is disabled and nothing is polling). The operator then binds a connection and enables it. Consequently **`connection_id` is nullable**: a freshly-discovered source has none, and the engine skips any enabled source with no connection (recording a "no connection bound" status). The admin API rejects enabling a source that has no effective connection.
**Per-source interval** is honoured as a floor: the global poll task runs at `default_poll_interval_seconds`, and within a cycle the engine skips a source whose `last_run_at` is newer than its own `poll_interval_seconds` (falling back to the default). A source therefore polls *at most* every N seconds, never more often than the global cadence.
### Schema changes (host)
Rework the PR #43 tables; **no FK into `request_integrations`**:
- `autoscan_connections` — `id`, `name`, `kind` (e.g. `sonarr`/`radarr`), and *either* own credentials (`base_url`, `api_key_ref` encrypted) *or* a nullable soft link `request_integration_id` (`ON DELETE SET NULL`).
- `autoscan_sources` — `id`, `(installation_id, capability_id)` (unique; the discovered identity), **nullable** `connection_id → autoscan_connections` (`ON DELETE RESTRICT`; null until the operator binds one), `enabled`, `poll_interval_seconds` (nullable; null = use default), `marker` (opaque text, nullable), `last_run_at`, `last_error`. Creation-time validity for connections (own creds OR a link) is enforced at the application layer (the DB CHECK would conflict with `ON DELETE SET NULL` on the reuse link).
- `autoscan_settings` (optional) — retain a global enable + default interval; per-source interval overrides it.
A migration supersedes PR #43's `171_autoscan` schema (which is not yet upstream, so this is a forward redefinition, not a production migration of live data on `origin/main`).
## 10. Plugin: Sonarr/Radarr scan-source
A new installable repo structured like `silo-plugin-tmdb` (standalone Go module, `manifest.json` declaring `scan_source.v1`, gRPC `main.go`, depends on the new tagged `silo-plugin-sdk`). Receives resolved arr `{base_url, api_key}` as config. Implements `PollChanges`:
1. Treat `marker` as the "since" timestamp (empty ⇒ now).
2. Call arr `/api/v3/history/since`, capped by a bounded window (24h max-lookback floor + overlap buffer — relocated from PR #43) so a long outage cannot pull an oversized response.
3. Extract paths: **imports** (`downloadFolderImported.importedPath`) and **renames** (`episodeFileRenamed`/`movieFileRenamed`: both new `path` and old `sourcePath`). Deletes ignored (upgrade-deletes are covered by the paired import; standalone deletes carry no path).
4. Return `{source_paths = the RAW arr paths, next_marker = newest history timestamp}`. The plugin does NOT rewrite — the host normalizes separators and applies the source's rewrite rules. _(Revised post-merge.)_
**Rewrite config + "Sync from arr":** the rewrite list and the suffix-match suggester (which matches arr root folders to Silo media folders) live in the plugin. The suggester needs Silo's media-folder list, obtained via the existing host library-listing service (`internal/pluginhost` library lister) exposed to plugins. The relocated logic is PR #43's `suggest.go`/`suggest_deps.go`/`rewrite.go`/`history.go`.
## 11. Backward compatibility (hard requirement)
Adding `scan_source.v1` **must not break existing plugins.** It doesn't, by construction:
- Capabilities are independent opt-in declarations. Each host subsystem **filters for its own type and ignores the rest** (verified: `metadata_provider.v1`, `scheduled_task.v1`, `event_consumer.v1`, … each use `if type != mine { skip }`). There is **no global allowlist that rejects unknown capability types.**
- **Additive-only rule:** the SDK change *adds* new proto messages/service + a new capability string. It never modifies the shape of an existing capability. (Mutating an existing capability is the only thing that would break plugins; we forbid it.)
- **No forced upgrades:** existing plugin repos stay pinned to their current SDK version and build unchanged. Only a plugin that *wants* to be a scan source adopts the new SDK version.
## 12. Build order & repo decomposition
One shared contract, three repos (+ catalog), in dependency order:
1. **`silo-plugin-sdk`** (long pole): define `scan_source.v1` (proto + generated code + manifest helper + runtime server scaffold), tag a new version (e.g. `v0.5.0`). The tagged release is a hard serialization point — host and plugin cannot build against the contract until it exists.
2. **silo-server (host)** — bump SDK dep; build AutoscanStore (generalized), the `scan_source.v1` caller (timer → `PollChanges`), connections + resolution, the Autoscan category UI, retire the in-process arr autoscan, add the reworked migration. *Parallelizable with (3) once (1) is tagged.*
3. **arr plugin (new repo)** — depends on SDK `v0.5.0`; implement `PollChanges`, rewrites, "Sync from arr", bounded window.
4. **`silo-plugins`** (catalog) — add the entry so the arr plugin is installable.
Each repo gets its **own implementation plan** (separate `writing-plans` pass); this single spec defines the shared contract and per-side responsibilities they all build against.
**Local logistics:** only `silo-server` and `silo-plugin-tmdb` are currently checked out. Implementation will require `silo-plugin-sdk` and a new arr-plugin repo (and later `silo-plugins`) available locally.
## 13. Salvage map (from closed PR #43)
| PR #43 code | Destination |
|---|---|
| arr `/history` client, imports + renames, bounded window (`history.go`) | arr plugin |
| path rewrite apply (`rewrite.go`) + "Sync from arr" suggester (`suggest.go`, `suggest_deps.go`) | arr plugin |
| `last_poll_at` cursor | host, generalized to opaque per-source `marker` |
| resolve + dedupe + suppress + enqueue (`service.go`, `dedupe.go`, `suppress.go`) | host AutoscanStore (generic) |
| `171_autoscan` schema (FK → `request_integrations`) | host, reworked: `autoscan_connections` + `autoscan_sources`, no requests FK |
| Autoscan admin tab under Requests (`AdminRequests.tsx`) | host, as its own Autoscan category |
## 14. Risks & open questions
1. **Plugin multiplicity for multiple arr servers.** Does the runtime support multiple configured instances of one plugin (one per arr server), or is config per-installation only? If the latter, "two Sonarrs + one Radarr" implies either multiple installations or a plugin that manages several connections internally. *Resolve before the host/plugin plans by reading `internal/plugins` installation + capability-instance handling.*
2. **Plugin config UI richness.** The rewrite table + a "Sync from arr" action are richer than typical key-value plugin config. Verify the plugin config-schema/admin-config UI can host a structured list and an action button; if not, design a minimal affordance (e.g. suggester runs on first configure, or a dedicated plugin HTTP route via `http_routes.v1`).
3. **Library-folder list exposure to plugins.** Confirm the `internal/pluginhost` library lister is reachable by a `scan_source` plugin for the suggester. If not, add a host service method.
4. **Marker persistence vs at-least-once.** Ensure marker write happens only after successful enqueue, matching PR #43's "don't advance on failure" guarantee.
5. **Closed PR disposition.** The in-process autoscan on local `main` (34 commits) should be removed/replaced rather than shipped; sequence the host plan to retire it cleanly.
## 15. Testing strategy
- **SDK:** proto compiles; manifest helper round-trips `scan_source.v1`; runtime scaffold serves `PollChanges`.
- **Host engine:** table-driven tests for resolve→suppress→enqueue→marker (port PR #43's `service_test.go`, generalized to a fake `ScanSource` client): dedupe, distinct-paths-same-folder, disabled no-op, poll failure keeps marker, enqueue failure releases claims + keeps marker, opaque-marker pass-through. Run in the libvips-equipped environment (host handler tests link libvips via CGO — see workspace note).
- **Connections:** resolution of own vs linked; linked falls back gracefully when the Requests row is deleted (`SET NULL`).
- **arr plugin:** history parsing (imports + renames, ignores unrelated events), rewrite application, bounded window, first-run "from now"; suffix-match suggester cases (unique/ambiguous/covered/no-op/normalization) ported from PR #43.
- **Compat:** an existing-style plugin manifest with no `scan_source.v1` is unaffected by host scan-source dispatch.
## 16. Out of scope / future
- inotify and Ceph-notify scan-source plugins (the contract accommodates them via the buffer-and-flush pattern of §6).
- A push/event delivery mode, if pull latency ever proves insufficient.
- A `TestConnection` RPC on the capability.
- Migrating Requests onto a shared connection abstraction (today's reuse is a one-directional soft link only).
docs/superpowers/specs/2026-06-02-autoscan-rewrite-sync-design.md
+173
View File
@@ -0,0 +1,173 @@
# Autoscan: Sync Path Rewrites from Arr Root Folders
**Status:** Design approved, ready for implementation plan
**Date:** 2026-06-02
**Area:** `internal/autoscan`, `internal/api`, `web/src/pages/AdminRequests.tsx`
**Extends:** the autoscan feature (`docs/superpowers/specs/2026-06-02-autoscan-arr-polling-design.md`)
## Summary
Autoscan resolves an imported file path against Silo's media folders, but
Radarr/Sonarr often report files under paths that differ from how Silo mounts
the same content — so operators must hand-author per-root `path_rewrites`. For a
multi-instance, many-root deployment this can be dozens of rows.
This adds a **"Sync rewrites from arr"** convenience: for one instance, query its
root folders, match each root to a Silo media folder by **shared trailing path
segments**, and present the resulting `{from, to}` rewrites as a **preview** the
admin reviews and commits. Manual rewrites remain the source of truth; sync only
*fills the editor* — nothing persists until the admin saves.
The matcher is purely structural (no assumptions about any particular path
layout) so it helps any deployment whose arr and Silo share folder names at some
depth, and degrades gracefully (no shared names → "unmatched" → manual entry)
when they do not.
## Goals
- One-click suggestion of `path_rewrites` for an instance from its arr root folders.
- Correct disambiguation when multiple Silo folders share a leaf name.
- Preview-then-commit: the admin sees proposed/unmatched/ambiguous and saves
through the existing per-source save. Manual rows are never removed or overwritten.
- Deployment-agnostic: no hard-coded paths; works for any structure; surfaces
match confidence so weak matches can be rejected.
## Non-Goals
- No automatic/background writing of rewrites — the admin always commits.
- No new persistence path — apply is the existing `PUT /sources/{id}`.
- No case-insensitive matching (case-sensitive segments; correct on Linux). A
toggle can be added later if needed.
- No "sync all instances at once" — per-instance, matching the editor UI.
## Design
### 1. Matching function (core, pure)
A single side-effect-free function in `internal/autoscan`:
```go
func suggestRewrites(arrRoots []string, siloFolderPaths []string, existing []PathRewrite) RewriteSuggestions
```
```go
type RewriteSuggestions struct {
Proposed []ProposedRewrite `json:"proposed"`
Unmatched []string `json:"unmatched"`
Ambiguous []AmbiguousRoot `json:"ambiguous"`
Covered []string `json:"covered"`
}
type ProposedRewrite struct {
From string `json:"from"` // arr root
To string `json:"to"` // matched Silo folder path
MatchDepth int `json:"match_depth"` // trailing segments shared (confidence)
}
type AmbiguousRoot struct {
Root string `json:"root"`
Candidates []string `json:"candidates"`
}
```
**Normalization** (applied to every path before comparison): backslashes → `/`
(reuse `normalizeSeparators`), strip a trailing `/`, collapse duplicate slashes.
This makes any path format (POSIX, Windows, single- or many-root) comparable.
**Per arr root:**
1. **Covered-first:** if any `existing` rewrite's `From` already matches the root
(exact, or root starts with `From + "/"`, using the same boundary rule as
`applyRewrites`), classify as **Covered** and skip — so sync never re-proposes
what a manual or broad rule already handles.
2. Split both the root and each Silo folder path into segments; compute the
**longest common trailing-segment suffix** (full-segment equality, so `4ktv7`
does not match `4ktv70`).
3. Among Silo folders, take the maximum suffix length `> 0` and the folders that
achieve it:
- exactly one → **Proposed** `{From: root, To: folder, MatchDepth: n}`
- more than one → **Ambiguous** `{root, candidates}`
- zero → **Unmatched**
No deployment constants appear in the function; it operates only on the string
slices it is given.
### 2. Preview endpoint & service
**Endpoint:** `GET /api/v1/admin/autoscan/sources/{id}/rewrite-suggestions` →
`RewriteSuggestions` JSON (admin-gated, like the other autoscan routes).
**Service method** `Service.SuggestRewrites(ctx, integrationID) (RewriteSuggestions, error)`:
1. `Repository.GetSource(id)` — a new single-source variant of the existing join
that returns `kind`/`base_url`/`api_key_ref`/`path_rewrites` even when the
source is disabled. Not found → `ErrIntegrationNotFound`.
2. Decrypt the API key via the existing `SecretResolver`.
3. `RootFolderClient.RootFolders(ctx, baseURL, apiKey)` → arr root paths.
4. `FolderLister.ListFolderPaths(ctx)` → all Silo media-folder paths.
5. Return `suggestRewrites(arrRoots, siloPaths, source.PathRewrites)`.
**Two new narrow `Service` dependencies** (injected like `history`/`resolver`):
- `RootFolderClient.RootFolders(ctx, baseURL, apiKey string) ([]string, error)` —
an `arrclient`-backed impl reusing `arrclient.ListRootFolders` (returns paths only).
- `FolderLister.ListFolderPaths(ctx) ([]string, error)` — a thin adapter over
`catalog.FolderRepository.List()` flattening each folder's `Paths`.
**Wiring:** both constructed where the autoscan handler is built in `router.go`
(`deps.FolderRepo` already in scope; the arr client is trivial). The handler's
service interface gains `SuggestRewrites`.
**Errors:** instance not found → 404; arr unreachable / bad key → 502 with the
arr error surfaced; Silo folder-list failure → 500.
No new write path — the frontend merges `proposed` into the editor and the admin
saves via the existing `PUT /sources/{id}`.
### 3. Frontend (preview UI)
In `AutoscanSourceEditor` (`web/src/pages/AdminRequests.tsx`), add a **"Sync from
arr"** button beside Save. On click, a new hook
(`useAutoscanRewriteSuggestions`) calls the endpoint and opens a preview panel:
- **Proposed** — each row: `From → To` + a **confidence badge** from `match_depth`
(e.g. "3 segments" strong, "1 segment — weak"), with a checkbox (default checked).
- **Unmatched** — info-only list of arr roots with no Silo match ("add manually").
- **Ambiguous** — arr root + candidate Silo paths (info-only; admin adds a manual row).
**"Add selected to rewrites"** merges the checked proposals into the editor's
existing rewrite rows, **deduped by `from`, never removing manual rows**. The
admin then clicks the normal **Save**. New types in `web/src/api/types.ts`
(`AutoscanRewriteSuggestions`, `ProposedRewrite`, `AmbiguousRoot`) and hook in
`web/src/hooks/queries/useAutoscan.ts`. Reuse existing
`Button`/`Badge`/`Dialog`/`Field` primitives.
### 4. Testing
- **`suggestRewrites`** — table-driven, fixtures spanning layouts: multi-segment
match, single-segment unique, longest-suffix disambiguation (`Anime/Subs` vs
`Anime2/Subs`), unmatched (no shared segment), already-covered via a broad
prefix rule, an ambiguous tie, plus **unlike-this-deployment** cases
(`/data/Movies` arr vs `/library/Films` Silo → unmatched; `/srv/tv/Show` vs
`/tank/television/Show` → leaf match) and normalization (trailing slash,
backslashes, duplicate slashes).
- **`Service.SuggestRewrites`** — fakes for source store / `RootFolderClient` /
`FolderLister` / secrets: asserts proposed/unmatched/ambiguous; arr-unreachable
surfaces the error.
- **Handler** — returns the JSON; 404 for missing instance; 502 when the arr is
unreachable.
- **Frontend** — the button hits the endpoint; the preview renders all three
buckets with confidence badges; "Add selected" merges deduped rows; Save
persists via the existing mutation.
## Risks & open considerations
- **Weak (1-segment) matches** can be coincidental across very different layouts;
mitigated by surfacing `match_depth` so the admin rejects them before saving.
- **Case-sensitivity** means case-mismatched mounts produce Unmatched rather than
a wrong auto-match — a deliberate safe default; revisit only if requested.
- **Arr reachability** is required for sync (unlike the rest of autoscan which
polls on a schedule); a 502 with the arr error tells the admin to check the
instance, and manual entry still works.
## References
- Reused: `internal/requests/arrclient.ListRootFolders`,
`internal/catalog/FolderRepository.List`, `internal/autoscan` (`PathRewrite`,
`normalizeSeparators`, `Repository`, `Service`, the existing
`PUT /autoscan/sources/{id}` save).
docs/superpowers/specs/2026-06-05-autoscan-source-labels-design.md
+161
View File
@@ -0,0 +1,161 @@
# Autoscan source labels — design
**Date:** 2026-06-05
**Status:** Approved, pending implementation plan
**Scope:** Admin UI labeling of autoscan scan sources, generic across plugins, plus an operator-editable per-source label.
Commands assume the repository root is the cwd.
## Problem
Autoscan sources are labeled in the admin UI (Sources tab and Activity tab) by their
`capability_id` plus `plugin #<installation_id>`. The `arr` scan-source plugin fans out
one source per connection (Sonarr, Sonarr4k, Radarr, Radarr4k) under a single generic
`arr` capability, so every row rendered an identical "arr (plugin #4)" label. A prior
change made the Sources/Activity labels lead with the bound connection name, which fixes
`arr`, but the fallback for any other plugin is still the cryptic `capability_id` +
`plugin #N`.
Two gaps remain:
1. **Other plugins look cryptic.** A connectionless plugin (e.g. cephfs) or any future
plugin still falls back to the raw `capability_id`, ignoring the human-friendly
`display_name` the plugin already declares in its manifest.
2. **No operator control.** An operator cannot name a source themselves. This is the only
fully-general answer for plugin shapes the automatic chain cannot disambiguate (e.g. a
single plugin installation creating several connectionless sources distinguished only
by their `source_config`).
## Goals
- Generic, plugin-agnostic labels driven by metadata that already exists
(manifest `display_name`, connection `name`), so future plugins look good with **no
host code per plugin**.
- An optional operator-editable label per source that overrides the automatic chain.
- A single shared label-resolution helper, consumed by both the Sources and Activity
panels, so the logic is not duplicated.
## Non-goals
- Plugin-declared label templates in the manifest/SDK (the "option C" from brainstorming).
Deferred: it requires a cross-repo SDK/manifest contract change. The chain below leaves a
clean seam for it.
- Backend-side label resolution returned on API payloads. The resolution stays in the
frontend helper for now; promote it to the backend only if the mobile clients need the
same labels.
## Resolution chain
A source's display label resolves through four rungs, most-specific first:
```
operator label -> connection name -> manifest display_name -> capability_id
```
- **operator label** — the optional per-source text the admin sets (this design's new field).
- **connection name** — the operator-named `autoscan_connections.name` bound to the source.
- **manifest display_name** — from the plugin manifest, surfaced by the
`scan-source-plugins` endpoint (`AutoscanAvailableSource.display_name`).
- **capability_id** — last-resort raw identifier; never worse than today.
The chosen rung is the **title**. The remaining identity (plugin display name / capability +
`plugin #<installation_id>`) renders as a **subtitle**, so no information is lost when a
higher rung wins. Example: operator label `4K Movies` as title, `Radarr4k · plugin #4` as
subtitle.
### Per-plugin behavior
| Source shape | Title | Subtitle |
| --- | --- | --- |
| arr, connection bound, no operator label | `Radarr4k` (connection) | `arr · plugin #4` |
| arr, operator label set | `4K Movies` (operator) | `Radarr4k · plugin #4` |
| cephfs, no connection | `CephFS Watcher` (manifest display_name) | `plugin #5` |
| future plugin, nothing declared | `capability_id` | `plugin #N` |
| Activity event whose source was deleted (`source_id` null) | manifest display_name or `capability_id` from the event's stored fields | `plugin #N` |
## Components
### Shared helper — `web/src/lib/autoscanLabels.ts` (new)
Pure, React-free, unit-tested.
- `composeSourceLabel({ operatorLabel, connectionName, displayName, capabilityId, installationId })`
`=> { name: string; detail: string }` — implements the four-rung chain and the
subtitle composition.
- `pluginDisplayNameKey(installationId, capabilityId) => string` — stable map key.
- `buildPluginDisplayNames(available: AutoscanAvailableSource[]) => Map<string, string>` —
builds the `installation_id + capability_id -> display_name` lookup both panels use.
Future seam: `composeSourceLabel` may later accept an optional `configLabel` rung (for the
deferred plugin-template option) without changing either panel's call shape.
### Migration — `migrations/174_autoscan_source_label.{up,down}.sql` (new)
```sql
-- up
ALTER TABLE public.autoscan_sources ADD COLUMN label text NOT NULL DEFAULT '';
-- down
ALTER TABLE public.autoscan_sources DROP COLUMN label;
```
`label` empty string = unset; the resolution chain falls through to the automatic rungs.
174 is a fresh, unused migration number (no collision with the reused-172 history).
### Backend (Go)
- `internal/autoscan/types.go` — add `Label string` to the `Source` struct and the source
input type.
- `internal/autoscan/repository.go` — add `label` to `sourceColumns`, `scanSource`, and the
create/update statements.
- `internal/api/handlers/autoscan.go` — accept `label` on source update; include it in the
`autoscanStatusSource` and source responses. Trim whitespace and cap length at 120
characters server-side.
### Frontend
- `web/src/api/types.ts` — add `label` to `AutoscanSource` and `AutoscanSourceInput`.
- `web/src/pages/admin/autoscan/SourcesPanel.tsx` — replace the inline `sourceIdentity`
block with a call to `composeSourceLabel`. Add an optional "Label" text input to the
source row, saved on blur via the existing `fullBody` PATCH path (mirrors the interval
field's `handleIntervalBlur` auto-save). Build the display-name map from the existing
`useAvailableScanSources()` query.
- `web/src/pages/admin/autoscan/ActivityPanel.tsx` — add `useAvailableScanSources()`, build
the display-name map, map `source_id -> source` to obtain the operator label and
connection, and route `pollSourceName` / `scanSourceName` through `composeSourceLabel`,
returning `.name`. Preserve the existing `"Autoscan"` fallback for scans with no
capability.
## Edit UX
Inline per-row, reusing the existing on-blur auto-save pattern (no modal, no new
interaction model). The label input is blank by default with placeholder text indicating
it is optional. Saving sends the full row state through the same source-update mutation
used for connection/interval/rewrites.
## Error handling and edge cases
- **Deleted source on an Activity event** — `autoscan_events.source_id` is
`ON DELETE SET NULL`; with a null `source_id` there is no operator label or connection,
so the chain resolves from the event row's own stored `capability_id` / `installation_id`
(plus manifest display_name if the plugin is still installed).
- **Uninstalled plugin** — `display_name` does not resolve; the chain falls to
`capability_id`.
- **Bound connection** — `autoscan_sources.connection_id` is `ON DELETE RESTRICT`, so a
bound connection always resolves to a name.
- **Length / whitespace** — the operator label is trimmed and capped at 120 characters
server-side; an all-whitespace label normalizes to empty (unset).
## Testing
- `web/src/lib/autoscanLabels.test.ts` — covers each rung of the chain: operator label
wins, connection name wins, manifest display_name wins, capability fallback, and the
subtitle composition.
- Extend the existing autoscan repository/handler Go tests to cover the `label` column
round-trip (create/update/read) and the server-side trim + length cap.
## Scope summary
One migration pair, three Go files, four frontend files (one new). No SDK changes, no
multi-repo coordination. Operator-editable labels (this design) and the automatic chain
share the same helper, so the deferred plugin-template option layers on later with no
rework.
go.mod
+1 -1
View File
@@ -48,7 +48,7 @@ require (
)
require (
github.com/Silo-Server/silo-plugin-sdk v0.4.0
github.com/Silo-Server/silo-plugin-sdk v0.4.1-0.20260604220604-0d7865171220
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.21 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.21 // indirect
go.sum
+2 -2
View File
@@ -1,7 +1,7 @@
entgo.io/ent v0.14.3 h1:wokAV/kIlH9TeklJWGGS7AYJdVckr0DloWjIcO9iIIQ=
entgo.io/ent v0.14.3/go.mod h1:aDPE/OziPEu8+OWbzy4UlvWmD2/kbRuWfK2A40hcxJM=
github.com/Silo-Server/silo-plugin-sdk v0.4.0 h1:DJkRROQfr/kfwnF5dUdkhmbCso1KDLZc7uK7YfJnaO0=
github.com/Silo-Server/silo-plugin-sdk v0.4.0/go.mod h1:etqmxLTwjxpFH9goAjBDfNDoqHMv2/sqUXu8yx3hNfA=
github.com/Silo-Server/silo-plugin-sdk v0.4.1-0.20260604220604-0d7865171220 h1:tiwfULdh9yOW7HO6fYf8v+lXIsrCrGfp3GM8ymhrBME=
github.com/Silo-Server/silo-plugin-sdk v0.4.1-0.20260604220604-0d7865171220/go.mod h1:etqmxLTwjxpFH9goAjBDfNDoqHMv2/sqUXu8yx3hNfA=
github.com/abadojack/whatlanggo v1.0.1 h1:19N6YogDnf71CTHm3Mp2qhYfkRdyvbgwWdd2EPxJRG4=
github.com/abadojack/whatlanggo v1.0.1/go.mod h1:66WiQbSbJBIlOZMsvbKe5m6pzQovxCH9B/K8tQB2uoc=
github.com/aws/aws-sdk-go-v2 v1.41.5 h1:dj5kopbwUsVUVFgO4Fi5BIT3t4WyqIDjGKCangnV/yY=
internal/api/autoscan_wiring.go
+175
View File
@@ -0,0 +1,175 @@
package api
import (
"context"
"fmt"
"strings"
"github.com/redis/go-redis/v9"
"github.com/Silo-Server/silo-server/internal/autoscan"
"github.com/Silo-Server/silo-server/internal/catalog"
"github.com/Silo-Server/silo-server/internal/plugins"
mediarequests "github.com/Silo-Server/silo-server/internal/requests"
"github.com/Silo-Server/silo-server/internal/scantrigger"
)
// autoscanQueuer is the scan-enqueue surface BuildAutoscanService needs; it is
// satisfied by *scanqueue.Service (autoscan.Queuer's concrete production impl).
type autoscanQueuer = autoscan.Queuer
// RequestIntegrationLookup adapts the Requests repository to the autoscan
// connection resolver's RequestIntegrationLookup: it resolves a soft-linked
// Requests integration to its base URL and (write-only) api-key ref.
type RequestIntegrationLookup struct {
Repo *mediarequests.Repository
}
func (l RequestIntegrationLookup) Get(ctx context.Context, integrationID string) (baseURL, apiKeyRef string, err error) {
integration, err := l.Repo.GetIntegration(ctx, integrationID)
if err != nil {
return "", "", err
}
// A reused Requests connection must honor the integration's live state. The
// v1 poll gated on `WHERE ri.enabled = true`; here we surface a disabled or
// unconfigured (blank base_url) integration as an error so the engine turns
// it into a logged skip / RecordError rather than polling an unusable target.
if err := checkRequestIntegrationUsable(integrationID, integration.Enabled, integration.BaseURL); err != nil {
return "", "", err
}
return integration.BaseURL, integration.APIKeyRef, nil
}
// checkRequestIntegrationUsable returns a non-nil error when a linked Requests
// integration cannot be polled: it is disabled, or it has no base_url. Extracted
// as a pure function so the gating is unit-testable without a DB-backed repo.
func checkRequestIntegrationUsable(integrationID string, enabled bool, baseURL string) error {
if !enabled {
return fmt.Errorf("linked requests integration %q is disabled", integrationID)
}
if strings.TrimSpace(baseURL) == "" {
return fmt.Errorf("linked requests integration %q has no base_url configured", integrationID)
}
return nil
}
// PluginScanSourceAdapter adapts plugins.Service to autoscan.ScanSourceResolver.
// plugins.Service.ScanSourceClient returns the concrete
// *pluginhost.ScanSourceClient; that concrete type satisfies
// autoscan.PollChangesClient (it has the matching PollChanges method), so the
// adapter declares the exported interface as its return type and returns the
// concrete value (Go has no return-type covariance, so the method signature must
// name the interface exactly to satisfy ScanSourceResolver).
type PluginScanSourceAdapter struct {
Svc *plugins.Service
}
func (a PluginScanSourceAdapter) ScanSourceClient(ctx context.Context, installationID int, capabilityID string) (autoscan.PollChangesClient, error) {
return a.Svc.ScanSourceClient(ctx, installationID, capabilityID)
}
// scanSourceCapabilityType is the plugin capability type autoscan discovery
// enumerates.
const scanSourceCapabilityType = "scan_source.v1"
// PluginScanSourceLister adapts the plugin installation store to
// autoscan.ScanSourceLister: it enumerates every installed scan_source.v1
// capability across ALL installed plugins, regardless of enabled state.
//
// Using List (not ListEnabled) is deliberate for orphan detection: a temporarily
// DISABLED-but-installed plugin must NOT be treated as orphaned. If we dropped
// disabled plugins here, PollOnce would prune their sources and skip them with no
// last_error — they'd vanish silently. Keeping them present means PollOnce still
// attempts them; the plugin client fails to load (not running) and the source
// gets a visible RecordError instead. Only a fully UNINSTALLED plugin (gone from
// List entirely) is a true orphan. The Add-source picker shares this all-installed
// set, which is fine — operators can bind sources against an installed capability.
type PluginScanSourceLister struct {
Store *plugins.InstallationStore
}
func (l PluginScanSourceLister) ListScanSources(ctx context.Context) ([]autoscan.DiscoveredSource, error) {
installations, err := l.Store.List(ctx)
if err != nil {
return nil, err
}
var out []autoscan.DiscoveredSource
for _, inst := range installations {
caps, err := l.Store.ListCapabilities(ctx, inst.ID)
if err != nil {
return nil, err
}
for _, c := range caps {
if c == nil || c.Type != scanSourceCapabilityType {
continue
}
out = append(out, autoscan.DiscoveredSource{
InstallationID: c.InstallationID,
CapabilityID: c.ID,
PluginID: inst.PluginID,
DisplayName: scanSourceDisplayName(inst.PluginID, c),
})
}
}
return out, nil
}
// scanSourceDisplayName derives a human-friendly label for a scan_source
// capability: the capability manifest's display_name when present, else the
// plugin id (with the capability id appended when it adds information).
func scanSourceDisplayName(pluginID string, c *plugins.Capability) string {
if c != nil && c.Metadata != nil {
if name, ok := c.Metadata["display_name"].(string); ok && strings.TrimSpace(name) != "" {
return strings.TrimSpace(name)
}
}
switch {
case pluginID != "" && c != nil && c.ID != "":
return pluginID + " / " + c.ID
case pluginID != "":
return pluginID
case c != nil:
return c.ID
default:
return ""
}
}
// AutoscanSecretResolver resolves an encrypted api-key reference to plaintext.
// It is satisfied by the server settings repo (Get(ctx, key) (string, error)).
type AutoscanSecretResolver interface {
Get(ctx context.Context, ref string) (string, error)
}
// BuildAutoscanService wires the v2 autoscan engine from its concrete
// dependencies. Both the HTTP router (manual trigger) and the background poll
// task share this constructor so the adapter wiring lives in exactly one place.
func BuildAutoscanService(
repo *autoscan.Repository,
pluginService *plugins.Service,
installationStore *plugins.InstallationStore,
requestsRepo *mediarequests.Repository,
secrets AutoscanSecretResolver,
folderRepo *catalog.FolderRepository,
queue autoscanQueuer,
redisClient *redis.Client,
) *autoscan.Service {
provider := autoscan.NewPluginProvider(PluginScanSourceAdapter{pluginService})
connRes := autoscan.NewConnectionResolver(RequestIntegrationLookup{requestsRepo}, secrets)
svc := autoscan.NewService(
repo,
provider,
connRes,
scantrigger.NewResolver(folderRepo),
queue,
autoscan.NewRedisSuppressor(redisClient),
PluginScanSourceLister{installationStore},
)
// Wire the connection-test + rewrite-suggester deps: a (long-timeout)
// arr root-folder/status client and a Silo media-folder lister.
svc.SetSuggesterDeps(
autoscan.NewArrRootFolderClient(nil),
autoscan.NewCatalogFolderLister(folderRepo),
)
return svc
}
internal/api/autoscan_wiring_test.go
+24
View File
@@ -0,0 +1,24 @@
package api
import "testing"
// TestCheckRequestIntegrationUsable verifies the Fix 4 gate: a reused Requests
// connection is rejected when the integration is disabled or has no base_url, so
// the autoscan engine skips it instead of polling an unusable target.
func TestCheckRequestIntegrationUsable(t *testing.T) {
t.Run("enabled with base_url is usable", func(t *testing.T) {
if err := checkRequestIntegrationUsable("req-1", true, "http://radarr:7878"); err != nil {
t.Fatalf("expected usable, got %v", err)
}
})
t.Run("disabled is rejected", func(t *testing.T) {
if err := checkRequestIntegrationUsable("req-1", false, "http://radarr:7878"); err == nil {
t.Fatal("expected error for disabled integration")
}
})
t.Run("blank base_url is rejected", func(t *testing.T) {
if err := checkRequestIntegrationUsable("req-1", true, " "); err == nil {
t.Fatal("expected error for blank base_url")
}
})
}
internal/api/handlers/autoscan.go
+1007
View File
File diff suppressed because it is too large Load Diff
internal/api/handlers/autoscan_test.go
+1302
View File
File diff suppressed because it is too large Load Diff
internal/api/router.go
+44 -1
View File
@@ -23,6 +23,7 @@ import (
"github.com/Silo-Server/silo-server/internal/api/handlers"
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
"github.com/Silo-Server/silo-server/internal/auth"
"github.com/Silo-Server/silo-server/internal/autoscan"
"github.com/Silo-Server/silo-server/internal/cache"
"github.com/Silo-Server/silo-server/internal/catalog"
"github.com/Silo-Server/silo-server/internal/catalogseed"
@@ -340,6 +341,7 @@ func NewRouter(deps Dependencies) chi.Router {
var calendarRepo *catalog.CalendarRepository
var webhookSyncHandler *handlers.WebhookSyncHandler
var requestHandler *handlers.RequestsHandler
var autoscanHandler *handlers.AutoscanHandler
if deps.DB != nil {
browseRepo := catalog.NewBrowseRepository(deps.DB)
itemRepo = catalog.NewItemRepository(deps.DB)
@@ -403,8 +405,9 @@ func NewRouter(deps Dependencies) chi.Router {
if deps.Config != nil {
tmdbAPIKey = deps.Config.TMDBAPIKey
}
requestsRepo := mediarequests.NewRepository(deps.DB)
requestSvc := mediarequests.NewService(
mediarequests.NewRepository(deps.DB),
requestsRepo,
tmdb.NewClient(tmdbAPIKey, 40),
mediarequests.NewCatalogPresence(itemRepo, providerIDRepo),
)
@@ -415,6 +418,26 @@ func NewRouter(deps Dependencies) chi.Router {
}
requestHandler = handlers.NewRequestsHandler(requestSvc)
autoscanRepo := autoscan.NewRepository(deps.DB)
if deps.FolderRepo != nil && deps.LibraryScanQueue != nil && deps.PluginService != nil {
autoscanSvc := BuildAutoscanService(
autoscanRepo,
deps.PluginService,
plugins.NewInstallationStore(deps.DB),
requestsRepo,
settingsRepo,
deps.FolderRepo,
deps.LibraryScanQueue,
deps.RedisClient,
)
autoscanHandler = handlers.NewAutoscanHandler(autoscanRepo, autoscanSvc)
// Wire the optional poll-task rescheduler so a settings change
// re-applies the poll interval without a restart.
if deps.TaskManager != nil {
autoscanHandler.SetTriggerUpdater(deps.TaskManager)
}
}
if deps.PersonRepo != nil {
peopleHandler = handlers.NewPeopleHandler(deps.PersonRepo, browseRepo, itemRepo, detailSvc)
peopleHandler.SetItemsHandler(itemsHandler)
@@ -2049,6 +2072,26 @@ func NewRouter(deps Dependencies) chi.Router {
r.Post("/request-integrations/{id}/options", requestHandler.HandleLoadIntegrationOptions)
}
if autoscanHandler != nil {
r.Get("/autoscan/settings", autoscanHandler.HandleGetSettings)
r.Put("/autoscan/settings", autoscanHandler.HandleUpdateSettings)
r.Get("/autoscan/connections", autoscanHandler.HandleListConnections)
r.Post("/autoscan/connections", autoscanHandler.HandleCreateConnection)
r.Put("/autoscan/connections/{id}", autoscanHandler.HandleUpdateConnection)
r.Delete("/autoscan/connections/{id}", autoscanHandler.HandleDeleteConnection)
r.Post("/autoscan/connections/test", autoscanHandler.HandleTestConnection)
r.Get("/autoscan/scan-source-plugins", autoscanHandler.HandleListAvailableScanSources)
r.Get("/autoscan/sources", autoscanHandler.HandleListSources)
r.Post("/autoscan/sources", autoscanHandler.HandleCreateSource)
r.Put("/autoscan/sources/{id}", autoscanHandler.HandleUpdateSource)
r.Delete("/autoscan/sources/{id}", autoscanHandler.HandleDeleteSource)
r.Get("/autoscan/sources/{id}/rewrite-suggestions", autoscanHandler.HandleRewriteSuggestions)
r.Get("/autoscan/scans", autoscanHandler.HandleListScans)
r.Get("/autoscan/events", autoscanHandler.HandleListEvents)
r.Post("/autoscan/trigger", autoscanHandler.HandleTrigger)
r.Get("/autoscan/status", autoscanHandler.HandleStatus)
}
if deps.ActivityLogRepo != nil {
adminIPHandler := handlers.NewAdminIPHandler(deps.ActivityLogRepo)
r.Get("/users/{id}/ips", adminIPHandler.HandleGetUserIPs)
internal/autoscan/connection.go
+66
View File
@@ -0,0 +1,66 @@
package autoscan
import (
"context"
"fmt"
"strings"
)
// ResolvedConnection is concrete credentials handed to the plugin.
type ResolvedConnection struct {
BaseURL string
APIKey string
}
// RequestIntegrationLookup resolves a soft-linked Requests integration to its
// base URL and encrypted api-key reference.
type RequestIntegrationLookup interface {
Get(ctx context.Context, integrationID string) (baseURL, apiKeyRef string, err error)
}
// SecretResolver resolves an encrypted api-key reference to its plaintext value.
type SecretResolver interface {
Get(ctx context.Context, ref string) (string, error)
}
type ConnectionResolver struct {
requests RequestIntegrationLookup
secrets SecretResolver
}
func NewConnectionResolver(r RequestIntegrationLookup, s SecretResolver) *ConnectionResolver {
return &ConnectionResolver{requests: r, secrets: s}
}
// Resolve turns a stored Connection into concrete credentials. When the
// connection is linked to a Requests integration the live base URL + key ref are
// read from there; otherwise the connection's own fields are used. The api-key
// ref is then resolved to plaintext via the secrets resolver.
func (cr *ConnectionResolver) Resolve(ctx context.Context, c Connection) (ResolvedConnection, error) {
baseURL, apiKeyRef := c.BaseURL, c.APIKeyRef
// A pointer-to-empty-string request_integration_id is NOT a live link (it can
// arise from a both-NULL orphan or a stripped link). Treat empty/whitespace as
// "no link" so we don't call requests.Get("") and use the connection's own
// fields instead.
if c.RequestIntegrationID != nil && strings.TrimSpace(*c.RequestIntegrationID) != "" {
if cr.requests == nil {
return ResolvedConnection{}, fmt.Errorf("autoscan: linked requests integration %q: no lookup configured", *c.RequestIntegrationID)
}
u, ref, err := cr.requests.Get(ctx, *c.RequestIntegrationID)
if err != nil {
return ResolvedConnection{}, fmt.Errorf("autoscan: linked requests integration %q: %w", *c.RequestIntegrationID, err)
}
baseURL, apiKeyRef = u, ref
}
apiKey := strings.TrimSpace(apiKeyRef)
if cr.secrets != nil && apiKey != "" {
resolved, err := cr.secrets.Get(ctx, apiKey)
if err != nil {
return ResolvedConnection{}, fmt.Errorf("autoscan: resolve api key: %w", err)
}
if resolved = strings.TrimSpace(resolved); resolved != "" {
apiKey = resolved
}
}
return ResolvedConnection{BaseURL: baseURL, APIKey: apiKey}, nil
}
internal/autoscan/connection_test.go
+129
View File
@@ -0,0 +1,129 @@
package autoscan
import (
"context"
"errors"
"testing"
)
// fakeRequestLookup implements RequestIntegrationLookup from a static map.
type fakeRequestLookup struct {
entries map[string]struct{ baseURL, apiKeyRef string }
err error
}
func (f fakeRequestLookup) Get(_ context.Context, id string) (string, string, error) {
if f.err != nil {
return "", "", f.err
}
e, ok := f.entries[id]
if !ok {
return "", "", errors.New("integration not found: " + id)
}
return e.baseURL, e.apiKeyRef, nil
}
// fakeSecrets implements SecretResolver from a static map; unknown refs resolve
// to empty (so the ref itself is used).
type fakeSecrets struct {
values map[string]string
err error
}
func (f fakeSecrets) Get(_ context.Context, ref string) (string, error) {
if f.err != nil {
return "", f.err
}
return f.values[ref], nil
}
func TestResolveConnectionOwnCredentials(t *testing.T) {
secrets := fakeSecrets{values: map[string]string{"ownref": "OWNKEY"}}
r := NewConnectionResolver(fakeRequestLookup{}, secrets)
got, err := r.Resolve(context.Background(), Connection{
BaseURL: "http://own:8989",
APIKeyRef: "ownref",
})
if err != nil {
t.Fatalf("Resolve own: %v", err)
}
if got.BaseURL != "http://own:8989" || got.APIKey != "OWNKEY" {
t.Fatalf("own resolve = %+v", got)
}
}
func TestResolveConnectionLinked(t *testing.T) {
lookup := fakeRequestLookup{entries: map[string]struct{ baseURL, apiKeyRef string }{
"req-1": {baseURL: "http://req:7878", apiKeyRef: "rk"},
}}
secrets := fakeSecrets{values: map[string]string{"rk": "REQKEY"}}
r := NewConnectionResolver(lookup, secrets)
id := "req-1"
got, err := r.Resolve(context.Background(), Connection{RequestIntegrationID: &id})
if err != nil {
t.Fatalf("Resolve linked: %v", err)
}
if got.BaseURL != "http://req:7878" || got.APIKey != "REQKEY" {
t.Fatalf("linked resolve = %+v", got)
}
}
func TestResolveConnectionLinkedMissingErrors(t *testing.T) {
r := NewConnectionResolver(fakeRequestLookup{}, fakeSecrets{})
id := "gone"
if _, err := r.Resolve(context.Background(), Connection{RequestIntegrationID: &id}); err == nil {
t.Fatal("expected error when linked Requests integration is missing")
}
}
func TestResolveConnectionLinkedLookupErrors(t *testing.T) {
r := NewConnectionResolver(fakeRequestLookup{err: errors.New("boom")}, fakeSecrets{})
id := "req-1"
if _, err := r.Resolve(context.Background(), Connection{RequestIntegrationID: &id}); err == nil {
t.Fatal("expected error when the lookup itself fails")
}
}
func TestResolveConnectionEmptyIntegrationIDIsNotALink(t *testing.T) {
// A pointer-to-empty/whitespace request_integration_id must NOT be treated as
// a live link: it must fall back to the connection's own fields, never call
// requests.Get(""). The lookup here errors on any call, so reaching it fails.
r := NewConnectionResolver(
fakeRequestLookup{err: errors.New("lookup must not be called")},
fakeSecrets{values: map[string]string{"ownref": "OWNKEY"}},
)
for _, empty := range []string{"", " "} {
id := empty
got, err := r.Resolve(context.Background(), Connection{
BaseURL: "http://own:8989",
APIKeyRef: "ownref",
RequestIntegrationID: &id,
})
if err != nil {
t.Fatalf("Resolve with empty integration id %q: %v", empty, err)
}
if got.BaseURL != "http://own:8989" || got.APIKey != "OWNKEY" {
t.Fatalf("empty integration id %q should use own creds, got %+v", empty, got)
}
}
}
func TestResolveConnectionTrimsAndFallsBackOnEmptySecret(t *testing.T) {
// Whitespace-padded ref; secret resolves to whitespace-only -> fall back to
// the trimmed ref (parity with requests.resolveAPIKey).
secrets := fakeSecrets{values: map[string]string{"ownref": " "}}
r := NewConnectionResolver(fakeRequestLookup{}, secrets)
got, err := r.Resolve(context.Background(), Connection{
BaseURL: "http://own:8989",
APIKeyRef: " ownref ",
})
if err != nil {
t.Fatalf("Resolve: %v", err)
}
if got.APIKey != "ownref" {
t.Fatalf("expected trimmed ref fallback, got %q", got.APIKey)
}
}
internal/autoscan/dedupe.go
+22
View File
@@ -0,0 +1,22 @@
package autoscan
import "path/filepath"
// uniqueParentDirs maps imported file paths to their distinct parent directories,
// dropping empties. A season's episodes in one folder collapse to one entry.
func uniqueParentDirs(paths []string) []string {
seen := make(map[string]struct{})
var out []string
for _, p := range paths {
if p == "" {
continue
}
dir := filepath.Dir(p)
if _, ok := seen[dir]; ok {
continue
}
seen[dir] = struct{}{}
out = append(out, dir)
}
return out
}
internal/autoscan/dedupe_test.go
+22
View File
@@ -0,0 +1,22 @@
package autoscan
import (
"reflect"
"sort"
"testing"
)
func TestUniqueParentDirs(t *testing.T) {
in := []string{
"/mnt/media/Show/Season 01/E01.mkv",
"/mnt/media/Show/Season 01/E02.mkv",
"/mnt/media/Movie/Movie.mkv",
"",
}
got := uniqueParentDirs(in)
sort.Strings(got)
want := []string{"/mnt/media/Movie", "/mnt/media/Show/Season 01"}
if !reflect.DeepEqual(got, want) {
t.Fatalf("uniqueParentDirs = %v, want %v", got, want)
}
}
internal/autoscan/discovery.go
+88
View File
@@ -0,0 +1,88 @@
package autoscan
import (
"context"
"fmt"
)
// DiscoveredSource identifies one installed scan_source.v1 capability instance,
// enriched with the metadata the Add-source picker needs (plugin id + a
// human-friendly display name).
type DiscoveredSource struct {
InstallationID int
CapabilityID string
// PluginID is the installation's plugin id (e.g. "sonarr"); empty when the
// lister cannot supply it.
PluginID string
// DisplayName is a human-friendly label for the capability (from the
// capability's manifest display_name, falling back to plugin/capability ids).
DisplayName string
}
// ScanSourceLister enumerates every installed scan_source.v1 capability so the
// engine can (a) offer them in the Add-source picker and (b) detect orphaned
// source rows whose plugin has been uninstalled.
type ScanSourceLister interface {
// ListScanSources returns one entry per installed scan_source.v1 capability,
// enriched with plugin id + display name.
ListScanSources(ctx context.Context) ([]DiscoveredSource, error)
}
// AvailableScanSource is one installed scan_source capability an operator can
// create a source against (the Add-source picker list).
type AvailableScanSource struct {
InstallationID int `json:"installation_id"`
CapabilityID string `json:"capability_id"`
PluginID string `json:"plugin_id"`
DisplayName string `json:"display_name"`
}
// installedKey identifies an installed scan_source capability for set
// membership tests (orphan detection in PollOnce).
type installedKey struct {
InstallationID int
CapabilityID string
}
// ListAvailableScanSources enumerates every installed scan_source capability so
// an operator can pick one when creating a source. When no lister is configured
// it returns an empty list. The handler also uses this to validate that a
// create request targets a currently-installed capability.
func (s *Service) ListAvailableScanSources(ctx context.Context) ([]AvailableScanSource, error) {
if s.lister == nil {
return []AvailableScanSource{}, nil
}
discovered, err := s.lister.ListScanSources(ctx)
if err != nil {
return nil, fmt.Errorf("list scan sources: %w", err)
}
out := make([]AvailableScanSource, 0, len(discovered))
for _, d := range discovered {
out = append(out, AvailableScanSource{
InstallationID: d.InstallationID,
CapabilityID: d.CapabilityID,
PluginID: d.PluginID,
DisplayName: d.DisplayName,
})
}
return out, nil
}
// installedScanSources returns the set of currently-installed scan_source
// capabilities so PollOnce can skip orphaned source rows (their plugin is
// gone). A nil set means the set is unavailable (no lister) — callers must treat
// that as "discovery unavailable" and NOT prune.
func (s *Service) installedScanSources(ctx context.Context) (map[installedKey]struct{}, error) {
if s.lister == nil {
return nil, nil
}
discovered, err := s.lister.ListScanSources(ctx)
if err != nil {
return nil, fmt.Errorf("list scan sources: %w", err)
}
present := make(map[installedKey]struct{}, len(discovered))
for _, d := range discovered {
present[installedKey{InstallationID: d.InstallationID, CapabilityID: d.CapabilityID}] = struct{}{}
}
return present, nil
}
internal/autoscan/discovery_test.go
+72
View File
@@ -0,0 +1,72 @@
package autoscan
import (
"context"
"testing"
)
// fakeLister returns a fixed set of discovered scan sources.
type fakeLister struct{ sources []DiscoveredSource }
func (f fakeLister) ListScanSources(context.Context) ([]DiscoveredSource, error) {
return f.sources, nil
}
func TestListAvailableScanSourcesEnumeratesInstalled(t *testing.T) {
lister := fakeLister{sources: []DiscoveredSource{
{InstallationID: 1, CapabilityID: "arr-a", PluginID: "sonarr", DisplayName: "Sonarr"},
{InstallationID: 2, CapabilityID: "arr-b", PluginID: "radarr", DisplayName: "Radarr"},
}}
svc := &Service{lister: lister}
available, err := svc.ListAvailableScanSources(context.Background())
if err != nil {
t.Fatalf("ListAvailableScanSources: %v", err)
}
if len(available) != 2 {
t.Fatalf("expected 2 available, got %d: %+v", len(available), available)
}
if available[0] != (AvailableScanSource{InstallationID: 1, CapabilityID: "arr-a", PluginID: "sonarr", DisplayName: "Sonarr"}) {
t.Fatalf("unexpected first available: %+v", available[0])
}
}
func TestListAvailableScanSourcesNilListerEmpty(t *testing.T) {
svc := &Service{lister: nil}
available, err := svc.ListAvailableScanSources(context.Background())
if err != nil {
t.Fatalf("ListAvailableScanSources: %v", err)
}
if len(available) != 0 {
t.Fatalf("nil lister must return empty, got %+v", available)
}
}
func TestInstalledScanSourcesSetMembership(t *testing.T) {
lister := fakeLister{sources: []DiscoveredSource{
{InstallationID: 1, CapabilityID: "arr-a"},
}}
svc := &Service{lister: lister}
present, err := svc.installedScanSources(context.Background())
if err != nil {
t.Fatalf("installedScanSources: %v", err)
}
if _, ok := present[installedKey{1, "arr-a"}]; !ok {
t.Fatalf("expected 1/arr-a present: %+v", present)
}
if _, ok := present[installedKey{2, "arr-b"}]; ok {
t.Fatalf("did not expect 2/arr-b present: %+v", present)
}
}
func TestInstalledScanSourcesNilListerNilSet(t *testing.T) {
svc := &Service{lister: nil}
present, err := svc.installedScanSources(context.Background())
if err != nil {
t.Fatalf("installedScanSources: %v", err)
}
if present != nil {
t.Fatalf("nil lister must return a nil set, got %+v", present)
}
}
internal/autoscan/errors.go
+7
View File
@@ -0,0 +1,7 @@
package autoscan
import "errors"
// ErrNotFound is returned when an autoscan operation references a connection or
// source that does not exist. The admin API surfaces it as a 404.
var ErrNotFound = errors.New("autoscan: not found")
internal/autoscan/labels.go
+20
View File
@@ -0,0 +1,20 @@
package autoscan
import (
"strings"
"unicode/utf8"
)
// MaxSourceLabelLen bounds an operator-set source label in runes.
const MaxSourceLabelLen = 120
// NormalizeSourceLabel trims surrounding whitespace and caps the label at
// MaxSourceLabelLen runes without splitting a multi-byte rune. An all-whitespace
// label normalizes to "" (unset).
func NormalizeSourceLabel(s string) string {
s = strings.TrimSpace(s)
if utf8.RuneCountInString(s) <= MaxSourceLabelLen {
return s
}
return string([]rune(s)[:MaxSourceLabelLen])
}
internal/autoscan/labels_test.go
+39
View File
@@ -0,0 +1,39 @@
package autoscan
import (
"strings"
"testing"
"unicode/utf8"
)
func TestNormalizeSourceLabel(t *testing.T) {
cases := []struct {
name string
in string
want string
}{
{"trims surrounding whitespace", " 4K Movies ", "4K Movies"},
{"empty stays empty", "", ""},
{"all whitespace becomes empty", " ", ""},
{"under cap unchanged", strings.Repeat("x", 120), strings.Repeat("x", 120)},
{"over cap truncated to 120 runes", strings.Repeat("x", 200), strings.Repeat("x", 120)},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
if got := NormalizeSourceLabel(c.in); got != c.want {
t.Fatalf("NormalizeSourceLabel(%q) = %q, want %q", c.in, got, c.want)
}
})
}
}
func TestNormalizeSourceLabelRuneSafe(t *testing.T) {
// 200 multi-byte runes must cap to 120 runes and stay valid UTF-8.
got := NormalizeSourceLabel(strings.Repeat("é", 200))
if utf8.RuneCountInString(got) != 120 {
t.Fatalf("rune count = %d, want 120", utf8.RuneCountInString(got))
}
if !utf8.ValidString(got) {
t.Fatal("result is not valid UTF-8")
}
}
internal/autoscan/probe.go
+54
View File
@@ -0,0 +1,54 @@
package autoscan
import (
"context"
"fmt"
)
// ArrStatusProbe probes an arr instance's /api/v3/system/status to confirm the
// base URL + api key are reachable and valid. It returns the reported version
// on success.
type ArrStatusProbe interface {
SystemStatus(ctx context.Context, baseURL, apiKey string) (version string, err error)
}
// ConnectionTestResult is the outcome of probing a connection's arr endpoint.
type ConnectionTestResult struct {
OK bool
Version string
Err string
}
// TestConnection resolves the given connection to concrete credentials and
// probes its arr /api/v3/system/status. A resolve failure or an unreachable /
// unauthorized / non-200 endpoint yields OK=false with a human-readable Err
// (never an HTTP error from this method itself — probe failures are part of the
// successful response payload). A nil probe dependency is a wiring error and is
// returned as a real error.
func (s *Service) TestConnection(ctx context.Context, c Connection) (ConnectionTestResult, error) {
if s.rootFolders == nil {
return ConnectionTestResult{}, fmt.Errorf("autoscan: connection probe not configured")
}
probe, ok := s.rootFolders.(ArrStatusProbe)
if !ok {
return ConnectionTestResult{}, fmt.Errorf("autoscan: connection probe not configured")
}
resolved, err := s.connres.Resolve(ctx, c)
if err != nil {
return ConnectionTestResult{OK: false, Err: err.Error()}, nil
}
version, err := probe.SystemStatus(ctx, resolved.BaseURL, resolved.APIKey)
if err != nil {
return ConnectionTestResult{OK: false, Err: err.Error()}, nil
}
return ConnectionTestResult{OK: true, Version: version}, nil
}
// TestConnectionByID loads a stored connection by id and tests it.
func (s *Service) TestConnectionByID(ctx context.Context, id string) (ConnectionTestResult, error) {
c, err := s.store.GetConnection(ctx, id)
if err != nil {
return ConnectionTestResult{}, err
}
return s.TestConnection(ctx, c)
}
internal/autoscan/probe_test.go
+97
View File
@@ -0,0 +1,97 @@
package autoscan
import (
"context"
"errors"
"testing"
)
// fakeArrClient implements both RootFolderClient and ArrStatusProbe for the
// connection-test and suggester service tests.
type fakeArrClient struct {
roots []string
rootsErr error
version string
statusErr error
gotBaseURL string
gotAPIKey string
}
func (f *fakeArrClient) RootFolders(_ context.Context, baseURL, apiKey string) ([]string, error) {
f.gotBaseURL, f.gotAPIKey = baseURL, apiKey
return f.roots, f.rootsErr
}
func (f *fakeArrClient) SystemStatus(_ context.Context, baseURL, apiKey string) (string, error) {
f.gotBaseURL, f.gotAPIKey = baseURL, apiKey
return f.version, f.statusErr
}
type fakeFolderLister struct{ paths []string }
func (f fakeFolderLister) ListFolderPaths(context.Context) ([]string, error) {
return f.paths, nil
}
func TestTestConnectionOK(t *testing.T) {
arr := &fakeArrClient{version: "4.1.2.3"}
svc := &Service{connres: passthroughConnRes{}, rootFolders: arr}
res, err := svc.TestConnection(context.Background(), Connection{BaseURL: "http://radarr:7878", APIKeyRef: "k"})
if err != nil {
t.Fatalf("TestConnection: %v", err)
}
if !res.OK || res.Version != "4.1.2.3" {
t.Fatalf("expected ok with version, got %+v", res)
}
}
func TestTestConnectionProbeFailureIsPayload(t *testing.T) {
arr := &fakeArrClient{statusErr: errors.New("arr: HTTP 401")}
svc := &Service{connres: passthroughConnRes{}, rootFolders: arr}
res, err := svc.TestConnection(context.Background(), Connection{BaseURL: "http://radarr:7878"})
if err != nil {
t.Fatalf("probe failure must not be a method error: %v", err)
}
if res.OK || res.Err == "" {
t.Fatalf("expected ok=false with error, got %+v", res)
}
}
func TestTestConnectionNoProbeConfigured(t *testing.T) {
svc := &Service{connres: passthroughConnRes{}}
if _, err := svc.TestConnection(context.Background(), Connection{BaseURL: "http://x"}); err == nil {
t.Fatal("expected error when no probe configured")
}
}
func TestSuggestRewritesMatchesArrRootsToSiloFolders(t *testing.T) {
store := &fakeStore{
sources: []Source{
{ID: "src-1", InstallationID: 1, CapabilityID: "arr", ConnectionID: strptr("c1")},
},
}
arr := &fakeArrClient{roots: []string{"/data/tv"}}
folders := fakeFolderLister{paths: []string{"/mnt/media/tv"}}
svc := &Service{store: store, connres: passthroughConnRes{}, rootFolders: arr, folders: folders}
sugg, err := svc.SuggestRewrites(context.Background(), "src-1")
if err != nil {
t.Fatalf("SuggestRewrites: %v", err)
}
if len(sugg.Proposed) != 1 || sugg.Proposed[0].From != "/data/tv" || sugg.Proposed[0].To != "/mnt/media/tv" {
t.Fatalf("unexpected proposals: %+v", sugg.Proposed)
}
}
func TestSuggestRewritesNoConnection(t *testing.T) {
store := &fakeStore{
sources: []Source{{ID: "src-1", InstallationID: 1, CapabilityID: "arr", ConnectionID: nil}},
}
svc := &Service{store: store, connres: passthroughConnRes{}, rootFolders: &fakeArrClient{}, folders: fakeFolderLister{}}
if _, err := svc.SuggestRewrites(context.Background(), "src-1"); !errors.Is(err, ErrNoConnection) {
t.Fatalf("expected ErrNoConnection, got %v", err)
}
}
internal/autoscan/provider.go
+83
View File
@@ -0,0 +1,83 @@
package autoscan
import (
"context"
pluginv1 "github.com/Silo-Server/silo-plugin-sdk/pkg/pluginproto/silo/plugin/v1"
)
// ScanSourceProvider yields changed paths for one source. The engine calls
// PollChanges; production wraps the plugins.Service scan_source resolver.
type ScanSourceProvider interface {
PollChanges(ctx context.Context, installationID int, capabilityID, marker string, conn ResolvedConnection, sourceConfig map[string]string) (changes []Change, nextMarker string, err error)
}
// PollChangesClient is the slice of *pluginhost.ScanSourceClient used here. It
// is exported so wiring in the api package can declare an adapter whose
// ScanSourceClient returns it (Go interface method signatures must match
// exactly across packages, and the unexported form could not be named there).
type PollChangesClient interface {
PollChanges(ctx context.Context, req *pluginv1.PollChangesRequest) (*pluginv1.PollChangesResponse, error)
}
// ScanSourceResolver yields a per-(installation, capability) scan-source client.
// Exported for the same cross-package adapter reason as PollChangesClient.
type ScanSourceResolver interface {
ScanSourceClient(ctx context.Context, installationID int, capabilityID string) (PollChangesClient, error)
}
type pluginProvider struct{ resolver ScanSourceResolver }
// NewPluginProvider builds the production scan-source provider over the plugins
// resolver.
func NewPluginProvider(resolver ScanSourceResolver) ScanSourceProvider {
return &pluginProvider{resolver: resolver}
}
func (p *pluginProvider) PollChanges(ctx context.Context, installationID int, capabilityID, marker string, conn ResolvedConnection, sourceConfig map[string]string) ([]Change, string, error) {
client, err := p.resolver.ScanSourceClient(ctx, installationID, capabilityID)
if err != nil {
return nil, "", err
}
resp, err := client.PollChanges(ctx, &pluginv1.PollChangesRequest{
CapabilityId: capabilityID,
Marker: marker,
Connection: &pluginv1.ResolvedConnection{BaseUrl: conn.BaseURL, ApiKey: conn.APIKey},
SourceConfig: sourceConfig,
})
if err != nil {
return nil, "", err
}
if structured := resp.GetChanges(); len(structured) > 0 {
changes := make([]Change, 0, len(structured))
for _, change := range structured {
if change == nil {
continue
}
changes = append(changes, Change{
SourcePath: change.GetSourcePath(),
Scope: scanSourceScope(change.GetScope()),
})
}
return changes, resp.GetNextMarker(), nil
}
// Legacy plugins return only source_paths. Treat them as auto/file-like
// paths so the existing parent-directory collapse behavior is preserved.
changes := make([]Change, 0, len(resp.GetSourcePaths()))
for _, path := range resp.GetSourcePaths() {
changes = append(changes, Change{SourcePath: path, Scope: ChangeScopeAuto})
}
return changes, resp.GetNextMarker(), nil
}
func scanSourceScope(scope pluginv1.ScanSourceChangeScope) ChangeScope {
switch scope {
case pluginv1.ScanSourceChangeScope_SCAN_SOURCE_CHANGE_SCOPE_FILE:
return ChangeScopeFile
case pluginv1.ScanSourceChangeScope_SCAN_SOURCE_CHANGE_SCOPE_SUBTREE:
return ChangeScopeSubtree
default:
return ChangeScopeAuto
}
}
internal/autoscan/provider_test.go
+99
View File
@@ -0,0 +1,99 @@
package autoscan
import (
"context"
"testing"
pluginv1 "github.com/Silo-Server/silo-plugin-sdk/pkg/pluginproto/silo/plugin/v1"
)
// capturingClient records the last PollChangesRequest it was handed.
type capturingClient struct {
last *pluginv1.PollChangesRequest
}
func (c *capturingClient) PollChanges(_ context.Context, req *pluginv1.PollChangesRequest) (*pluginv1.PollChangesResponse, error) {
c.last = req
return &pluginv1.PollChangesResponse{SourcePaths: []string{"/mnt/media/x"}, NextMarker: "m2"}, nil
}
// capturingResolver yields a fixed PollChangesClient.
type capturingResolver struct{ client PollChangesClient }
func (r capturingResolver) ScanSourceClient(context.Context, int, string) (PollChangesClient, error) {
return r.client, nil
}
// TestPluginProviderPopulatesConnection asserts the resolved connection is
// delivered to the plugin on the PollChangesRequest.
func TestPluginProviderPopulatesConnection(t *testing.T) {
client := &capturingClient{}
prov := NewPluginProvider(capturingResolver{client})
conn := ResolvedConnection{BaseURL: "https://arr.example", APIKey: "secret-key"}
sourceConfig := map[string]string{"exclusions": ".downloads"}
changes, next, err := prov.PollChanges(context.Background(), 1, "cap", "m1", conn, sourceConfig)
if err != nil {
t.Fatalf("PollChanges: %v", err)
}
if len(changes) != 1 || changes[0].SourcePath != "/mnt/media/x" || changes[0].Scope != ChangeScopeAuto || next != "m2" {
t.Fatalf("unexpected provider result: changes=%v next=%q", changes, next)
}
if client.last == nil {
t.Fatal("expected a PollChangesRequest to be sent")
}
if client.last.GetCapabilityId() != "cap" || client.last.GetMarker() != "m1" {
t.Fatalf("unexpected request fields: cap=%q marker=%q", client.last.GetCapabilityId(), client.last.GetMarker())
}
rc := client.last.GetConnection()
if rc == nil {
t.Fatal("expected connection to be populated on the request")
}
if rc.GetBaseUrl() != conn.BaseURL || rc.GetApiKey() != conn.APIKey {
t.Fatalf("connection not delivered: base_url=%q api_key=%q", rc.GetBaseUrl(), rc.GetApiKey())
}
if client.last.GetSourceConfig()["exclusions"] != ".downloads" {
t.Fatalf("source_config not delivered: %#v", client.last.GetSourceConfig())
}
}
func TestPluginProviderPrefersStructuredChanges(t *testing.T) {
client := &capturingStructuredClient{}
prov := NewPluginProvider(capturingResolver{client: client})
changes, next, err := prov.PollChanges(context.Background(), 1, "cap", "m1", ResolvedConnection{}, nil)
if err != nil {
t.Fatalf("PollChanges: %v", err)
}
if next != "m3" || len(changes) != 2 {
t.Fatalf("unexpected provider result: changes=%v next=%q", changes, next)
}
if changes[0] != (Change{SourcePath: "/ceph/movie/Movie", Scope: ChangeScopeSubtree}) {
t.Fatalf("first change = %#v", changes[0])
}
if changes[1] != (Change{SourcePath: "/ceph/show/S01/E01.mkv", Scope: ChangeScopeFile}) {
t.Fatalf("second change = %#v", changes[1])
}
}
type capturingStructuredClient struct {
last *pluginv1.PollChangesRequest
}
func (c *capturingStructuredClient) PollChanges(_ context.Context, req *pluginv1.PollChangesRequest) (*pluginv1.PollChangesResponse, error) {
c.last = req
return &pluginv1.PollChangesResponse{
SourcePaths: []string{"/legacy/ignored.mkv"},
NextMarker: "m3",
Changes: []*pluginv1.ScanSourceChange{
{
SourcePath: "/ceph/movie/Movie",
Scope: pluginv1.ScanSourceChangeScope_SCAN_SOURCE_CHANGE_SCOPE_SUBTREE,
},
{
SourcePath: "/ceph/show/S01/E01.mkv",
Scope: pluginv1.ScanSourceChangeScope_SCAN_SOURCE_CHANGE_SCOPE_FILE,
},
},
}, nil
}
internal/autoscan/repository.go
+861
View File
@@ -0,0 +1,861 @@
package autoscan
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"unicode/utf8"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgconn"
"github.com/jackc/pgx/v5/pgxpool"
)
type Repository struct{ pool *pgxpool.Pool }
func NewRepository(pool *pgxpool.Pool) *Repository { return &Repository{pool: pool} }
// --- Settings ---
func (r *Repository) GetSettings(ctx context.Context) (Settings, error) {
var s Settings
err := r.pool.QueryRow(ctx, `
SELECT enabled, default_poll_interval_seconds, debounce_seconds
FROM autoscan_settings WHERE id = true`).
Scan(&s.Enabled, &s.DefaultPollIntervalSeconds, &s.DebounceSeconds)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return Settings{
Enabled: false,
DefaultPollIntervalSeconds: 600,
DebounceSeconds: 60,
}, nil
}
return Settings{}, fmt.Errorf("get autoscan settings: %w", err)
}
return s, nil
}
func (r *Repository) UpdateSettings(ctx context.Context, s Settings) (Settings, error) {
var out Settings
err := r.pool.QueryRow(ctx, `
INSERT INTO autoscan_settings (
id, enabled, default_poll_interval_seconds, debounce_seconds, updated_at
)
VALUES (true, $1, $2, $3, now())
ON CONFLICT (id) DO UPDATE SET
enabled = EXCLUDED.enabled,
default_poll_interval_seconds = EXCLUDED.default_poll_interval_seconds,
debounce_seconds = EXCLUDED.debounce_seconds,
updated_at = now()
RETURNING enabled, default_poll_interval_seconds, debounce_seconds`,
s.Enabled, s.DefaultPollIntervalSeconds, s.DebounceSeconds).
Scan(&out.Enabled, &out.DefaultPollIntervalSeconds, &out.DebounceSeconds)
if err != nil {
return Settings{}, fmt.Errorf("update autoscan settings: %w", err)
}
return out, nil
}
// --- Connections ---
const connectionColumns = `id, name, kind, base_url, api_key_ref, request_integration_id`
func scanConnection(row interface{ Scan(...any) error }) (Connection, error) {
var c Connection
var baseURL, apiKeyRef, reqIntegrationID *string
if err := row.Scan(&c.ID, &c.Name, &c.Kind, &baseURL, &apiKeyRef, &reqIntegrationID); err != nil {
return Connection{}, err
}
if baseURL != nil {
c.BaseURL = *baseURL
}
if apiKeyRef != nil {
c.APIKeyRef = *apiKeyRef
}
c.RequestIntegrationID = reqIntegrationID
return c, nil
}
// nullable returns nil for empty strings so they map to SQL NULL.
func nullable(s string) any {
if s == "" {
return nil
}
return s
}
func (r *Repository) CreateConnection(ctx context.Context, c Connection) (Connection, error) {
row := r.pool.QueryRow(ctx, `
INSERT INTO autoscan_connections (name, kind, base_url, api_key_ref, request_integration_id)
VALUES ($1, $2, $3, $4, $5)
RETURNING `+connectionColumns,
c.Name, c.Kind, nullable(c.BaseURL), nullable(c.APIKeyRef), c.RequestIntegrationID)
out, err := scanConnection(row)
if err != nil {
return Connection{}, fmt.Errorf("create autoscan connection: %w", err)
}
return out, nil
}
func (r *Repository) UpdateConnection(ctx context.Context, c Connection) (Connection, error) {
// A blank incoming api_key_ref KEEPS the existing stored value: the UI
// deliberately omits the key on a metadata-only edit ("leave blank to keep
// existing"), so unconditionally writing it would NULL the key and break the
// next poll. Mirrors requests.UpdateIntegration's CASE-WHEN keep-semantics.
// Pass the raw trimmed string (not nullable()) so the empty-string sentinel
// reaches the CASE.
row := r.pool.QueryRow(ctx, `
UPDATE autoscan_connections
SET name = $2, kind = $3, base_url = $4,
api_key_ref = CASE WHEN $5 = '' THEN api_key_ref ELSE $5 END,
request_integration_id = $6, updated_at = now()
WHERE id = $1
RETURNING `+connectionColumns,
c.ID, c.Name, c.Kind, nullable(c.BaseURL), c.APIKeyRef, c.RequestIntegrationID)
out, err := scanConnection(row)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return Connection{}, fmt.Errorf("%w: connection %s", ErrNotFound, c.ID)
}
return Connection{}, fmt.Errorf("update autoscan connection: %w", err)
}
return out, nil
}
func (r *Repository) DeleteConnection(ctx context.Context, id string) error {
tag, err := r.pool.Exec(ctx, `DELETE FROM autoscan_connections WHERE id = $1`, id)
if err != nil {
// A source still references this connection (ON DELETE RESTRICT, 23503).
var pgErr *pgconn.PgError
if errors.As(err, &pgErr) && pgErr.Code == "23503" {
return fmt.Errorf("autoscan: connection %s is in use by a source", id)
}
return fmt.Errorf("delete autoscan connection: %w", err)
}
if tag.RowsAffected() == 0 {
return fmt.Errorf("%w: connection %s", ErrNotFound, id)
}
return nil
}
func (r *Repository) ListConnections(ctx context.Context) ([]Connection, error) {
rows, err := r.pool.Query(ctx, `SELECT `+connectionColumns+`
FROM autoscan_connections ORDER BY name`)
if err != nil {
return nil, fmt.Errorf("list autoscan connections: %w", err)
}
defer rows.Close()
var out []Connection
for rows.Next() {
c, err := scanConnection(rows)
if err != nil {
return nil, err
}
out = append(out, c)
}
return out, rows.Err()
}
func (r *Repository) GetConnection(ctx context.Context, id string) (Connection, error) {
row := r.pool.QueryRow(ctx, `SELECT `+connectionColumns+`
FROM autoscan_connections WHERE id = $1`, id)
c, err := scanConnection(row)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return Connection{}, fmt.Errorf("%w: connection %s", ErrNotFound, id)
}
return Connection{}, fmt.Errorf("get autoscan connection: %w", err)
}
return c, nil
}
// --- Sources ---
const sourceColumns = `id, installation_id, capability_id, connection_id, enabled,
poll_interval_seconds, path_rewrites, source_config, label, marker, last_run_at, last_error`
func scanSource(row interface{ Scan(...any) error }) (Source, error) {
var s Source
var pathRewrites []byte
var sourceConfig []byte
if err := row.Scan(&s.ID, &s.InstallationID, &s.CapabilityID, &s.ConnectionID,
&s.Enabled, &s.PollIntervalSeconds, &pathRewrites, &sourceConfig, &s.Label, &s.Marker, &s.LastRunAt, &s.LastError); err != nil {
return Source{}, err
}
rewrites, err := unmarshalPathRewrites(pathRewrites)
if err != nil {
return Source{}, err
}
s.PathRewrites = rewrites
config, err := unmarshalSourceConfig(sourceConfig)
if err != nil {
return Source{}, err
}
s.SourceConfig = config
return s, nil
}
// unmarshalPathRewrites decodes the jsonb path_rewrites column into a slice. A
// NULL/empty column maps to an empty (non-nil) slice so callers never see a nil.
func unmarshalPathRewrites(raw []byte) ([]PathRewrite, error) {
if len(raw) == 0 {
return []PathRewrite{}, nil
}
var out []PathRewrite
if err := json.Unmarshal(raw, &out); err != nil {
return nil, fmt.Errorf("decode autoscan path_rewrites: %w", err)
}
if out == nil {
out = []PathRewrite{}
}
return out, nil
}
// marshalPathRewrites encodes path rewrites for the jsonb column. A nil slice is
// stored as an empty JSON array (matching the column default '[]').
func marshalPathRewrites(rewrites []PathRewrite) ([]byte, error) {
if rewrites == nil {
rewrites = []PathRewrite{}
}
b, err := json.Marshal(rewrites)
if err != nil {
return nil, fmt.Errorf("encode autoscan path_rewrites: %w", err)
}
return b, nil
}
func unmarshalSourceConfig(raw []byte) (map[string]string, error) {
if len(raw) == 0 {
return map[string]string{}, nil
}
var out map[string]string
if err := json.Unmarshal(raw, &out); err != nil {
return nil, fmt.Errorf("decode autoscan source_config: %w", err)
}
if out == nil {
out = map[string]string{}
}
return out, nil
}
func marshalSourceConfig(config map[string]string) ([]byte, error) {
normalized := make(map[string]string, len(config))
for key, value := range config {
key = strings.TrimSpace(key)
if key == "" {
continue
}
normalized[key] = strings.TrimSpace(value)
}
b, err := json.Marshal(normalized)
if err != nil {
return nil, fmt.Errorf("encode autoscan source_config: %w", err)
}
return b, nil
}
// connectionIDArg maps a nullable connection id to a SQL value: nil pointer and
// whitespace-only ids map to SQL NULL.
func connectionIDArg(id *string) any {
if id == nil {
return nil
}
return nullable(*id)
}
// CreateSource inserts a new autoscan source row. One installed scan_source
// capability can back many sources (e.g. one Sonarr plugin install fronting 4
// arr servers, one source per connection), so this is a plain INSERT with a
// fresh uuid rather than an upsert. A non-existent connection trips the FK
// constraint and maps to ErrNotFound.
func (r *Repository) CreateSource(ctx context.Context, s Source) (Source, error) {
rewrites, err := marshalPathRewrites(s.PathRewrites)
if err != nil {
return Source{}, err
}
sourceConfig, err := marshalSourceConfig(s.SourceConfig)
if err != nil {
return Source{}, err
}
row := r.pool.QueryRow(ctx, `
INSERT INTO autoscan_sources (
installation_id, capability_id, connection_id, enabled, poll_interval_seconds, path_rewrites, source_config, label
)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
RETURNING `+sourceColumns,
s.InstallationID, s.CapabilityID, connectionIDArg(s.ConnectionID), s.Enabled, s.PollIntervalSeconds, rewrites, sourceConfig, s.Label)
out, err := scanSource(row)
if err != nil {
if connID, ok := connectionFKViolation(err, s.ConnectionID); ok {
return Source{}, fmt.Errorf("%w: connection %s", ErrNotFound, connID)
}
return Source{}, fmt.Errorf("create autoscan source: %w", err)
}
return out, nil
}
// UpdateSource updates a source's binding/scheduling fields by id. Identity
// (installation_id, capability_id) and bookkeeping fields (marker/last_run_at/
// last_error) are left untouched. An unknown id maps to ErrNotFound; a
// non-existent connection trips the FK constraint and also maps to ErrNotFound.
func (r *Repository) UpdateSource(ctx context.Context, s Source) (Source, error) {
rewrites, err := marshalPathRewrites(s.PathRewrites)
if err != nil {
return Source{}, err
}
sourceConfig, err := marshalSourceConfig(s.SourceConfig)
if err != nil {
return Source{}, err
}
row := r.pool.QueryRow(ctx, `
UPDATE autoscan_sources
SET connection_id = $2,
enabled = $3,
poll_interval_seconds = $4,
path_rewrites = $5,
source_config = $6,
label = $7,
updated_at = now()
WHERE id = $1
RETURNING `+sourceColumns,
s.ID, connectionIDArg(s.ConnectionID), s.Enabled, s.PollIntervalSeconds, rewrites, sourceConfig, s.Label)
out, err := scanSource(row)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return Source{}, fmt.Errorf("%w: source %s", ErrNotFound, s.ID)
}
if connID, ok := connectionFKViolation(err, s.ConnectionID); ok {
return Source{}, fmt.Errorf("%w: connection %s", ErrNotFound, connID)
}
return Source{}, fmt.Errorf("update autoscan source: %w", err)
}
return out, nil
}
// connectionFKViolation reports whether err is the connection FK constraint
// violation (23503), returning the offending connection id for the error
// message.
func connectionFKViolation(err error, connectionID *string) (string, bool) {
var pgErr *pgconn.PgError
if !errors.As(err, &pgErr) || pgErr.Code != "23503" {
return "", false
}
connID := ""
if connectionID != nil {
connID = *connectionID
}
return connID, true
}
func (r *Repository) ListSources(ctx context.Context) ([]Source, error) {
rows, err := r.pool.Query(ctx, `SELECT `+sourceColumns+`
FROM autoscan_sources ORDER BY installation_id, capability_id`)
if err != nil {
return nil, fmt.Errorf("list autoscan sources: %w", err)
}
defer rows.Close()
return collectSources(rows)
}
func (r *Repository) ListEnabledSources(ctx context.Context) ([]Source, error) {
rows, err := r.pool.Query(ctx, `SELECT `+sourceColumns+`
FROM autoscan_sources WHERE enabled = true
ORDER BY installation_id, capability_id`)
if err != nil {
return nil, fmt.Errorf("list enabled autoscan sources: %w", err)
}
defer rows.Close()
return collectSources(rows)
}
func collectSources(rows pgx.Rows) ([]Source, error) {
var out []Source
for rows.Next() {
s, err := scanSource(rows)
if err != nil {
return nil, err
}
out = append(out, s)
}
return out, rows.Err()
}
func (r *Repository) GetSource(ctx context.Context, id string) (Source, error) {
row := r.pool.QueryRow(ctx, `SELECT `+sourceColumns+`
FROM autoscan_sources WHERE id = $1`, id)
s, err := scanSource(row)
if err != nil {
if errors.Is(err, pgx.ErrNoRows) {
return Source{}, fmt.Errorf("%w: source %s", ErrNotFound, id)
}
return Source{}, fmt.Errorf("get autoscan source: %w", err)
}
return s, nil
}
// DeleteSource removes a source row by id. It lets an operator clear an orphaned
// source (one whose scan_source plugin was uninstalled/disabled). An unknown id
// maps to ErrNotFound.
func (r *Repository) DeleteSource(ctx context.Context, id string) error {
tag, err := r.pool.Exec(ctx, `DELETE FROM autoscan_sources WHERE id = $1`, id)
if err != nil {
return fmt.Errorf("delete autoscan source: %w", err)
}
if tag.RowsAffected() == 0 {
return fmt.Errorf("%w: source %s", ErrNotFound, id)
}
return nil
}
// AdvanceMarker stores the opaque next marker for a source, stamps last_run_at,
// and clears any prior error. Called only after a successful enqueue.
func (r *Repository) AdvanceMarker(ctx context.Context, sourceID, marker string) error {
tag, err := r.pool.Exec(ctx, `
UPDATE autoscan_sources
SET marker = $2, last_run_at = now(), last_error = NULL, updated_at = now()
WHERE id = $1`, sourceID, nullable(marker))
if err != nil {
return fmt.Errorf("advance autoscan marker: %w", err)
}
if tag.RowsAffected() == 0 {
return fmt.Errorf("%w: source %s", ErrNotFound, sourceID)
}
return nil
}
// maxLastErrorLen bounds the stored last_error (in bytes) so a pathological
// provider error can't bloat the row.
const maxLastErrorLen = 2048
// truncateUTF8 caps s to at most maxBytes bytes without splitting a multi-byte
// UTF-8 rune: if the byte cut lands mid-rune it backs off to the last valid
// rune boundary, so the stored value is always valid UTF-8.
func truncateUTF8(s string, maxBytes int) string {
if len(s) <= maxBytes {
return s
}
cut := maxBytes
for cut > 0 && !utf8.RuneStart(s[cut]) {
cut--
}
return s[:cut]
}
// RecordError records a poll failure for a source: it stamps last_run_at and
// stores the (length-bounded) error message without advancing the marker.
func (r *Repository) RecordError(ctx context.Context, sourceID, msg string) error {
msg = truncateUTF8(msg, maxLastErrorLen)
tag, err := r.pool.Exec(ctx, `
UPDATE autoscan_sources
SET last_error = $2, last_run_at = now(), updated_at = now()
WHERE id = $1`, sourceID, msg)
if err != nil {
return fmt.Errorf("record autoscan error: %w", err)
}
if tag.RowsAffected() == 0 {
return fmt.Errorf("%w: source %s", ErrNotFound, sourceID)
}
return nil
}
const eventColumns = `id, source_id, installation_id, capability_id, started_at, completed_at,
duration_ms, status, changes_returned, changes_resolved, targets_claimed, scans_created,
scans_reused, scans_suppressed, error_message, marker_before, marker_after`
func scanEvent(row interface{ Scan(...any) error }) (Event, error) {
var e Event
var status string
if err := row.Scan(
&e.ID,
&e.SourceID,
&e.InstallationID,
&e.CapabilityID,
&e.StartedAt,
&e.CompletedAt,
&e.DurationMS,
&status,
&e.ChangesReturned,
&e.ChangesResolved,
&e.TargetsClaimed,
&e.ScansCreated,
&e.ScansReused,
&e.ScansSuppressed,
&e.ErrorMessage,
&e.MarkerBefore,
&e.MarkerAfter,
); err != nil {
return Event{}, err
}
e.Status = EventStatus(status)
return e, nil
}
func (r *Repository) CreateEvent(ctx context.Context, in EventCreate) (int64, error) {
started := in.StartedAt
if started.IsZero() {
started = time.Now()
}
row := r.pool.QueryRow(ctx, `
INSERT INTO autoscan_events (
source_id, installation_id, capability_id, started_at, completed_at,
duration_ms, status, error_message, marker_before
)
VALUES ($1, $2, $3, $4, $4, 0, $5, $6, $7)
RETURNING id`,
nullable(in.SourceID),
in.InstallationID,
in.CapabilityID,
started,
string(EventStatusError),
"poll started but did not finish",
nullable(in.MarkerBefore),
)
var id int64
if err := row.Scan(&id); err != nil {
return 0, fmt.Errorf("create autoscan event: %w", err)
}
return id, nil
}
func (r *Repository) FinishEvent(ctx context.Context, in EventFinish) error {
if in.ID == 0 {
return nil
}
completed := in.CompletedAt
if completed.IsZero() {
completed = time.Now()
}
msg := truncateUTF8(in.ErrorMessage, maxLastErrorLen)
tag, err := r.pool.Exec(ctx, `
UPDATE autoscan_events
SET completed_at = $2,
duration_ms = GREATEST(0, EXTRACT(EPOCH FROM ($2 - started_at)) * 1000)::bigint,
status = $3,
changes_returned = $4,
changes_resolved = $5,
targets_claimed = $6,
scans_created = $7,
scans_reused = $8,
scans_suppressed = $9,
error_message = $10,
marker_after = $11
WHERE id = $1`,
in.ID,
completed,
string(in.Status),
in.ChangesReturned,
in.ChangesResolved,
in.TargetsClaimed,
in.ScansCreated,
in.ScansReused,
in.ScansSuppressed,
msg,
nullable(in.MarkerAfter),
)
if err != nil {
return fmt.Errorf("finish autoscan event: %w", err)
}
if tag.RowsAffected() == 0 {
return fmt.Errorf("%w: autoscan event %d", ErrNotFound, in.ID)
}
return nil
}
// clampAutoscanLimit bounds a requested page size to a sane window: the
// default page when unset, capped so a single query can never fan out.
func clampAutoscanLimit(limit int) int {
if limit <= 0 {
return 50
}
if limit > 200 {
return 200
}
return limit
}
// eventFilterClauses builds the shared WHERE clauses (and positional args) for
// autoscan event queries, so listing and counting filter identically and the
// search SQL lives in exactly one place.
func eventFilterClauses(filter EventListFilter) ([]string, []any) {
clauses := []string{"true"}
args := []any{}
if strings.TrimSpace(filter.SourceID) != "" {
args = append(args, strings.TrimSpace(filter.SourceID))
clauses = append(clauses, fmt.Sprintf("source_id = $%d", len(args)))
}
if filter.Status != "" {
args = append(args, string(filter.Status))
clauses = append(clauses, fmt.Sprintf("status = $%d", len(args)))
}
if search := strings.ToLower(strings.TrimSpace(filter.Search)); search != "" {
args = append(args, "%"+search+"%")
param := fmt.Sprintf("$%d", len(args))
clauses = append(clauses, `(
lower(capability_id) LIKE `+param+`
OR lower(status) LIKE `+param+`
OR lower(error_message) LIKE `+param+`
OR lower(COALESCE(source_id::text, '')) LIKE `+param+`
OR EXISTS (
SELECT 1
FROM scan_runs sr
WHERE sr.autoscan_event_id = autoscan_events.id
AND (
lower(sr.id) LIKE `+param+`
OR lower(sr.mode) LIKE `+param+`
OR lower(sr.path) LIKE `+param+`
OR lower(sr.status) LIKE `+param+`
OR lower(COALESCE(sr.error_message, '')) LIKE `+param+`
)
)
)`)
}
return clauses, args
}
// scanFilterClauses builds the shared WHERE clauses for autoscan scan queries.
// The clauses reference the `sr` (scan_runs) and `e` (autoscan_events) aliases,
// so callers must select FROM scan_runs sr LEFT JOIN autoscan_events e.
func scanFilterClauses(filter ScanListFilter) ([]string, []any) {
clauses := []string{"sr.trigger = 'autoscan'"}
args := []any{}
if strings.TrimSpace(filter.Status) != "" {
args = append(args, strings.TrimSpace(filter.Status))
clauses = append(clauses, fmt.Sprintf("sr.status = $%d", len(args)))
}
if search := strings.ToLower(strings.TrimSpace(filter.Search)); search != "" {
args = append(args, "%"+search+"%")
param := fmt.Sprintf("$%d", len(args))
clauses = append(clauses, `(
lower(sr.id) LIKE `+param+`
OR lower(sr.mode) LIKE `+param+`
OR lower(sr.path) LIKE `+param+`
OR lower(sr.status) LIKE `+param+`
OR lower(COALESCE(sr.error_message, '')) LIKE `+param+`
OR lower(COALESCE(e.capability_id, '')) LIKE `+param+`
OR lower(COALESCE(e.status, '')) LIKE `+param+`
OR lower(COALESCE(e.source_id::text, '')) LIKE `+param+`
)`)
}
return clauses, args
}
// CountEvents returns the total number of autoscan events matching filter,
// ignoring limit/offset. It powers the "of N" total in paginated views.
func (r *Repository) CountEvents(ctx context.Context, filter EventListFilter) (int, error) {
clauses, args := eventFilterClauses(filter)
var total int
if err := r.pool.QueryRow(ctx, `
SELECT COUNT(*)
FROM autoscan_events
WHERE `+strings.Join(clauses, " AND "),
args...,
).Scan(&total); err != nil {
return 0, fmt.Errorf("count autoscan events: %w", err)
}
return total, nil
}
// CountAutoscanScans returns the total number of autoscan scan runs matching
// filter, ignoring limit/offset. It powers the "of N" total in paginated views.
func (r *Repository) CountAutoscanScans(ctx context.Context, filter ScanListFilter) (int, error) {
clauses, args := scanFilterClauses(filter)
var total int
if err := r.pool.QueryRow(ctx, `
SELECT COUNT(*)
FROM scan_runs sr
LEFT JOIN autoscan_events e ON e.id = sr.autoscan_event_id
WHERE `+strings.Join(clauses, " AND "),
args...,
).Scan(&total); err != nil {
return 0, fmt.Errorf("count autoscan scans: %w", err)
}
return total, nil
}
func (r *Repository) ListEvents(ctx context.Context, filter EventListFilter) ([]EventWithRuns, error) {
limit := clampAutoscanLimit(filter.Limit)
offset := filter.Offset
if offset < 0 {
offset = 0
}
clauses, args := eventFilterClauses(filter)
args = append(args, limit)
limitParam := len(args)
args = append(args, offset)
offsetParam := len(args)
rows, err := r.pool.Query(ctx, `
SELECT `+eventColumns+`
FROM autoscan_events
WHERE `+strings.Join(clauses, " AND ")+`
ORDER BY completed_at DESC, id DESC
LIMIT $`+fmt.Sprint(limitParam)+` OFFSET $`+fmt.Sprint(offsetParam),
args...,
)
if err != nil {
return nil, fmt.Errorf("list autoscan events: %w", err)
}
defer rows.Close()
events := make([]EventWithRuns, 0)
ids := make([]int64, 0)
indexByID := map[int64]int{}
for rows.Next() {
event, scanErr := scanEvent(rows)
if scanErr != nil {
return nil, scanErr
}
indexByID[event.ID] = len(events)
ids = append(ids, event.ID)
events = append(events, EventWithRuns{Event: event, Runs: []ScanRunSummary{}})
}
if err := rows.Err(); err != nil {
return nil, err
}
if len(ids) == 0 {
return events, nil
}
runRows, err := r.pool.Query(ctx, `
SELECT autoscan_event_id, id, media_folder_id, mode, path, trigger, status,
COALESCE(error_message, ''), requested_at, started_at, completed_at
FROM scan_runs
WHERE autoscan_event_id = ANY($1)
ORDER BY requested_at ASC`,
ids,
)
if err != nil {
return nil, fmt.Errorf("list autoscan event scan runs: %w", err)
}
defer runRows.Close()
for runRows.Next() {
var eventID int64
var run ScanRunSummary
if err := runRows.Scan(
&eventID,
&run.ID,
&run.MediaFolderID,
&run.Mode,
&run.Path,
&run.Trigger,
&run.Status,
&run.ErrorMessage,
&run.RequestedAt,
&run.StartedAt,
&run.CompletedAt,
); err != nil {
return nil, err
}
if idx, ok := indexByID[eventID]; ok {
events[idx].Runs = append(events[idx].Runs, run)
}
}
return events, runRows.Err()
}
func (r *Repository) ListAutoscanScans(ctx context.Context, filter ScanListFilter) ([]ScanWithEvent, error) {
limit := clampAutoscanLimit(filter.Limit)
offset := filter.Offset
if offset < 0 {
offset = 0
}
clauses, args := scanFilterClauses(filter)
args = append(args, limit)
limitParam := len(args)
args = append(args, offset)
offsetParam := len(args)
rows, err := r.pool.Query(ctx, `
SELECT
sr.id,
sr.media_folder_id,
sr.mode,
sr.path,
sr.trigger,
sr.status,
COALESCE(sr.error_message, ''),
sr.requested_at,
sr.started_at,
sr.completed_at,
sr.autoscan_event_id,
e.source_id,
e.installation_id,
COALESCE(e.capability_id, ''),
COALESCE(e.status, ''),
e.completed_at
FROM scan_runs sr
LEFT JOIN autoscan_events e ON e.id = sr.autoscan_event_id
WHERE `+strings.Join(clauses, " AND ")+`
ORDER BY COALESCE(sr.completed_at, sr.started_at, sr.requested_at) DESC, sr.id DESC
LIMIT $`+fmt.Sprint(limitParam)+` OFFSET $`+fmt.Sprint(offsetParam),
args...,
)
if err != nil {
return nil, fmt.Errorf("list autoscan scans: %w", err)
}
defer rows.Close()
scans := make([]ScanWithEvent, 0)
for rows.Next() {
var scan ScanWithEvent
var eventStatus string
if err := rows.Scan(
&scan.ID,
&scan.MediaFolderID,
&scan.Mode,
&scan.Path,
&scan.Trigger,
&scan.Status,
&scan.ErrorMessage,
&scan.RequestedAt,
&scan.StartedAt,
&scan.CompletedAt,
&scan.AutoscanEventID,
&scan.SourceID,
&scan.InstallationID,
&scan.CapabilityID,
&eventStatus,
&scan.EventCompletedAt,
); err != nil {
return nil, err
}
scan.EventStatus = EventStatus(eventStatus)
scans = append(scans, scan)
}
return scans, rows.Err()
}
func (r *Repository) GetQueueSummary(ctx context.Context) (QueueSummary, error) {
var summary QueueSummary
err := r.pool.QueryRow(ctx, `
SELECT
COUNT(*) FILTER (WHERE status = ANY($1))::int,
COUNT(*) FILTER (WHERE status = $2)::int,
COUNT(*) FILTER (WHERE status = $3)::int
FROM scan_runs
WHERE trigger = $4`,
[]string{"accepted", "running"},
"accepted",
"running",
"autoscan",
).Scan(&summary.Active, &summary.Accepted, &summary.Running)
if err != nil {
return QueueSummary{}, fmt.Errorf("get autoscan queue summary: %w", err)
}
return summary, nil
}
func (r *Repository) LatestEventAt(ctx context.Context) (*time.Time, error) {
var latest *time.Time
err := r.pool.QueryRow(ctx, `SELECT max(completed_at) FROM autoscan_events`).Scan(&latest)
if err != nil {
return nil, fmt.Errorf("get latest autoscan event time: %w", err)
}
return latest, nil
}
internal/autoscan/rewrite.go
+48
View File
@@ -0,0 +1,48 @@
package autoscan
import "strings"
// normalizeSeparators converts Windows backslash separators to forward slashes
// so paths from a Windows-hosted arr resolve on the Linux host. (filepath.ToSlash
// is a no-op on Linux, so the replacement is explicit.)
func normalizeSeparators(path string) string {
return strings.ReplaceAll(path, "\\", "/")
}
// applyRewrites returns path with the MOST-SPECIFIC matching prefix rewrite
// applied, or path unchanged when none match.
//
// "Most-specific" means the longest matching From wins, not the first one in the
// slice. A first-match strategy lets a broad rewrite (From="/data") shadow a
// nested, more-specific one (From="/data/media") when the broad rule happens to
// be listed first; the arr plugin review flagged exactly this. Selecting the
// longest matching prefix makes the result independent of rule ordering.
func applyRewrites(path string, rewrites []PathRewrite) string {
bestIdx := -1
bestLen := -1
var bestTrimmed string
for i, rw := range rewrites {
// Normalize the stored From the SAME way coveredBy/normalizePath does
// (backslash→slash, collapse '//', strip trailing '/') so a Windows-style
// or dup-slash rewrite that suggest-time reports as "covered" actually
// matches at poll time. Without this, a From like `D:\data\tv` would be
// "covered" in the suggester yet never match the normalized incoming path.
trimmed := normalizePath(rw.From)
if trimmed == "" {
continue
}
// Match only at a path-segment boundary: exact, or prefix followed by '/'.
// This prevents From="/data/media" from rewriting "/data/media2/x".
if path == trimmed || strings.HasPrefix(path, trimmed+"/") {
if len(trimmed) > bestLen {
bestLen = len(trimmed)
bestIdx = i
bestTrimmed = trimmed
}
}
}
if bestIdx < 0 {
return path
}
return strings.TrimSpace(rewrites[bestIdx].To) + strings.TrimPrefix(path, bestTrimmed)
}
internal/autoscan/rewrite_test.go
+97
View File
@@ -0,0 +1,97 @@
package autoscan
import "testing"
func TestApplyRewrites(t *testing.T) {
rw := []PathRewrite{{From: "/data/media", To: "/mnt/media"}}
cases := []struct{ in, want string }{
{"/data/media/Movies/Dune/Dune.mkv", "/mnt/media/Movies/Dune/Dune.mkv"},
{"/other/path/file.mkv", "/other/path/file.mkv"}, // no-match pass-through
}
for _, tc := range cases {
if got := applyRewrites(tc.in, rw); got != tc.want {
t.Fatalf("applyRewrites(%q) = %q, want %q", tc.in, got, tc.want)
}
}
// Exact match (no trailing segment) rewrites the whole path.
if got := applyRewrites("/data/media", rw); got != "/mnt/media" {
t.Fatalf("exact match: got %q want %q", got, "/mnt/media")
}
// nil/empty rewrites pass through unchanged.
if got := applyRewrites("/data/media/x", nil); got != "/data/media/x" {
t.Fatalf("nil rewrites: got %q", got)
}
// Segment-boundary matching: a sibling dir sharing the prefix must NOT match.
boundary := []PathRewrite{{From: "/data/media", To: "/mnt"}}
if got := applyRewrites("/data/media2/x", boundary); got != "/data/media2/x" {
t.Fatalf("boundary: /data/media2/x must not rewrite, got %q", got)
}
}
// TestApplyRewritesNormalizesStoredFrom verifies that a Windows-style / dup-slash
// stored From is normalized the same way coveredBy/normalizePath does, so a
// rewrite the suggester reports as "covered" actually matches at poll time. The
// incoming path is already separator-normalized by PollOnce before applyRewrites.
func TestApplyRewritesNormalizesStoredFrom(t *testing.T) {
// Backslash From: a Windows-hosted arr root stored verbatim.
winFrom := []PathRewrite{{From: `D:\data\tv`, To: "/mnt/media/tv"}}
if got := applyRewrites("D:/data/tv/Show/S01/E01.mkv", winFrom); got != "/mnt/media/tv/Show/S01/E01.mkv" {
t.Fatalf("backslash From should match normalized path, got %q", got)
}
// coveredBy must agree with applyRewrites: the normalized root is covered.
if !coveredBy("D:/data/tv", winFrom) {
t.Fatalf("coveredBy should report the backslash From as covering the root")
}
// Dup-slash From collapses too.
dupFrom := []PathRewrite{{From: "/data//tv/", To: "/mnt/media/tv"}}
if got := applyRewrites("/data/tv/Show/E.mkv", dupFrom); got != "/mnt/media/tv/Show/E.mkv" {
t.Fatalf("dup-slash From should match collapsed path, got %q", got)
}
}
// TestApplyRewritesMostSpecificWins verifies the longest matching From wins
// regardless of slice ordering: a broad rule must not shadow a nested one.
func TestApplyRewritesMostSpecificWins(t *testing.T) {
// Broad rule listed FIRST: a first-match strategy would (wrongly) pick /data
// and yield "/A/media/x". Most-specific must pick /data/media -> "/B/x".
broadFirst := []PathRewrite{
{From: "/data", To: "/A"},
{From: "/data/media", To: "/B"},
}
if got := applyRewrites("/data/media/x", broadFirst); got != "/B/x" {
t.Fatalf("broad-first: got %q want %q", got, "/B/x")
}
// Same rules, specific listed first: result must be identical (order-independent).
specificFirst := []PathRewrite{
{From: "/data/media", To: "/B"},
{From: "/data", To: "/A"},
}
if got := applyRewrites("/data/media/x", specificFirst); got != "/B/x" {
t.Fatalf("specific-first: got %q want %q", got, "/B/x")
}
// A path under the broad rule but NOT the specific one still uses the broad rule.
if got := applyRewrites("/data/other/x", broadFirst); got != "/A/other/x" {
t.Fatalf("broad fallthrough: got %q want %q", got, "/A/other/x")
}
}
func TestNormalizeSeparators(t *testing.T) {
if got := normalizeSeparators(`C:\Media\Movies\Dune\Dune.mkv`); got != "C:/Media/Movies/Dune/Dune.mkv" {
t.Fatalf("normalizeSeparators(windows) = %q", got)
}
// POSIX paths are unchanged.
if got := normalizeSeparators("/mnt/media/x.mkv"); got != "/mnt/media/x.mkv" {
t.Fatalf("normalizeSeparators(posix) = %q", got)
}
// A normalized Windows path then rewrites on the Linux host.
rw := []PathRewrite{{From: "C:/Media", To: "/mnt/media"}}
if got := applyRewrites(normalizeSeparators(`C:\Media\ShowA\S01\E01.mkv`), rw); got != "/mnt/media/ShowA/S01/E01.mkv" {
t.Fatalf("windows rewrite = %q", got)
}
}
internal/autoscan/service.go
+464
View File
@@ -0,0 +1,464 @@
package autoscan
import (
"context"
"errors"
"fmt"
"log/slog"
"time"
"github.com/Silo-Server/silo-server/internal/scantrigger"
)
const scanTrigger = "autoscan"
// Store is the persistence surface the engine needs. The repository implements
// the full CRUD surface; this interface is the read/bookkeeping subset the poll
// loop touches.
type Store interface {
GetSettings(ctx context.Context) (Settings, error)
ListEnabledSources(ctx context.Context) ([]Source, error)
GetSource(ctx context.Context, id string) (Source, error)
GetConnection(ctx context.Context, id string) (Connection, error)
AdvanceMarker(ctx context.Context, sourceID, marker string) error
RecordError(ctx context.Context, sourceID, msg string) error
CreateEvent(ctx context.Context, event EventCreate) (int64, error)
FinishEvent(ctx context.Context, event EventFinish) error
}
// Resolver maps a Silo-native path to a concrete scan target (library folder).
type Resolver interface {
Resolve(ctx context.Context, req scantrigger.Request) (*scantrigger.Target, error)
ResolveMissingSubtree(ctx context.Context, subtreePath, trigger string) (*scantrigger.Target, error)
}
// Queuer enqueues resolved scan targets.
type Queuer interface {
EnqueueScans(ctx context.Context, targets []scantrigger.Target) error
EnqueueAutoscanScans(ctx context.Context, targets []scantrigger.Target, eventID int64) (created, reused int, err error)
}
type resolveStats struct {
ChangesResolved int
TargetsClaimed int
Suppressed int
}
// connectionResolver resolves a stored connection to concrete credentials.
type connectionResolver interface {
Resolve(ctx context.Context, c Connection) (ResolvedConnection, error)
}
// RootFolderClient lists a Radarr/Sonarr instance's configured root folder
// paths (used by the rewrite suggester). The concrete impl additionally
// satisfies ArrStatusProbe for the connection-test endpoint.
type RootFolderClient interface {
RootFolders(ctx context.Context, baseURL, apiKey string) ([]string, error)
}
// FolderLister lists every Silo media-folder path (used by the rewrite
// suggester to match arr roots against Silo folders).
type FolderLister interface {
ListFolderPaths(ctx context.Context) ([]string, error)
}
type Service struct {
store Store
provider ScanSourceProvider
connres connectionResolver
resolver Resolver
queue Queuer
suppress Suppressor
lister ScanSourceLister
// Optional deps for the connection-test and rewrite-suggester endpoints.
// Wired via setters so the poll-loop constructor stays unchanged and tests
// that only exercise PollOnce need not supply them.
rootFolders RootFolderClient
folders FolderLister
}
// SetSuggesterDeps wires the dependencies the rewrite-suggester and
// connection-test endpoints need: an arr root-folder/status client and a Silo
// media-folder lister. Optional — when unset, SuggestRewrites/TestConnection
// return an error.
func (s *Service) SetSuggesterDeps(rootFolders RootFolderClient, folders FolderLister) {
s.rootFolders = rootFolders
s.folders = folders
}
// NewService builds the autoscan engine. The provider supplies changed paths
// from scan_source plugins; connres resolves a source's connection to concrete
// credentials; resolver/queue/suppress drive the resolve→suppress→enqueue loop.
// lister enumerates installed scan_source capabilities so the Add-source picker
// can offer them and PollOnce can skip orphaned source rows; it may be nil
// (the picker then returns an empty list and orphan-pruning is disabled).
func NewService(
store Store,
provider ScanSourceProvider,
connres connectionResolver,
resolver Resolver,
queue Queuer,
suppress Suppressor,
lister ScanSourceLister,
) *Service {
return &Service{
store: store,
provider: provider,
connres: connres,
resolver: resolver,
queue: queue,
suppress: suppress,
lister: lister,
}
}
// PollOnce runs one autoscan cycle. Per-source failures are logged and skipped;
// only settings/listing errors propagate. The opaque next marker returned by the
// provider is stored verbatim, but only when the cycle's work is genuinely
// consumed: the provider returned no paths, or it returned paths and at least one
// resolved+enqueued. When paths come back but NONE resolve to a library folder
// (e.g. a freshly-enabled source with unconfigured rewrites) the marker is held
// and an error recorded, so those imports aren't skipped forever.
func (s *Service) PollOnce(ctx context.Context) error {
// Fetch the set of currently-installed scan_source capabilities so we can skip
// orphaned source rows (their plugin was uninstalled). Listing failures are
// non-fatal: a nil set means the installed set is unavailable, in which case
// we do NOT prune orphans — every enabled source is assumed present.
present, derr := s.installedScanSources(ctx)
if derr != nil {
slog.WarnContext(ctx, "autoscan: list installed scan sources failed", "err", derr)
present = nil
}
settings, err := s.store.GetSettings(ctx)
if err != nil {
return err
}
if !settings.Enabled {
return nil
}
sources, err := s.store.ListEnabledSources(ctx)
if err != nil {
return err
}
ttl := time.Duration(settings.DebounceSeconds) * time.Second
now := time.Now()
for _, src := range sources {
// Skip orphaned sources: an enabled source whose scan_source plugin has
// been uninstalled/disabled is no longer in the discovered set, so polling
// it would error every cycle. Skip it quietly (no RecordError) so the spam
// stops; the operator can delete it via the source delete endpoint. A nil
// `present` set means discovery is unavailable — don't prune in that case.
if present != nil {
if _, ok := present[installedKey{InstallationID: src.InstallationID, CapabilityID: src.CapabilityID}]; !ok {
continue
}
}
// Honor the per-source poll interval as a "poll at most every N seconds"
// floor: the global task fires at the default cadence, so a source with a
// longer interval is skipped until enough time has elapsed.
interval := time.Duration(settings.DefaultPollIntervalSeconds) * time.Second
if src.PollIntervalSeconds != nil {
interval = time.Duration(*src.PollIntervalSeconds) * time.Second
}
if src.LastRunAt != nil && now.Sub(*src.LastRunAt) < interval {
continue
}
marker := ""
if src.Marker != nil {
marker = *src.Marker
}
eventID := s.createEvent(ctx, src, marker, time.Now())
// A connection is OPTIONAL. Server-based providers (Sonarr/Radarr) bind a
// connection and the resolved {base_url, api_key} is handed to the plugin.
// Other providers (e.g. a filesystem/CephFS watcher) need none and get an
// empty connection they ignore. If a plugin requires a connection it didn't
// get, it returns an error that is RecordError'd below — so the operator
// still sees "needs attention" without the host assuming every source is
// credential-based.
var conn ResolvedConnection
if src.ConnectionID != nil {
resolved, cerr := s.resolveConnection(ctx, *src.ConnectionID)
if cerr != nil {
slog.WarnContext(ctx, "autoscan: resolve connection failed", "source_id", src.ID, "err", cerr)
if rerr := s.store.RecordError(ctx, src.ID, cerr.Error()); rerr != nil {
slog.WarnContext(ctx, "autoscan: record error failed", "source_id", src.ID, "err", rerr)
}
s.finishEvent(ctx, eventID, EventFinish{
Status: EventStatusError,
ErrorMessage: cerr.Error(),
MarkerAfter: marker,
})
continue
}
conn = resolved
}
changes, next, perr := s.provider.PollChanges(ctx, src.InstallationID, src.CapabilityID, marker, conn, src.SourceConfig)
if perr != nil {
slog.WarnContext(ctx, "autoscan: poll changes failed", "source_id", src.ID, "err", perr)
if rerr := s.store.RecordError(ctx, src.ID, perr.Error()); rerr != nil {
slog.WarnContext(ctx, "autoscan: record error failed", "source_id", src.ID, "err", rerr)
}
s.finishEvent(ctx, eventID, EventFinish{
Status: EventStatusError,
ErrorMessage: perr.Error(),
MarkerAfter: marker,
})
continue // do NOT advance marker
}
rewritten := rewriteChanges(changes, src.PathRewrites)
targets, claimed, resolvedAny, stats := s.resolveAndClaim(ctx, rewritten, ttl)
var enqueue EnqueueResult
if len(targets) > 0 {
var eerr error
if eventID != 0 {
enqueue.Created, enqueue.Reused, eerr = s.queue.EnqueueAutoscanScans(ctx, targets, eventID)
} else {
eerr = s.queue.EnqueueScans(ctx, targets)
enqueue.Created = len(targets)
}
if eerr != nil {
s.releaseClaims(ctx, claimed)
slog.WarnContext(ctx, "autoscan: enqueue failed", "source_id", src.ID, "err", eerr)
s.finishEvent(ctx, eventID, EventFinish{
Status: EventStatusError,
ChangesReturned: len(changes),
ChangesResolved: stats.ChangesResolved,
TargetsClaimed: stats.TargetsClaimed,
ScansSuppressed: stats.Suppressed,
ErrorMessage: eerr.Error(),
MarkerAfter: marker,
})
continue // do NOT advance marker
}
}
// Advancing the marker is what tells the provider "I've consumed up to
// here". Advance it ONLY when the work it represents is genuinely done:
// - provider returned ZERO paths → nothing to do, advance normally.
// - returned paths AND ≥1 resolved → enqueued (above), advance.
// - returned paths, resolved but all
// suppressed (recently scanned /
// debounced) → work is effectively done; advance.
// - returned paths AND NOTHING resolved → do NOT advance. This is the
// freshly-enabled-source state (path_rewrites not configured yet); the
// incoming paths don't map to any Silo library folder. Advancing here
// would skip those imports forever. Surface why so the operator can fix
// the rewrites, then a later poll re-reads the same window and resolves.
// (Some-but-not-all resolving counts as resolved — the unresolved paths are
// legitimately outside Silo's libraries, so advancing is correct.)
//
// NOTE: len(targets)==0 alone is NOT misconfiguration — paths can resolve
// yet be fully suppressed. Gate the error on resolvedAny, not targets.
if len(changes) > 0 && !resolvedAny {
msg := fmt.Sprintf("returned %d path(s) but none matched a Silo library folder — check this source's path rewrites", len(changes))
if rerr := s.store.RecordError(ctx, src.ID, msg); rerr != nil {
slog.WarnContext(ctx, "autoscan: record error failed", "source_id", src.ID, "err", rerr)
}
s.finishEvent(ctx, eventID, EventFinish{
Status: EventStatusUnresolved,
ChangesReturned: len(changes),
ErrorMessage: msg,
MarkerAfter: marker,
})
continue // do NOT advance marker
}
if aerr := s.store.AdvanceMarker(ctx, src.ID, next); aerr != nil {
slog.WarnContext(ctx, "autoscan: advance marker failed", "source_id", src.ID, "err", aerr)
s.finishEvent(ctx, eventID, EventFinish{
Status: EventStatusError,
ChangesReturned: len(changes),
ChangesResolved: stats.ChangesResolved,
TargetsClaimed: stats.TargetsClaimed,
ScansCreated: enqueue.Created,
ScansReused: enqueue.Reused,
ScansSuppressed: stats.Suppressed,
ErrorMessage: aerr.Error(),
MarkerAfter: marker,
})
continue
}
s.finishEvent(ctx, eventID, EventFinish{
Status: EventStatusSuccess,
ChangesReturned: len(changes),
ChangesResolved: stats.ChangesResolved,
TargetsClaimed: stats.TargetsClaimed,
ScansCreated: enqueue.Created,
ScansReused: enqueue.Reused,
ScansSuppressed: stats.Suppressed,
MarkerAfter: next,
})
}
return nil
}
func (s *Service) createEvent(ctx context.Context, src Source, marker string, startedAt time.Time) int64 {
if s == nil || s.store == nil {
return 0
}
id, err := s.store.CreateEvent(ctx, EventCreate{
SourceID: src.ID,
InstallationID: src.InstallationID,
CapabilityID: src.CapabilityID,
StartedAt: startedAt,
MarkerBefore: marker,
})
if err != nil {
slog.WarnContext(ctx, "autoscan: create event failed", "source_id", src.ID, "err", err)
return 0
}
return id
}
func (s *Service) finishEvent(ctx context.Context, eventID int64, finish EventFinish) {
if eventID == 0 || s == nil || s.store == nil {
return
}
finish.ID = eventID
finish.CompletedAt = time.Now()
if finish.Status == "" {
finish.Status = EventStatusSuccess
}
if err := s.store.FinishEvent(ctx, finish); err != nil {
slog.WarnContext(ctx, "autoscan: finish event failed", "event_id", eventID, "err", err)
}
}
// resolveConnection loads and resolves a source's connection to credentials.
func (s *Service) resolveConnection(ctx context.Context, connectionID string) (ResolvedConnection, error) {
conn, err := s.store.GetConnection(ctx, connectionID)
if err != nil {
return ResolvedConnection{}, err
}
return s.connres.Resolve(ctx, conn)
}
func rewriteChanges(changes []Change, rewrites []PathRewrite) []Change {
rewritten := make([]Change, 0, len(changes))
for _, change := range changes {
path := applyRewrites(normalizeSeparators(change.SourcePath), rewrites)
rewritten = append(rewritten, Change{SourcePath: path, Scope: change.Scope})
}
return rewritten
}
// resolveAndClaim resolves changes to scan targets and atomically claims them
// via the suppressor. Legacy/auto changes retain the historical parent-dir
// collapse. Structured file changes resolve exact files, and subtree changes
// resolve exact subtree paths even when the path no longer exists.
func (s *Service) resolveAndClaim(ctx context.Context, changes []Change, ttl time.Duration) (targets []scantrigger.Target, claimed []string, resolvedAny bool, stats resolveStats) {
seenTargets := make(map[string]struct{})
var legacyPaths []string
for _, change := range changes {
switch change.Scope {
case ChangeScopeFile, ChangeScopeSubtree:
target, ok := s.resolveChange(ctx, change)
if !ok {
continue
}
resolvedAny = true
stats.ChangesResolved++
if !s.claimTarget(ctx, *target, ttl, seenTargets, &targets, &claimed) {
stats.Suppressed++
}
default:
legacyPaths = append(legacyPaths, change.SourcePath)
}
}
for _, dir := range uniqueParentDirs(legacyPaths) {
target, rerr := s.resolver.Resolve(ctx, scantrigger.Request{Path: dir, Trigger: scanTrigger})
if rerr != nil {
var reqErr *scantrigger.RequestError
if errors.As(rerr, &reqErr) {
// Path outside Silo's media folders (or otherwise unresolvable)
// — an expected skip, not an error worth logging every cycle.
continue
}
slog.WarnContext(ctx, "autoscan: resolve failed", "path", dir, "err", rerr)
continue
}
if target == nil || target.Folder == nil {
continue
}
resolvedAny = true
stats.ChangesResolved++
if !s.claimTarget(ctx, *target, ttl, seenTargets, &targets, &claimed) {
stats.Suppressed++
}
}
stats.TargetsClaimed = len(targets)
return targets, claimed, resolvedAny, stats
}
func (s *Service) resolveChange(ctx context.Context, change Change) (*scantrigger.Target, bool) {
if change.SourcePath == "" {
return nil, false
}
var (
target *scantrigger.Target
err error
)
switch change.Scope {
case ChangeScopeSubtree:
target, err = s.resolver.ResolveMissingSubtree(ctx, change.SourcePath, scanTrigger)
case ChangeScopeFile:
target, err = s.resolver.Resolve(ctx, scantrigger.Request{Path: change.SourcePath, Trigger: scanTrigger})
if err == nil && target != nil && target.Mode != scantrigger.ModeFile {
return nil, false
}
default:
target, err = s.resolver.Resolve(ctx, scantrigger.Request{Path: change.SourcePath, Trigger: scanTrigger})
}
if err != nil {
var reqErr *scantrigger.RequestError
if errors.As(err, &reqErr) {
return nil, false
}
slog.WarnContext(ctx, "autoscan: resolve failed", "path", change.SourcePath, "scope", change.Scope, "err", err)
return nil, false
}
if target == nil || target.Folder == nil {
return nil, false
}
return target, true
}
func (s *Service) claimTarget(
ctx context.Context,
target scantrigger.Target,
ttl time.Duration,
seenTargets map[string]struct{},
targets *[]scantrigger.Target,
claimed *[]string,
) bool {
targetKey := fmt.Sprintf("%d|%s|%s", target.Folder.ID, target.Mode, target.Path)
if _, seen := seenTargets[targetKey]; seen {
return false
}
seenTargets[targetKey] = struct{}{}
key := fmt.Sprintf("%d|%s", target.Folder.ID, target.Path)
ok, serr := s.suppress.ShouldScan(ctx, key, ttl)
if serr != nil || !ok {
return false
}
target.Trigger = scanTrigger
*targets = append(*targets, target)
*claimed = append(*claimed, key)
return true
}
// releaseClaims drops suppression claims (used when the scan enqueue fails so a
// later cycle can retry the same targets).
func (s *Service) releaseClaims(ctx context.Context, claimed []string) {
for _, k := range claimed {
if rerr := s.suppress.Release(ctx, k); rerr != nil {
slog.WarnContext(ctx, "autoscan: release claim failed", "key", k, "err", rerr)
}
}
}
internal/autoscan/service_test.go
+746
View File
@@ -0,0 +1,746 @@
package autoscan
import (
"context"
"path/filepath"
"strings"
"testing"
"time"
"github.com/Silo-Server/silo-server/internal/models"
"github.com/Silo-Server/silo-server/internal/scantrigger"
)
type fakeStore struct {
settings Settings
sources []Source
connection Connection
advanced map[string]string // source ID -> marker
recorded map[string]string // source ID -> error message
createdEvents []EventCreate
events []EventFinish
}
func (f *fakeStore) GetSettings(context.Context) (Settings, error) { return f.settings, nil }
func (f *fakeStore) ListEnabledSources(context.Context) ([]Source, error) {
return f.sources, nil
}
func (f *fakeStore) GetSource(_ context.Context, id string) (Source, error) {
for _, s := range f.sources {
if s.ID == id {
return s, nil
}
}
return Source{}, ErrNotFound
}
func (f *fakeStore) GetConnection(context.Context, string) (Connection, error) {
return f.connection, nil
}
func (f *fakeStore) AdvanceMarker(_ context.Context, sourceID, marker string) error {
if f.advanced == nil {
f.advanced = map[string]string{}
}
f.advanced[sourceID] = marker
return nil
}
func (f *fakeStore) RecordError(_ context.Context, sourceID, msg string) error {
if f.recorded == nil {
f.recorded = map[string]string{}
}
f.recorded[sourceID] = msg
return nil
}
func (f *fakeStore) CreateEvent(_ context.Context, event EventCreate) (int64, error) {
f.createdEvents = append(f.createdEvents, event)
return int64(len(f.createdEvents)), nil
}
func (f *fakeStore) FinishEvent(_ context.Context, event EventFinish) error {
f.events = append(f.events, event)
return nil
}
// fakeProvider implements ScanSourceProvider, keyed by capability ID.
type fakeProvider struct {
paths map[string][]string // key: capabilityID, legacy source_paths behavior
changes map[string][]Change // key: capabilityID, structured changes
nextMarker string
err error
lastConfig map[string]string
}
func (f *fakeProvider) PollChanges(_ context.Context, _ int, capabilityID, _ string, _ ResolvedConnection, sourceConfig map[string]string) ([]Change, string, error) {
f.lastConfig = sourceConfig
if f.err != nil {
return nil, "", f.err
}
if changes, ok := f.changes[capabilityID]; ok {
return changes, f.nextMarker, nil
}
changes := make([]Change, 0, len(f.paths[capabilityID]))
for _, path := range f.paths[capabilityID] {
changes = append(changes, Change{SourcePath: path, Scope: ChangeScopeAuto})
}
return changes, f.nextMarker, nil
}
func TestPollOncePassesSourceConfigToProvider(t *testing.T) {
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{
ID: "s1", InstallationID: 1, CapabilityID: "cephfs", Enabled: true,
SourceConfig: map[string]string{"exclusions": ".downloads\n.recyclebin"},
}},
}
prov := &fakeProvider{nextMarker: "m1"}
svc := newService(store, prov, &recordingQueuer{}, allowSuppressor{})
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if got := prov.lastConfig["exclusions"]; got != ".downloads\n.recyclebin" {
t.Fatalf("source config = %#v", prov.lastConfig)
}
}
// passthroughConnRes resolves to empty credentials; the engine doesn't inspect
// them in tests (the fake provider ignores conn).
type passthroughConnRes struct{}
func (passthroughConnRes) Resolve(context.Context, Connection) (ResolvedConnection, error) {
return ResolvedConnection{}, nil
}
type fakeResolver struct{}
func (fakeResolver) Resolve(_ context.Context, req scantrigger.Request) (*scantrigger.Target, error) {
if strings.HasPrefix(req.Path, "/mnt/media/") {
mode := scantrigger.ModeSubtree
if filepath.Ext(req.Path) == ".mkv" {
mode = scantrigger.ModeFile
}
return &scantrigger.Target{Folder: &models.MediaFolder{ID: 7}, Mode: mode, Path: req.Path, Trigger: req.Trigger}, nil
}
return nil, nil
}
func (fakeResolver) ResolveMissingSubtree(_ context.Context, subtreePath, trigger string) (*scantrigger.Target, error) {
if strings.HasPrefix(subtreePath, "/mnt/media/") {
return &scantrigger.Target{Folder: &models.MediaFolder{ID: 7}, Mode: scantrigger.ModeSubtree, Path: subtreePath, Trigger: trigger}, nil
}
return nil, &scantrigger.RequestError{Status: 400, Code: "bad_request", Message: "outside media folders"}
}
// unresolvableResolver treats every path as outside Silo's media folders,
// returning a RequestError — the "none resolved → misconfiguration" signal.
type unresolvableResolver struct{}
func (unresolvableResolver) Resolve(_ context.Context, req scantrigger.Request) (*scantrigger.Target, error) {
return nil, &scantrigger.RequestError{Status: 400, Code: "bad_request", Message: "outside media folders"}
}
func (unresolvableResolver) ResolveMissingSubtree(context.Context, string, string) (*scantrigger.Target, error) {
return nil, &scantrigger.RequestError{Status: 400, Code: "bad_request", Message: "outside media folders"}
}
// denySuppressor resolves paths normally but denies every claim, simulating a
// recently-scanned / debounced target (resolved but suppressed).
type denySuppressor struct{}
func (denySuppressor) ShouldScan(context.Context, string, time.Duration) (bool, error) {
return false, nil
}
func (denySuppressor) Release(context.Context, string) error { return nil }
type recordingQueuer struct {
enqueued []scantrigger.Target
autoscanEvents []int64
createdCount *int
reusedCount int
}
func (q *recordingQueuer) EnqueueScans(_ context.Context, targets []scantrigger.Target) error {
q.enqueued = append(q.enqueued, targets...)
return nil
}
func (q *recordingQueuer) EnqueueAutoscanScans(_ context.Context, targets []scantrigger.Target, eventID int64) (int, int, error) {
q.enqueued = append(q.enqueued, targets...)
q.autoscanEvents = append(q.autoscanEvents, eventID)
if q.createdCount != nil {
return *q.createdCount, q.reusedCount, nil
}
return len(targets), q.reusedCount, nil
}
type allowSuppressor struct{}
func (allowSuppressor) ShouldScan(context.Context, string, time.Duration) (bool, error) {
return true, nil
}
func (allowSuppressor) Release(context.Context, string) error { return nil }
type recordingSuppressor struct {
claimed []string
released []string
}
func (s *recordingSuppressor) ShouldScan(_ context.Context, key string, _ time.Duration) (bool, error) {
s.claimed = append(s.claimed, key)
return true, nil
}
func (s *recordingSuppressor) Release(_ context.Context, key string) error {
s.released = append(s.released, key)
return nil
}
type failingQueuer struct{}
func (failingQueuer) EnqueueScans(context.Context, []scantrigger.Target) error {
return context.DeadlineExceeded
}
func (failingQueuer) EnqueueAutoscanScans(context.Context, []scantrigger.Target, int64) (int, int, error) {
return 0, 0, context.DeadlineExceeded
}
func newService(store Store, provider ScanSourceProvider, queue Queuer, suppress Suppressor) *Service {
return NewService(store, provider, passthroughConnRes{}, fakeResolver{}, queue, suppress, nil)
}
// strptr is a tiny helper for the *string ConnectionID field in tests.
func strptr(s string) *string { return &s }
func TestPollOnceEnqueuesDedupedFolders(t *testing.T) {
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{
ID: "s1", InstallationID: 1, CapabilityID: "arr", ConnectionID: strptr("c1"), Enabled: true,
}},
}
prov := &fakeProvider{paths: map[string][]string{
"arr": {
"/mnt/media/Show/S01/E01.mkv",
"/mnt/media/Show/S01/E02.mkv",
"/outside/lib/x.mkv",
},
}, nextMarker: "m1"}
q := &recordingQueuer{}
svc := newService(store, prov, q, allowSuppressor{})
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if len(q.enqueued) != 1 {
t.Fatalf("expected 1 deduped folder scan, got %d: %+v", len(q.enqueued), q.enqueued)
}
if q.enqueued[0].Trigger != "autoscan" || q.enqueued[0].Folder.ID != 7 {
t.Fatalf("unexpected target: %+v", q.enqueued[0])
}
if _, ok := store.advanced["s1"]; !ok {
t.Fatalf("expected marker advanced for s1")
}
}
func TestPollOnceRecordsSuccessfulEvent(t *testing.T) {
marker := "m0"
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{
ID: "s1", InstallationID: 1, CapabilityID: "arr", ConnectionID: strptr("c1"), Enabled: true, Marker: &marker,
}},
}
prov := &fakeProvider{paths: map[string][]string{
"arr": {"/mnt/media/Show/S01/E01.mkv"},
}, nextMarker: "m1"}
q := &recordingQueuer{}
svc := newService(store, prov, q, allowSuppressor{})
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if len(store.createdEvents) != 1 {
t.Fatalf("expected one event created, got %d", len(store.createdEvents))
}
if got := store.createdEvents[0].MarkerBefore; got != marker {
t.Fatalf("marker_before = %q, want %q", got, marker)
}
if len(q.autoscanEvents) != 1 || q.autoscanEvents[0] != 1 {
t.Fatalf("autoscan enqueue event ids = %v, want [1]", q.autoscanEvents)
}
if len(store.events) != 1 {
t.Fatalf("expected one finished event, got %d", len(store.events))
}
event := store.events[0]
if event.ID != 1 || event.Status != EventStatusSuccess {
t.Fatalf("event identity/status = %+v", event)
}
if event.ChangesReturned != 1 || event.ChangesResolved != 1 || event.TargetsClaimed != 1 {
t.Fatalf("event change counts = %+v", event)
}
if event.ScansCreated != 1 || event.ScansReused != 0 || event.ScansSuppressed != 0 {
t.Fatalf("event scan counts = %+v", event)
}
if event.MarkerAfter != "m1" || event.ErrorMessage != "" {
t.Fatalf("event completion fields = %+v", event)
}
}
func TestPollOnceRecordsReusedScanCounts(t *testing.T) {
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{
ID: "s1", InstallationID: 1, CapabilityID: "arr", ConnectionID: strptr("c1"), Enabled: true,
}},
}
prov := &fakeProvider{paths: map[string][]string{
"arr": {
"/mnt/media/ShowA/S01/E01.mkv",
"/mnt/media/ShowB/S01/E01.mkv",
},
}, nextMarker: "m1"}
created := 1
q := &recordingQueuer{createdCount: &created, reusedCount: 1}
svc := newService(store, prov, q, allowSuppressor{})
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if len(store.events) != 1 {
t.Fatalf("expected one finished event, got %d", len(store.events))
}
event := store.events[0]
if event.ScansCreated != 1 || event.ScansReused != 1 {
t.Fatalf("event scan counts = %+v", event)
}
if event.TargetsClaimed != 2 {
t.Fatalf("targets_claimed = %d, want 2", event.TargetsClaimed)
}
}
func TestPollOnceAppliesSourceRewritesBeforeEnqueue(t *testing.T) {
// The provider returns RAW source-namespace paths (/data/tv/...). The source's
// rewrite /data/tv -> /mnt/media/tv must be applied host-side so the resolved
// target uses the rewritten, Silo-native path. fakeResolver only resolves
// paths under /mnt/media/, so a missing rewrite would resolve to nothing.
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{
ID: "s1", InstallationID: 1, CapabilityID: "arr", ConnectionID: strptr("c1"), Enabled: true,
PathRewrites: []PathRewrite{{From: "/data/tv", To: "/mnt/media/tv"}},
}},
}
prov := &fakeProvider{paths: map[string][]string{
"arr": {"/data/tv/Show/S01/E01.mkv"},
}, nextMarker: "m1"}
q := &recordingQueuer{}
svc := newService(store, prov, q, allowSuppressor{})
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if len(q.enqueued) != 1 {
t.Fatalf("expected 1 enqueued target, got %d: %+v", len(q.enqueued), q.enqueued)
}
// uniqueParentDirs collapses E01.mkv to its parent dir; the rewritten target
// must be the Silo-native /mnt/media/tv/Show/S01.
if got := q.enqueued[0].Path; got != "/mnt/media/tv/Show/S01" {
t.Fatalf("expected rewritten target path /mnt/media/tv/Show/S01, got %q", got)
}
if _, ok := store.advanced["s1"]; !ok {
t.Fatalf("expected marker advanced for s1")
}
}
func TestPollOnceStructuredFileChangeEnqueuesExactFile(t *testing.T) {
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{
ID: "s1", InstallationID: 1, CapabilityID: "cephfs", Enabled: true,
PathRewrites: []PathRewrite{{From: "/ceph/tv", To: "/mnt/media/tv"}},
}},
}
prov := &fakeProvider{changes: map[string][]Change{
"cephfs": {{
SourcePath: "/ceph/tv/Show/S01/E01.mkv",
Scope: ChangeScopeFile,
}},
}, nextMarker: "m1"}
q := &recordingQueuer{}
svc := newService(store, prov, q, allowSuppressor{})
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if len(q.enqueued) != 1 {
t.Fatalf("expected 1 file scan, got %d: %+v", len(q.enqueued), q.enqueued)
}
if got := q.enqueued[0].Mode; got != scantrigger.ModeFile {
t.Fatalf("mode = %q, want %q", got, scantrigger.ModeFile)
}
if got := q.enqueued[0].Path; got != "/mnt/media/tv/Show/S01/E01.mkv" {
t.Fatalf("file path = %q", got)
}
if _, ok := store.advanced["s1"]; !ok {
t.Fatalf("expected marker advanced for s1")
}
}
func TestPollOnceStructuredSubtreeChangeEnqueuesExactSubtree(t *testing.T) {
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{
ID: "s1", InstallationID: 1, CapabilityID: "cephfs", Enabled: true,
PathRewrites: []PathRewrite{{From: "/ceph/movies", To: "/mnt/media/movies"}},
}},
}
prov := &fakeProvider{changes: map[string][]Change{
"cephfs": {{
SourcePath: "/ceph/movies/Movie (2026)",
Scope: ChangeScopeSubtree,
}},
}, nextMarker: "m1"}
q := &recordingQueuer{}
svc := newService(store, prov, q, allowSuppressor{})
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if len(q.enqueued) != 1 {
t.Fatalf("expected 1 subtree scan, got %d: %+v", len(q.enqueued), q.enqueued)
}
if got := q.enqueued[0].Mode; got != scantrigger.ModeSubtree {
t.Fatalf("mode = %q, want %q", got, scantrigger.ModeSubtree)
}
if got := q.enqueued[0].Path; got != "/mnt/media/movies/Movie (2026)" {
t.Fatalf("subtree path = %q", got)
}
if _, ok := store.advanced["s1"]; !ok {
t.Fatalf("expected marker advanced for s1")
}
}
func TestPollOnceScansDistinctPathsUnderSameFolder(t *testing.T) {
// Two imported subtrees resolve to the SAME folder ID (7) but DIFFERENT
// target paths. Keying suppression on folder ID alone would drop the second;
// keying on (folder, path) must scan both.
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{
ID: "s1", InstallationID: 1, CapabilityID: "arr", ConnectionID: strptr("c1"), Enabled: true,
}},
}
prov := &fakeProvider{paths: map[string][]string{
"arr": {
"/mnt/media/ShowA/S01/E01.mkv",
"/mnt/media/ShowB/S01/E01.mkv",
},
}, nextMarker: "m1"}
q := &recordingQueuer{}
sup := &recordingSuppressor{}
svc := newService(store, prov, q, sup)
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if len(q.enqueued) != 2 {
t.Fatalf("expected 2 enqueued targets (distinct paths), got %d: %+v", len(q.enqueued), q.enqueued)
}
if len(sup.claimed) != 2 {
t.Fatalf("expected 2 claimed keys, got %d: %v", len(sup.claimed), sup.claimed)
}
if sup.claimed[0] == sup.claimed[1] {
t.Fatalf("expected distinct claimed keys, both were %q", sup.claimed[0])
}
}
func TestPollOnceStoresOpaqueMarkerVerbatim(t *testing.T) {
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{
ID: "s1", InstallationID: 1, CapabilityID: "arr", ConnectionID: strptr("c1"), Enabled: true,
}},
}
const opaque = "eyJjdXJzb3IiOiJhYmMxMjMifQ==|2026-06-02T14:10:00Z"
prov := &fakeProvider{paths: map[string][]string{
"arr": {"/mnt/media/Show/S01/E01.mkv"},
}, nextMarker: opaque}
svc := newService(store, prov, &recordingQueuer{}, allowSuppressor{})
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if got := store.advanced["s1"]; got != opaque {
t.Fatalf("marker not stored verbatim: got %q want %q", got, opaque)
}
}
func TestPollOnceHoldsMarkerWhenPathsReturnedButNoneResolve(t *testing.T) {
// A freshly-enabled source with no path_rewrites: the provider returns paths
// that don't map to any Silo library folder (the resolver RequestErrors on
// every path). The marker must NOT advance (so the imports aren't skipped
// forever) and an explaining error is recorded.
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{
ID: "s1", InstallationID: 1, CapabilityID: "arr", ConnectionID: strptr("c1"), Enabled: true,
}},
}
prov := &fakeProvider{paths: map[string][]string{
"arr": {
"/data/tv/Show/S01/E01.mkv",
"/data/tv/Show/S01/E02.mkv",
},
}, nextMarker: "m1"}
q := &recordingQueuer{}
// unresolvableResolver rejects every path; allowSuppressor would happily claim
// anything, so the only reason nothing resolves is the resolver.
svc := NewService(store, prov, passthroughConnRes{}, unresolvableResolver{}, q, allowSuppressor{}, nil)
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if len(q.enqueued) != 0 {
t.Fatalf("nothing should enqueue when no path resolves, got %d", len(q.enqueued))
}
if _, ok := store.advanced["s1"]; ok {
t.Fatalf("marker must NOT advance when paths returned but none resolved")
}
msg, ok := store.recorded["s1"]
if !ok || msg == "" {
t.Fatalf("expected an explaining error recorded for s1, got %q ok=%v", msg, ok)
}
if want := "returned 2 path(s)"; !strings.Contains(msg, want) {
t.Fatalf("recorded error %q should mention %q", msg, want)
}
if len(store.events) != 1 {
t.Fatalf("expected one finished event, got %d", len(store.events))
}
event := store.events[0]
if event.Status != EventStatusUnresolved {
t.Fatalf("event status = %q, want %q", event.Status, EventStatusUnresolved)
}
if event.ChangesReturned != 2 || event.ChangesResolved != 0 || event.TargetsClaimed != 0 {
t.Fatalf("event counts = %+v", event)
}
if !strings.Contains(event.ErrorMessage, "returned 2 path(s)") {
t.Fatalf("event error = %q", event.ErrorMessage)
}
}
func TestPollOnceAdvancesMarkerWhenResolvedButAllSuppressed(t *testing.T) {
// Paths DO resolve to a Silo library folder, but every claim is denied by the
// suppressor (recently scanned / debounced). This is NOT a misconfiguration:
// the work is effectively done, so the marker must ADVANCE and no error is
// recorded. Regression guard for treating "resolved-but-suppressed" as
// "nothing resolved."
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{
ID: "s1", InstallationID: 1, CapabilityID: "arr", ConnectionID: strptr("c1"), Enabled: true,
// rewrite /data/tv -> /mnt/media/tv so fakeResolver resolves the paths.
PathRewrites: []PathRewrite{{From: "/data/tv", To: "/mnt/media/tv"}},
}},
}
prov := &fakeProvider{paths: map[string][]string{
"arr": {
"/data/tv/Show/S01/E01.mkv",
"/data/tv/Show/S01/E02.mkv",
},
}, nextMarker: "m1"}
q := &recordingQueuer{}
svc := newService(store, prov, q, denySuppressor{})
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if len(q.enqueued) != 0 {
t.Fatalf("nothing should enqueue when every target is suppressed, got %d", len(q.enqueued))
}
if got, ok := store.advanced["s1"]; !ok || got != "m1" {
t.Fatalf("marker must advance to %q when paths resolve but are all suppressed, got %q ok=%v", "m1", got, ok)
}
if _, ok := store.recorded["s1"]; ok {
t.Fatalf("no error should be recorded when targets merely suppressed (work is debounced, not misconfigured)")
}
}
func TestPollOnceAdvancesMarkerWhenZeroPathsReturned(t *testing.T) {
// The "nothing to do" case: provider returns no paths at all. The marker must
// advance normally and no error is recorded.
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{
ID: "s1", InstallationID: 1, CapabilityID: "arr", ConnectionID: strptr("c1"), Enabled: true,
}},
}
prov := &fakeProvider{paths: map[string][]string{"arr": {}}, nextMarker: "m1"}
q := &recordingQueuer{}
svc := newService(store, prov, q, allowSuppressor{})
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if got, ok := store.advanced["s1"]; !ok || got != "m1" {
t.Fatalf("marker must advance to %q when zero paths returned, got %q ok=%v", "m1", got, ok)
}
if _, ok := store.recorded["s1"]; ok {
t.Fatalf("no error should be recorded when there's simply nothing to do")
}
}
func TestPollOnceDisabledNoop(t *testing.T) {
store := &fakeStore{settings: Settings{Enabled: false}}
q := &recordingQueuer{}
svc := newService(store, &fakeProvider{}, q, allowSuppressor{})
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if len(q.enqueued) != 0 {
t.Fatalf("disabled autoscan should enqueue nothing, got %d", len(q.enqueued))
}
}
func TestPollOnceProviderErrorKeepsMarker(t *testing.T) {
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{ID: "s1", InstallationID: 1, CapabilityID: "arr", ConnectionID: strptr("c1"), Enabled: true}},
}
prov := &fakeProvider{err: context.DeadlineExceeded}
q := &recordingQueuer{}
svc := newService(store, prov, q, allowSuppressor{})
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce should not propagate per-source error: %v", err)
}
if _, ok := store.advanced["s1"]; ok {
t.Fatalf("marker must NOT advance on provider failure")
}
if msg, ok := store.recorded["s1"]; !ok || msg == "" {
t.Fatalf("expected provider error recorded for s1, got %q ok=%v", msg, ok)
}
}
func TestPollOnceReleasesClaimOnEnqueueFailure(t *testing.T) {
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{ID: "s1", InstallationID: 1, CapabilityID: "arr", ConnectionID: strptr("c1"), Enabled: true}},
}
prov := &fakeProvider{paths: map[string][]string{"arr": {"/mnt/media/Show/S01/E01.mkv"}}, nextMarker: "m1"}
sup := &recordingSuppressor{}
svc := newService(store, prov, failingQueuer{}, sup)
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if len(sup.claimed) != 1 || len(sup.released) != 1 || sup.released[0] != sup.claimed[0] {
t.Fatalf("expected claimed folder to be released, claimed=%v released=%v", sup.claimed, sup.released)
}
if _, ok := store.advanced["s1"]; ok {
t.Fatalf("marker must NOT advance when enqueue fails")
}
if len(store.events) != 1 {
t.Fatalf("expected one finished event, got %d", len(store.events))
}
event := store.events[0]
if event.Status != EventStatusError || !strings.Contains(event.ErrorMessage, context.DeadlineExceeded.Error()) {
t.Fatalf("event should record enqueue error, got %+v", event)
}
if event.ChangesReturned != 1 || event.ChangesResolved != 1 || event.TargetsClaimed != 1 {
t.Fatalf("event counts = %+v", event)
}
}
func TestPollOncePollsConnectionlessSource(t *testing.T) {
// A connection is OPTIONAL: a source with no bound connection is still polled
// (the provider gets an empty connection it may ignore — e.g. a filesystem
// watcher). Whether the plugin needs credentials is the plugin's concern.
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{
ID: "s1", InstallationID: 1, CapabilityID: "arr", ConnectionID: nil, Enabled: true,
}},
}
prov := &fakeProvider{paths: map[string][]string{"arr": {"/mnt/media/Show/S01/E01.mkv"}}, nextMarker: "m1"}
q := &recordingQueuer{}
svc := newService(store, prov, q, allowSuppressor{})
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if len(q.enqueued) != 1 {
t.Fatalf("a connection-less source must still poll + enqueue, got %d enqueued", len(q.enqueued))
}
if _, ok := store.recorded["s1"]; ok {
t.Fatalf("a connection-less source must not record a 'no connection' error")
}
if _, ok := store.advanced["s1"]; !ok {
t.Fatalf("marker must advance after a successful connection-less poll")
}
}
func TestPollOnceSkipsSourcePolledTooRecently(t *testing.T) {
recent := time.Now().Add(-30 * time.Second)
interval := 600
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{
ID: "s1", InstallationID: 1, CapabilityID: "arr", ConnectionID: strptr("c1"), Enabled: true,
PollIntervalSeconds: &interval, LastRunAt: &recent,
}},
}
prov := &fakeProvider{paths: map[string][]string{"arr": {"/mnt/media/Show/S01/E01.mkv"}}, nextMarker: "m1"}
q := &recordingQueuer{}
svc := newService(store, prov, q, allowSuppressor{})
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if len(q.enqueued) != 0 {
t.Fatalf("a source within its poll interval must be skipped, got %d enqueued", len(q.enqueued))
}
if _, ok := store.advanced["s1"]; ok {
t.Fatalf("marker must NOT advance for a skipped (too-recent) source")
}
}
func TestPollOnceRunsSourcePastItsInterval(t *testing.T) {
old := time.Now().Add(-20 * time.Minute)
interval := 600 // 10 min; last run was 20 min ago => eligible
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{{
ID: "s1", InstallationID: 1, CapabilityID: "arr", ConnectionID: strptr("c1"), Enabled: true,
PollIntervalSeconds: &interval, LastRunAt: &old,
}},
}
prov := &fakeProvider{paths: map[string][]string{"arr": {"/mnt/media/Show/S01/E01.mkv"}}, nextMarker: "m1"}
q := &recordingQueuer{}
svc := newService(store, prov, q, allowSuppressor{})
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if len(q.enqueued) != 1 {
t.Fatalf("a source past its interval must poll, got %d enqueued", len(q.enqueued))
}
if _, ok := store.advanced["s1"]; !ok {
t.Fatalf("expected marker advanced for s1")
}
}
func TestPollOnceSkipsOrphanedSourceQuietly(t *testing.T) {
// Two enabled sources: only "arr-live" is still discovered; "arr-gone"'s
// plugin has been uninstalled. The orphan must be skipped quietly — no poll,
// no RecordError spam — while the live source still polls normally.
store := &fakeStore{
settings: Settings{Enabled: true, DefaultPollIntervalSeconds: 600, DebounceSeconds: 60},
sources: []Source{
{ID: "live", InstallationID: 1, CapabilityID: "arr-live", ConnectionID: strptr("c1"), Enabled: true},
{ID: "gone", InstallationID: 2, CapabilityID: "arr-gone", ConnectionID: strptr("c1"), Enabled: true},
},
}
prov := &fakeProvider{paths: map[string][]string{
"arr-live": {"/mnt/media/Show/S01/E01.mkv"},
"arr-gone": {"/mnt/media/Other/S01/E01.mkv"},
}, nextMarker: "m1"}
q := &recordingQueuer{}
// Lister reports only the live capability; the orphan is absent.
lister := fakeLister{sources: []DiscoveredSource{{InstallationID: 1, CapabilityID: "arr-live"}}}
svc := NewService(store, prov, passthroughConnRes{}, fakeResolver{}, q, allowSuppressor{}, lister)
if err := svc.PollOnce(context.Background()); err != nil {
t.Fatalf("PollOnce: %v", err)
}
if len(q.enqueued) != 1 {
t.Fatalf("only the live source should enqueue, got %d: %+v", len(q.enqueued), q.enqueued)
}
if _, ok := store.advanced["live"]; !ok {
t.Fatalf("expected live source marker advanced")
}
if _, ok := store.advanced["gone"]; ok {
t.Fatalf("orphaned source must NOT poll/advance")
}
if _, ok := store.recorded["gone"]; ok {
t.Fatalf("orphaned source must be skipped quietly, but an error was recorded: %q", store.recorded["gone"])
}
}
internal/autoscan/suggest.go
+186
View File
@@ -0,0 +1,186 @@
package autoscan
import (
"context"
"errors"
"fmt"
"strings"
)
// ErrNoConnection is returned when the rewrite suggester is asked for a source
// that has no bound connection. The admin API surfaces it as a 400.
var ErrNoConnection = errors.New("autoscan: source has no bound connection")
// SuggestRewrites resolves the source's bound connection, lists the arr root
// folders and the Silo media-folder paths, and suffix-matches arr roots to Silo
// folders to propose path rewrites. The source's existing rewrites are passed so
// already-covered roots are reported separately. Returns ErrNoConnection when
// the source has no bound connection (the handler maps it to 400).
func (s *Service) SuggestRewrites(ctx context.Context, sourceID string) (RewriteSuggestions, error) {
if s.rootFolders == nil || s.folders == nil {
return RewriteSuggestions{}, fmt.Errorf("autoscan: rewrite suggester not configured")
}
src, err := s.store.GetSource(ctx, sourceID)
if err != nil {
return RewriteSuggestions{}, err
}
if src.ConnectionID == nil {
return RewriteSuggestions{}, ErrNoConnection
}
conn, err := s.store.GetConnection(ctx, *src.ConnectionID)
if err != nil {
return RewriteSuggestions{}, err
}
resolved, err := s.connres.Resolve(ctx, conn)
if err != nil {
return RewriteSuggestions{}, fmt.Errorf("autoscan: resolve connection: %w", err)
}
arrRoots, err := s.rootFolders.RootFolders(ctx, resolved.BaseURL, resolved.APIKey)
if err != nil {
return RewriteSuggestions{}, fmt.Errorf("autoscan: list arr root folders: %w", err)
}
siloFolders, err := s.folders.ListFolderPaths(ctx)
if err != nil {
return RewriteSuggestions{}, fmt.Errorf("autoscan: list silo folders: %w", err)
}
return suggestRewrites(arrRoots, siloFolders, src.PathRewrites), nil
}
// RewriteSuggestions is the result of matching arr root folders to Silo folders.
type RewriteSuggestions struct {
Proposed []ProposedRewrite `json:"proposed"`
Unmatched []string `json:"unmatched"`
Ambiguous []AmbiguousRoot `json:"ambiguous"`
Covered []string `json:"covered"`
}
// ProposedRewrite is a suggested rewrite plus its confidence (shared trailing segments).
type ProposedRewrite struct {
From string `json:"from"`
To string `json:"to"`
MatchDepth int `json:"match_depth"`
}
// AmbiguousRoot is an arr root that tied across multiple Silo folders.
type AmbiguousRoot struct {
Root string `json:"root"`
Candidates []string `json:"candidates"`
}
// normalizePath makes a path comparable: backslashes -> '/', collapse duplicate
// slashes, strip a trailing slash (but keep a bare "/").
func normalizePath(p string) string {
p = normalizeSeparators(strings.TrimSpace(p))
for strings.Contains(p, "//") {
p = strings.ReplaceAll(p, "//", "/")
}
if len(p) > 1 {
p = strings.TrimRight(p, "/")
}
return p
}
func segments(p string) []string {
p = strings.Trim(p, "/")
if p == "" {
return nil
}
return strings.Split(p, "/")
}
// commonSuffixLen counts equal trailing segments (full-segment equality).
func commonSuffixLen(a, b []string) int {
i, j, n := len(a)-1, len(b)-1, 0
for i >= 0 && j >= 0 && a[i] == b[j] {
n++
i--
j--
}
return n
}
// coveredBy reports whether an existing rewrite already matches root (same
// boundary rule as applyRewrites). The existing From is normalized too, so a
// stored Windows/dup-slash rewrite still covers a normalized root.
func coveredBy(root string, existing []PathRewrite) bool {
for _, rw := range existing {
from := normalizePath(rw.From)
if from == "" {
continue
}
if root == from || strings.HasPrefix(root, from+"/") {
return true
}
}
return false
}
// suggestRewrites matches each arr root to the Silo folder sharing the most
// trailing path segments (unique winner). Pure: no I/O, no deployment constants.
func suggestRewrites(arrRoots, siloFolderPaths []string, existing []PathRewrite) RewriteSuggestions {
siloNorm := make([]string, 0, len(siloFolderPaths))
siloSegs := make([][]string, 0, len(siloFolderPaths))
siloSeen := make(map[string]struct{})
for _, p := range siloFolderPaths {
n := normalizePath(p)
if n == "" {
continue
}
if _, dup := siloSeen[n]; dup { // dedup Silo paths so candidates are distinct
continue
}
siloSeen[n] = struct{}{}
siloNorm = append(siloNorm, n)
siloSegs = append(siloSegs, segments(n))
}
// Initialize to non-nil so the JSON response always has arrays ([]) rather
// than null — the frontend maps over these directly.
out := RewriteSuggestions{
Proposed: []ProposedRewrite{},
Unmatched: []string{},
Ambiguous: []AmbiguousRoot{},
Covered: []string{},
}
rootSeen := make(map[string]struct{})
for _, raw := range arrRoots {
root := normalizePath(raw)
if root == "" {
continue
}
if _, dup := rootSeen[root]; dup { // dedup arr roots that normalize alike
continue
}
rootSeen[root] = struct{}{}
if coveredBy(root, existing) {
out.Covered = append(out.Covered, root)
continue
}
rootSegs := segments(root)
best := 0
var winners []string
for i, segs := range siloSegs {
n := commonSuffixLen(rootSegs, segs)
if n == 0 {
continue
}
if n > best {
best, winners = n, []string{siloNorm[i]}
} else if n == best {
winners = append(winners, siloNorm[i])
}
}
switch {
case best == 0:
out.Unmatched = append(out.Unmatched, root)
case len(winners) == 1:
if winners[0] == root {
continue // arr path already equals the Silo path — no rewrite needed
}
out.Proposed = append(out.Proposed, ProposedRewrite{From: root, To: winners[0], MatchDepth: best})
default:
out.Ambiguous = append(out.Ambiguous, AmbiguousRoot{Root: root, Candidates: winners})
}
}
return out
}
internal/autoscan/suggest_deps.go
+81
View File
@@ -0,0 +1,81 @@
package autoscan
import (
"context"
"net/http"
"time"
"github.com/Silo-Server/silo-server/internal/catalog"
"github.com/Silo-Server/silo-server/internal/requests/arrclient"
)
// rootFolderTimeout is generous: Radarr/Sonarr compute unmappedFolders by
// scanning every root folder (often slow network shares), so a large library's
// /api/v3/rootfolder can take 20-30s+ — well past arrclient's 30s default.
const rootFolderTimeout = 2 * time.Minute
type arrRootFolderClient struct{ httpClient *http.Client }
// NewArrRootFolderClient returns a RootFolderClient backed by the shared arrclient.
// When no client is given it uses a long timeout, since the root-folder scan is slow.
func NewArrRootFolderClient(httpClient *http.Client) RootFolderClient {
if httpClient == nil {
httpClient = &http.Client{Timeout: rootFolderTimeout}
}
return &arrRootFolderClient{httpClient: httpClient}
}
func (c *arrRootFolderClient) RootFolders(ctx context.Context, baseURL, apiKey string) ([]string, error) {
client := arrclient.New(baseURL, apiKey, c.httpClient)
folders, err := arrclient.ListRootFolders(ctx, client)
if err != nil {
return nil, err
}
paths := make([]string, 0, len(folders))
for _, f := range folders {
if f.Path != "" {
paths = append(paths, f.Path)
}
}
return paths, nil
}
// statusTimeout bounds the connection-test probe: /api/v3/system/status is a
// cheap call, so a short timeout keeps a misconfigured/unreachable target from
// hanging the request.
const statusTimeout = 10 * time.Second
// SystemStatus probes the arr /api/v3/system/status endpoint and returns the
// reported version. It satisfies ArrStatusProbe. A dedicated short-timeout
// client is used (the suggester's long root-folder timeout is inappropriate for
// a liveness probe).
func (c *arrRootFolderClient) SystemStatus(ctx context.Context, baseURL, apiKey string) (string, error) {
httpClient := &http.Client{Timeout: statusTimeout}
client := arrclient.New(baseURL, apiKey, httpClient)
var status struct {
Version string `json:"version"`
}
if err := client.GetJSON(ctx, "/api/v3/system/status", &status); err != nil {
return "", err
}
return status.Version, nil
}
type catalogFolderLister struct{ repo *catalog.FolderRepository }
// NewCatalogFolderLister adapts catalog.FolderRepository to FolderLister.
func NewCatalogFolderLister(repo *catalog.FolderRepository) FolderLister {
return &catalogFolderLister{repo: repo}
}
func (l *catalogFolderLister) ListFolderPaths(ctx context.Context) ([]string, error) {
folders, err := l.repo.List(ctx)
if err != nil {
return nil, err
}
var paths []string
for _, f := range folders {
paths = append(paths, f.Paths...)
}
return paths, nil
}
internal/autoscan/suppress.go
+43
View File
@@ -0,0 +1,43 @@
package autoscan
import (
"context"
"time"
"github.com/redis/go-redis/v9"
)
// Suppressor prevents re-enqueuing a scan for the same (folder, path) target
// within a window. The key must match the scanqueue dedup granularity
// (folder ID + path) so two distinct subtrees under one library folder are
// not collapsed into a single suppression claim.
type Suppressor interface {
// ShouldScan atomically claims the target for scanning: returns true and sets
// a TTL key if no claim exists, false if a recent claim is still live.
ShouldScan(ctx context.Context, key string, ttl time.Duration) (bool, error)
// Release drops a suppression claim (used when the scan enqueue fails).
Release(ctx context.Context, key string) error
}
type redisSuppressor struct{ client *redis.Client }
func NewRedisSuppressor(client *redis.Client) Suppressor { return &redisSuppressor{client: client} }
func (s *redisSuppressor) ShouldScan(ctx context.Context, key string, ttl time.Duration) (bool, error) {
if s.client == nil || ttl <= 0 {
return true, nil // no suppression configured -> always scan
}
redisKey := "autoscan:scanned:" + key
ok, err := s.client.SetNX(ctx, redisKey, "1", ttl).Result()
if err != nil {
return true, nil // fail open: a Redis hiccup should not block scanning
}
return ok, nil
}
func (s *redisSuppressor) Release(ctx context.Context, key string) error {
if s.client == nil {
return nil
}
return s.client.Del(ctx, "autoscan:scanned:"+key).Err()
}
internal/autoscan/types.go
+167
View File
@@ -0,0 +1,167 @@
package autoscan
import "time"
// Settings is the global autoscan configuration (singleton row).
type Settings struct {
Enabled bool
DefaultPollIntervalSeconds int
DebounceSeconds int
}
// Connection is an arr server the host can reach: either own credentials, or a
// live reference to a Requests integration (RequestIntegrationID set).
type Connection struct {
ID string
Name string
Kind string
BaseURL string // own; empty when linked
APIKeyRef string // own; empty when linked
RequestIntegrationID *string
}
// PathRewrite is a per-source prefix translation from a raw arr/source-namespace
// path to a Silo-native path. The merged scan_source contract moved rewrite
// ownership from the plugin to the host, so these are applied host-side before a
// raw path is resolved/enqueued.
type PathRewrite struct {
From string `json:"from"`
To string `json:"to"`
}
// ChangeScope tells the host how to interpret a scan_source path.
type ChangeScope string
const (
ChangeScopeAuto ChangeScope = "auto"
ChangeScopeFile ChangeScope = "file"
ChangeScopeSubtree ChangeScope = "subtree"
)
// Change is one raw provider/source-namespace path returned by a scan_source
// plugin. The host applies PathRewrites before resolving the changed path.
type Change struct {
SourcePath string
Scope ChangeScope
}
// Source ties a scan_source plugin capability instance to a connection plus the
// host-owned scheduling/bookkeeping state.
type Source struct {
ID string
InstallationID int
CapabilityID string
ConnectionID *string // nil until an operator binds a connection
Enabled bool
PollIntervalSeconds *int // nil => use settings default
PathRewrites []PathRewrite // host-owned raw->Silo prefix rewrites
SourceConfig map[string]string
Label string // operator-set display label; "" = unset
Marker *string // opaque; nil on first run
LastRunAt *time.Time
LastError *string
}
type EventStatus string
const (
EventStatusSuccess EventStatus = "success"
EventStatusError EventStatus = "error"
EventStatusUnresolved EventStatus = "unresolved"
)
type Event struct {
ID int64
SourceID *string
InstallationID int
CapabilityID string
StartedAt time.Time
CompletedAt time.Time
DurationMS int64
Status EventStatus
ChangesReturned int
ChangesResolved int
TargetsClaimed int
ScansCreated int
ScansReused int
ScansSuppressed int
ErrorMessage string
MarkerBefore *string
MarkerAfter *string
}
type EventCreate struct {
SourceID string
InstallationID int
CapabilityID string
StartedAt time.Time
MarkerBefore string
}
type EventFinish struct {
ID int64
CompletedAt time.Time
Status EventStatus
ChangesReturned int
ChangesResolved int
TargetsClaimed int
ScansCreated int
ScansReused int
ScansSuppressed int
ErrorMessage string
MarkerAfter string
}
type EnqueueResult struct {
Created int
Reused int
}
type EventListFilter struct {
SourceID string
Status EventStatus
Search string
Limit int
Offset int
}
type ScanListFilter struct {
Status string
Search string
Limit int
Offset int
}
type EventWithRuns struct {
Event Event
Runs []ScanRunSummary
}
type ScanRunSummary struct {
ID string
MediaFolderID int
Mode string
Path string
Trigger string
Status string
ErrorMessage string
RequestedAt *time.Time
StartedAt *time.Time
CompletedAt *time.Time
}
type ScanWithEvent struct {
ScanRunSummary
AutoscanEventID *int64
SourceID *string
InstallationID *int
CapabilityID string
EventStatus EventStatus
EventCompletedAt *time.Time
}
type QueueSummary struct {
Active int
Accepted int
Running int
}
internal/models/scan_run.go
+14 -13
View File
@@ -7,17 +7,18 @@ import (
// ScanRun represents a persisted library scan orchestration row.
type ScanRun struct {
ID string
MediaFolderID int
Mode string
Path string
Trigger string
Status string
ResultPayload json.RawMessage
ErrorMessage string
RequestedAt time.Time
StartedAt *time.Time
CompletedAt *time.Time
HeartbeatAt *time.Time
UpdatedAt time.Time
ID string
MediaFolderID int
Mode string
Path string
Trigger string
Status string
ResultPayload json.RawMessage
ErrorMessage string
AutoscanEventID *int64
RequestedAt time.Time
StartedAt *time.Time
CompletedAt *time.Time
HeartbeatAt *time.Time
UpdatedAt time.Time
}
internal/pluginhost/client.go
+21
View File
@@ -44,6 +44,11 @@ type ScheduledTaskClient struct {
timeout time.Duration
}
type ScanSourceClient struct {
client pluginv1.ScanSourceClient
timeout time.Duration
}
type EventConsumerClient struct {
client pluginv1.EventConsumerClient
timeout time.Duration
@@ -118,6 +123,16 @@ func (c *Client) ScheduledTask(capabilityID string) (*ScheduledTaskClient, error
}, nil
}
func (c *Client) ScanSource(capabilityID string) (*ScanSourceClient, error) {
if err := c.requireCapability("scan_source.v1", capabilityID); err != nil {
return nil, err
}
return &ScanSourceClient{
client: c.rpc.ScanSource(),
timeout: DefaultScanSourceTimeout,
}, nil
}
func (c *Client) EventConsumer(capabilityID string) (*EventConsumerClient, error) {
if err := c.requireCapability("event_consumer.v1", capabilityID); err != nil {
return nil, err
@@ -229,6 +244,12 @@ func (c *ScheduledTaskClient) Run(ctx context.Context, req *pluginv1.RunSchedule
return c.client.Run(callCtx, req)
}
func (c *ScanSourceClient) PollChanges(ctx context.Context, req *pluginv1.PollChangesRequest) (*pluginv1.PollChangesResponse, error) {
callCtx, cancel := ensureDeadline(ctx, c.timeout)
defer cancel()
return c.client.PollChanges(callCtx, req)
}
func (c *EventConsumerClient) HandleEvent(ctx context.Context, req *pluginv1.HandleEventRequest) (*pluginv1.HandleEventResponse, error) {
callCtx, cancel := ensureDeadline(ctx, c.timeout)
defer cancel()
internal/pluginhost/client_test.go
+107
View File
@@ -0,0 +1,107 @@
package pluginhost
import (
"errors"
"testing"
pluginv1 "github.com/Silo-Server/silo-plugin-sdk/pkg/pluginproto/silo/plugin/v1"
sdkruntime "github.com/Silo-Server/silo-plugin-sdk/pkg/pluginsdk/runtime"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
// makeTestClient constructs a Client with the given declared capabilities.
// It creates a lazy (non-connecting) gRPC ClientConn so that accessors which
// call c.rpc.<Capability>() do not panic — no real connection is made.
func makeTestClient(t *testing.T, capabilities []*pluginv1.CapabilityDescriptor) *Client {
t.Helper()
conn, err := grpc.NewClient("passthrough:///bufnet",
grpc.WithTransportCredentials(insecure.NewCredentials()),
)
if err != nil {
t.Fatalf("grpc.NewClient: %v", err)
}
t.Cleanup(func() { _ = conn.Close() })
rpc := sdkruntime.NewClient(conn)
manifest := &pluginv1.PluginManifest{Capabilities: capabilities}
return newClient(0, rpc, manifest)
}
func TestClient_ScheduledTask_CapabilityGate(t *testing.T) {
t.Run("absent capability returns error", func(t *testing.T) {
c := makeTestClient(t, nil)
_, err := c.ScheduledTask("missing")
if err == nil {
t.Fatal("expected error for missing scheduled_task.v1 capability, got nil")
}
if !errors.Is(err, ErrCapabilityNotFound) {
t.Errorf("expected ErrCapabilityNotFound, got %v", err)
}
})
t.Run("declared capability returns no error", func(t *testing.T) {
c := makeTestClient(t, []*pluginv1.CapabilityDescriptor{
{Type: "scheduled_task.v1", Id: "nightly"},
})
got, err := c.ScheduledTask("nightly")
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if got == nil {
t.Fatal("expected non-nil ScheduledTaskClient")
}
})
t.Run("wrong id returns error", func(t *testing.T) {
c := makeTestClient(t, []*pluginv1.CapabilityDescriptor{
{Type: "scheduled_task.v1", Id: "nightly"},
})
_, err := c.ScheduledTask("weekly")
if err == nil {
t.Fatal("expected error for mismatched scheduled_task.v1 capability id, got nil")
}
if !errors.Is(err, ErrCapabilityNotFound) {
t.Errorf("expected ErrCapabilityNotFound, got %v", err)
}
})
}
func TestClient_ScanSource_CapabilityGate(t *testing.T) {
t.Run("absent capability returns error", func(t *testing.T) {
c := makeTestClient(t, nil)
_, err := c.ScanSource("missing")
if err == nil {
t.Fatal("expected error for missing scan_source.v1 capability, got nil")
}
if !errors.Is(err, ErrCapabilityNotFound) {
t.Errorf("expected ErrCapabilityNotFound, got %v", err)
}
})
t.Run("declared capability returns no error", func(t *testing.T) {
c := makeTestClient(t, []*pluginv1.CapabilityDescriptor{
{Type: "scan_source.v1", Id: "arr"},
})
got, err := c.ScanSource("arr")
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if got == nil {
t.Fatal("expected non-nil ScanSourceClient")
}
})
t.Run("wrong id returns error", func(t *testing.T) {
c := makeTestClient(t, []*pluginv1.CapabilityDescriptor{
{Type: "scan_source.v1", Id: "arr"},
})
_, err := c.ScanSource("inotify")
if err == nil {
t.Fatal("expected error for mismatched scan_source.v1 capability id, got nil")
}
if !errors.Is(err, ErrCapabilityNotFound) {
t.Errorf("expected ErrCapabilityNotFound, got %v", err)
}
})
}
internal/pluginhost/handshake.go
+8 -3
View File
@@ -14,9 +14,14 @@ const (
DefaultMetadataTimeout = 30 * time.Second
DefaultAnalyzerTimeout = 5 * time.Minute
DefaultControlTimeout = 10 * time.Second
DefaultEventTimeout = 10 * time.Second
DefaultAuthTimeout = 10 * time.Second
DefaultRouteTimeout = 10 * time.Second
// DefaultScanSourceTimeout covers a single PollChanges call. Most scan
// sources return quickly, but filesystem-backed sources may need a longer
// bounded window for an initial baseline; cooperative plugins should still
// checkpoint progress and return before this deadline.
DefaultScanSourceTimeout = 5 * time.Minute
DefaultEventTimeout = 10 * time.Second
DefaultAuthTimeout = 10 * time.Second
DefaultRouteTimeout = 10 * time.Second
)
func HandshakeConfig() plugin.HandshakeConfig {
internal/plugins/service.go
+13
View File
@@ -23,6 +23,7 @@ type pluginClient interface {
MetadataProvider(capabilityID string) (*pluginhost.MetadataProviderClient, error)
MediaAnalyzer(capabilityID string) (*pluginhost.MediaAnalyzerClient, error)
ScheduledTask(capabilityID string) (*pluginhost.ScheduledTaskClient, error)
ScanSource(capabilityID string) (*pluginhost.ScanSourceClient, error)
EventConsumer(capabilityID string) (*pluginhost.EventConsumerClient, error)
AuthProvider(capabilityID string) (*pluginhost.AuthProviderClient, error)
HTTPRoutes(capabilityID string) (*pluginhost.HTTPRoutesClient, error)
@@ -422,6 +423,18 @@ func (s *Service) ScheduledTaskClient(
return client.ScheduledTask(capabilityID)
}
func (s *Service) ScanSourceClient(
ctx context.Context,
installationID int,
capabilityID string,
) (*pluginhost.ScanSourceClient, error) {
client, err := s.ensureClient(ctx, installationID)
if err != nil {
return nil, err
}
return client.ScanSource(capabilityID)
}
func (s *Service) EventConsumerClient(
ctx context.Context,
installationID int,
internal/plugins/service_hot_reload_test.go
+4
View File
@@ -207,6 +207,10 @@ func (f *fakePluginClient) ScheduledTask(string) (*pluginhost.ScheduledTaskClien
return nil, nil
}
func (f *fakePluginClient) ScanSource(string) (*pluginhost.ScanSourceClient, error) {
return nil, nil
}
func (f *fakePluginClient) EventConsumer(string) (*pluginhost.EventConsumerClient, error) {
return nil, nil
}
internal/scanner/scanner.go
+75 -13
View File
@@ -504,6 +504,7 @@ func (s *Scanner) scanPaths(
var wg sync.WaitGroup
pathCh := make(chan string, s.workers)
var newCount, updatedCount, unchangedCount, errorCount, processedCount atomic.Int64
subtitleCache := newExternalSubtitleDirCache()
for range s.workers {
wg.Go(func() {
@@ -511,7 +512,7 @@ func (s *Scanner) scanPaths(
if ctx.Err() != nil {
continue // drain channel
}
action, updateReasons, processErr := s.processFile(ctx, path, folder, existingByPath, existingContentStatuses, rootInference.Assignments[path], groupInference.Assignments[path])
action, updateReasons, processErr := s.processFile(ctx, path, folder, existingByPath, existingContentStatuses, rootInference.Assignments[path], groupInference.Assignments[path], subtitleCache)
if processErr != nil {
slog.Error("scanner: file processing failed", "path", path, "error", processErr)
errorCount.Add(1)
@@ -980,6 +981,7 @@ func (s *Scanner) scanScope(
var wg sync.WaitGroup
pathCh := make(chan string, s.workers)
var newCount, updatedCount, unchangedCount, errorCount, processedCount atomic.Int64
subtitleCache := newExternalSubtitleDirCache()
for range s.workers {
wg.Go(func() {
@@ -987,7 +989,7 @@ func (s *Scanner) scanScope(
if ctx.Err() != nil {
continue
}
action, updateReasons, processErr := s.processFile(ctx, path, folder, existingByPath, existingContentStatuses, rootInference.Assignments[path], groupInference.Assignments[path])
action, updateReasons, processErr := s.processFile(ctx, path, folder, existingByPath, existingContentStatuses, rootInference.Assignments[path], groupInference.Assignments[path], subtitleCache)
if processErr != nil {
slog.Error("scanner: file processing failed", "path", path, "error", processErr)
errorCount.Add(1)
@@ -1399,7 +1401,7 @@ func (s *Scanner) ScanFile(ctx context.Context, filePath string, folder *models.
return fmt.Errorf("loading group overrides for file: %w", err)
}
applyGroupOverrides(&groupInference, groupOverrides)
_, _, err = s.processFile(ctx, filePath, folder, existingByPath, existingContentStatuses, rootInference.Assignments[filepath.Clean(filePath)], groupInference.Assignments[filepath.Clean(filePath)])
_, _, err = s.processFile(ctx, filePath, folder, existingByPath, existingContentStatuses, rootInference.Assignments[filepath.Clean(filePath)], groupInference.Assignments[filepath.Clean(filePath)], newExternalSubtitleDirCache())
if err != nil {
return err
}
@@ -1541,6 +1543,7 @@ func (s *Scanner) processFile(
existingContentStatuses map[string]string,
assignment fileRootAssignment,
groupAssignment fileGroupAssignment,
subtitleCache *externalSubtitleDirCache,
) (fileAction, []string, error) {
// Stat the file.
info, err := os.Stat(filePath)
@@ -1550,13 +1553,33 @@ func (s *Scanner) processFile(
fileSize := info.Size()
fileModifiedAt := normalizeFileModifiedAt(info.ModTime())
var externalSubs []ExternalSubtitleInfo
externalSubsLoaded := false
externalSubsChecked := false
loadExternalSubs := func() []ExternalSubtitleInfo {
if externalSubsLoaded {
return externalSubs
}
externalSubsLoaded = true
var subErr error
externalSubs, subErr = subtitleCache.Detect(filePath)
if subErr != nil {
slog.Warn("scanner: subtitle detection failed", "path", filePath, "error", subErr)
externalSubs = nil
externalSubsChecked = false
return externalSubs
}
externalSubsChecked = true
return externalSubs
}
// Check if unchanged (same size) only when playback-critical probe data
// is already present or this scanner cannot repair it on this node. This
// lets rescans repair legacy rows when local ffprobe metadata is available
// without forcing endless rewrites on nodes that cannot probe.
if existing, ok := existingByPath[filePath]; ok {
updateReasons := scanStateUpdateReasons(existing, fileSize, fileModifiedAt, assignment, groupAssignment, folder.Type, s.ffprobePath != "")
currentExternalSubtitlePaths := externalSubtitleInfoPaths(loadExternalSubs())
updateReasons := scanStateUpdateReasons(existing, fileSize, fileModifiedAt, currentExternalSubtitlePaths, externalSubsChecked, assignment, groupAssignment, folder.Type, s.ffprobePath != "")
if shouldSkipStableConfirmedScanState(existing, existingContentStatuses[existing.ContentID], fileSize, fileModifiedAt, updateReasons, s.ffprobePath != "") {
return actionUnchanged, nil, nil
}
@@ -1572,10 +1595,7 @@ func (s *Scanner) processFile(
probe, probeSource := s.probeFile(ctx, filePath)
// Detect external subtitles.
externalSubs, subErr := DetectExternalSubtitles(filePath)
if subErr != nil {
slog.Warn("scanner: subtitle detection failed", "path", filePath, "error", subErr)
}
externalSubs = loadExternalSubs()
// Check for intro/credits markers from S3.
markerFetcher := s.markerFetcher
@@ -1696,10 +1716,7 @@ func (s *Scanner) processFile(
probe, probeSource := s.probeFile(ctx, filePath)
// Detect external subtitles.
externalSubs, subErr := DetectExternalSubtitles(filePath)
if subErr != nil {
slog.Warn("scanner: subtitle detection failed", "path", filePath, "error", subErr)
}
externalSubs = loadExternalSubs()
// Check for intro/credits markers from S3.
markerFetcher := s.markerFetcher
@@ -1817,6 +1834,8 @@ func scanStateUpdateReasons(
existing *scanStateFile,
fileSize int64,
fileModifiedAt time.Time,
currentExternalSubtitlePaths []string,
externalSubtitlesChecked bool,
assignment fileRootAssignment,
groupAssignment fileGroupAssignment,
libraryType string,
@@ -1839,7 +1858,11 @@ func scanStateUpdateReasons(
if canRepairProbe && needsCriticalProbeRepairScanState(existing) {
reasons = append(reasons, "probe_repair")
}
if hasMissingExternalSubtitlePath(existing.ExternalSubtitlePaths) {
if externalSubtitlesChecked {
if !sameStringSet(existing.ExternalSubtitlePaths, currentExternalSubtitlePaths) {
reasons = append(reasons, "external_subtitle_changed")
}
} else if hasMissingExternalSubtitlePath(existing.ExternalSubtitlePaths) {
reasons = append(reasons, "external_subtitle_missing")
}
if scanStateRootAssignmentChanged(existing, assignment, libraryType) {
@@ -1851,6 +1874,45 @@ func scanStateUpdateReasons(
return reasons
}
func externalSubtitleInfoPaths(subtitles []ExternalSubtitleInfo) []string {
paths := make([]string, 0, len(subtitles))
for _, subtitle := range subtitles {
path := strings.TrimSpace(subtitle.Path)
if path == "" {
continue
}
paths = append(paths, filepath.Clean(path))
}
return paths
}
func sameStringSet(left, right []string) bool {
seen := make(map[string]int, len(left))
for _, value := range left {
value = filepath.Clean(strings.TrimSpace(value))
if value == "." || value == "" {
continue
}
seen[value]++
}
for _, value := range right {
value = filepath.Clean(strings.TrimSpace(value))
if value == "." || value == "" {
continue
}
if seen[value] == 0 {
return false
}
seen[value]--
}
for _, count := range seen {
if count != 0 {
return false
}
}
return true
}
func hasMissingExternalSubtitlePath(paths []string) bool {
for _, path := range paths {
if strings.TrimSpace(path) == "" {
internal/scanner/scanner_test.go
+32 -1
View File
@@ -112,7 +112,7 @@ func TestScanStateUpdateReasons_DetectsMissingExternalSubtitle(t *testing.T) {
ExternalSubtitlePaths: []string{filepath.Join(t.TempDir(), "Movie.en.srt")},
}
reasons := scanStateUpdateReasons(existing, 1_000, modifiedAt, fileRootAssignment{}, fileGroupAssignment{}, "movies", false)
reasons := scanStateUpdateReasons(existing, 1_000, modifiedAt, nil, false, fileRootAssignment{}, fileGroupAssignment{}, "movies", false)
if !testStringSliceContains(reasons, "external_subtitle_missing") {
t.Fatalf("expected external_subtitle_missing reason, got %#v", reasons)
}
@@ -121,6 +121,37 @@ func TestScanStateUpdateReasons_DetectsMissingExternalSubtitle(t *testing.T) {
}
}
func TestScanStateUpdateReasons_DetectsExternalSubtitleInventoryChange(t *testing.T) {
t.Parallel()
dir := t.TempDir()
modifiedAt := time.Now().UTC().Truncate(time.Microsecond)
existing := &scanStateFile{
ContentID: "matched-content",
FileSize: 1_000,
FileModifiedAt: &modifiedAt,
ExternalSubtitlePaths: []string{filepath.Join(dir, "Movie.en.srt")},
}
reasons := scanStateUpdateReasons(
existing,
1_000,
modifiedAt,
[]string{filepath.Join(dir, "Movie.en.srt"), filepath.Join(dir, "Movie.fr.srt")},
true,
fileRootAssignment{},
fileGroupAssignment{},
"movies",
false,
)
if !testStringSliceContains(reasons, "external_subtitle_changed") {
t.Fatalf("expected external_subtitle_changed reason, got %#v", reasons)
}
if shouldSkipStableConfirmedScanState(existing, "matched", 1_000, modifiedAt, reasons, false) {
t.Fatal("expected external subtitle change to bypass stable scan-state skip")
}
}
func testStringSliceContains(values []string, target string) bool {
for _, value := range values {
if value == target {
internal/scanner/subtitles.go
+51 -2
View File
@@ -5,6 +5,7 @@ import (
"os"
"path/filepath"
"strings"
"sync"
"github.com/Silo-Server/silo-server/internal/lang"
)
@@ -18,6 +19,49 @@ var subtitleExtensions = map[string]bool{
".sub": true,
}
type externalSubtitleDirCache struct {
mu sync.Mutex
dirs map[string]externalSubtitleDirListing
}
type externalSubtitleDirListing struct {
entries []os.DirEntry
err error
}
func newExternalSubtitleDirCache() *externalSubtitleDirCache {
return &externalSubtitleDirCache{
dirs: make(map[string]externalSubtitleDirListing),
}
}
func (c *externalSubtitleDirCache) Detect(mediaFilePath string) ([]ExternalSubtitleInfo, error) {
if c == nil {
return DetectExternalSubtitles(mediaFilePath)
}
dir := filepath.Dir(mediaFilePath)
c.mu.Lock()
listing, ok := c.dirs[dir]
c.mu.Unlock()
if !ok {
entries, err := os.ReadDir(dir)
listing = externalSubtitleDirListing{entries: entries, err: err}
c.mu.Lock()
if existing, found := c.dirs[dir]; found {
listing = existing
} else {
c.dirs[dir] = listing
}
c.mu.Unlock()
}
if listing.err != nil {
return nil, fmt.Errorf("subtitles: read dir %s: %w", dir, listing.err)
}
return externalSubtitlesFromEntries(mediaFilePath, listing.entries), nil
}
// DetectExternalSubtitles scans the directory containing the media file for
// sidecar subtitle files that match the media file's basename.
//
@@ -30,13 +74,18 @@ var subtitleExtensions = map[string]bool{
// Movie.forced.eng.srt -> language: "eng", forced: true
func DetectExternalSubtitles(mediaFilePath string) ([]ExternalSubtitleInfo, error) {
dir := filepath.Dir(mediaFilePath)
mediaBase := stripExtension(filepath.Base(mediaFilePath))
entries, err := os.ReadDir(dir)
if err != nil {
return nil, fmt.Errorf("subtitles: read dir %s: %w", dir, err)
}
return externalSubtitlesFromEntries(mediaFilePath, entries), nil
}
func externalSubtitlesFromEntries(mediaFilePath string, entries []os.DirEntry) []ExternalSubtitleInfo {
dir := filepath.Dir(mediaFilePath)
mediaBase := stripExtension(filepath.Base(mediaFilePath))
var results []ExternalSubtitleInfo
for _, entry := range entries {
@@ -75,7 +124,7 @@ func DetectExternalSubtitles(mediaFilePath string) ([]ExternalSubtitleInfo, erro
results = append(results, info)
}
return results, nil
return results
}
// parseSuffix extracts the language and forced flag from the suffix portion
internal/scanqueue/repository.go
+13 -9
View File
@@ -33,10 +33,11 @@ const (
var ErrScanRunNotFound = errors.New("scan run not found")
type CreateInput struct {
LibraryID int
Mode string
Path string
Trigger string
LibraryID int
Mode string
Path string
Trigger string
AutoscanEventID *int64
}
type Repository struct {
@@ -48,7 +49,7 @@ func NewRepository(pool *pgxpool.Pool) *Repository {
}
const scanRunColumns = `id, media_folder_id, mode, path, trigger, status, result_payload,
error_message, requested_at, started_at, completed_at, heartbeat_at, updated_at`
error_message, autoscan_event_id, requested_at, started_at, completed_at, heartbeat_at, updated_at`
func scanRunRow(row pgx.Row) (*models.ScanRun, error) {
var run models.ScanRun
@@ -61,6 +62,7 @@ func scanRunRow(row pgx.Row) (*models.ScanRun, error) {
&run.Status,
&run.ResultPayload,
&run.ErrorMessage,
&run.AutoscanEventID,
&run.RequestedAt,
&run.StartedAt,
&run.CompletedAt,
@@ -95,8 +97,8 @@ func scanRunRows(rows pgx.Rows) ([]*models.ScanRun, error) {
func (r *Repository) Create(ctx context.Context, input CreateInput) (*models.ScanRun, bool, error) {
run, err := scanRunRow(r.pool.QueryRow(ctx, `
INSERT INTO scan_runs (
id, media_folder_id, mode, path, trigger, status
) VALUES ($1, $2, $3, $4, $5, $6)
id, media_folder_id, mode, path, trigger, status, autoscan_event_id
) VALUES ($1, $2, $3, $4, $5, $6, $7)
RETURNING `+scanRunColumns,
ulid.Make().String(),
input.LibraryID,
@@ -104,6 +106,7 @@ func (r *Repository) Create(ctx context.Context, input CreateInput) (*models.Sca
input.Path,
input.Trigger,
StatusAccepted,
input.AutoscanEventID,
))
if err == nil {
return run, true, nil
@@ -136,8 +139,8 @@ func (r *Repository) CreateBatch(ctx context.Context, inputs []CreateInput) ([]*
for _, input := range inputs {
run, err := scanRunRow(tx.QueryRow(ctx, `
INSERT INTO scan_runs (
id, media_folder_id, mode, path, trigger, status
) VALUES ($1, $2, $3, $4, $5, $6)
id, media_folder_id, mode, path, trigger, status, autoscan_event_id
) VALUES ($1, $2, $3, $4, $5, $6, $7)
ON CONFLICT DO NOTHING
RETURNING `+scanRunColumns,
ulid.Make().String(),
@@ -146,6 +149,7 @@ func (r *Repository) CreateBatch(ctx context.Context, inputs []CreateInput) ([]*
input.Path,
input.Trigger,
StatusAccepted,
input.AutoscanEventID,
))
if err == nil {
runs = append(runs, run)
internal/scanqueue/service.go
+23 -8
View File
@@ -133,31 +133,46 @@ func (s *Service) EnqueueScan(ctx context.Context, folderID int, mode, path, tri
}
func (s *Service) EnqueueScans(ctx context.Context, targets []scantrigger.Target) error {
_, _, err := s.enqueueScans(ctx, targets, nil)
return err
}
func (s *Service) EnqueueAutoscanScans(ctx context.Context, targets []scantrigger.Target, eventID int64) (int, int, error) {
return s.enqueueScans(ctx, targets, &eventID)
}
func (s *Service) enqueueScans(ctx context.Context, targets []scantrigger.Target, autoscanEventID *int64) (int, int, error) {
if s == nil || s.repo == nil {
return fmt.Errorf("scan queue is not configured")
return 0, 0, fmt.Errorf("scan queue is not configured")
}
inputs := make([]CreateInput, 0, len(targets))
for _, target := range targets {
if target.Folder == nil {
return fmt.Errorf("scan queue: target is missing folder")
return 0, 0, fmt.Errorf("scan queue: target is missing folder")
}
inputs = append(inputs, CreateInput{
LibraryID: target.Folder.ID,
Mode: target.Mode,
Path: target.Path,
Trigger: target.Trigger,
LibraryID: target.Folder.ID,
Mode: target.Mode,
Path: target.Path,
Trigger: target.Trigger,
AutoscanEventID: autoscanEventID,
})
}
runs, created, err := s.repo.CreateBatch(ctx, inputs)
if err != nil {
return err
return 0, 0, err
}
createdCount := 0
reusedCount := 0
for i, run := range runs {
if i < len(created) && created[i] {
createdCount++
s.publish(ctx, "scan.accepted", run)
} else {
reusedCount++
}
}
return nil
return createdCount, reusedCount, nil
}
func (s *Service) CancelAcceptedByLibrary(ctx context.Context, libraryID int) (int, error) {
internal/scantrigger/scantrigger.go
+31
View File
@@ -95,6 +95,37 @@ func (r *Resolver) Resolve(ctx context.Context, req Request) (*Target, error) {
return r.resolve(ctx, req, nil, false)
}
// ResolveMissingSubtree resolves a subtree path that may no longer exist on
// disk. This is intentionally narrower than Resolve: it never stats the path
// and only returns ModeSubtree for paths below a configured library root.
func (r *Resolver) ResolveMissingSubtree(ctx context.Context, subtreePath, trigger string) (*Target, error) {
if r == nil || r.folders == nil {
return nil, &RequestError{Status: http.StatusServiceUnavailable, Code: "unavailable", Message: "Scanner not available"}
}
cleanPath := filepath.Clean(subtreePath)
if strings.TrimSpace(subtreePath) == "" || cleanPath == "." {
return nil, &RequestError{Status: http.StatusBadRequest, Code: "bad_request", Message: "Path is required"}
}
folders, err := r.folders.List(ctx)
if err != nil {
return nil, fmt.Errorf("listing libraries for scan: %w", err)
}
folder, matchedRoot, err := MatchFolderForPath(cleanPath, folders)
if err != nil {
return nil, err
}
if folder != nil && !folder.Enabled {
return nil, &RequestError{Status: http.StatusConflict, Code: "conflict", Message: "Library is disabled"}
}
if filepath.Clean(cleanPath) == filepath.Clean(matchedRoot) {
return nil, &RequestError{Status: http.StatusBadRequest, Code: "bad_request", Message: "Subtree path must be below a library root"}
}
if trigger = strings.TrimSpace(trigger); trigger == "" {
trigger = "path"
}
return &Target{Folder: folder, Mode: ModeSubtree, Path: cleanPath, Trigger: trigger}, nil
}
func (r *Resolver) resolve(ctx context.Context, req Request, pathFolders []*models.MediaFolder, usePathFolders bool) (*Target, error) {
if r == nil || r.folders == nil {
return nil, &RequestError{Status: http.StatusServiceUnavailable, Code: "unavailable", Message: "Scanner not available"}
internal/scantrigger/scantrigger_test.go
+38
View File
@@ -71,6 +71,44 @@ func TestResolverClassifiesSubtree(t *testing.T) {
}
}
func TestResolverResolvesMissingSubtreeWithoutStat(t *testing.T) {
root := t.TempDir()
missing := filepath.Join(root, "Missing Movie")
repo := &fakeFolderRepo{folders: []*models.MediaFolder{{
ID: 18,
Name: "Movies",
Enabled: true,
Paths: []string{root},
}}}
target, err := NewResolver(repo).ResolveMissingSubtree(context.Background(), missing, "autoscan")
if err != nil {
t.Fatalf("ResolveMissingSubtree returned error: %v", err)
}
if target.Folder == nil || target.Folder.ID != 18 || target.Mode != ModeSubtree || target.Path != filepath.Clean(missing) {
t.Fatalf("unexpected target: %#v", target)
}
}
func TestResolverRejectsMissingSubtreeAtLibraryRoot(t *testing.T) {
root := t.TempDir()
repo := &fakeFolderRepo{folders: []*models.MediaFolder{{
ID: 19,
Name: "Movies",
Enabled: true,
Paths: []string{root},
}}}
_, err := NewResolver(repo).ResolveMissingSubtree(context.Background(), root, "autoscan")
var reqErr *RequestError
if !errors.As(err, &reqErr) {
t.Fatalf("expected RequestError, got %T: %v", err, err)
}
if reqErr.Status != http.StatusBadRequest || reqErr.Code != "bad_request" {
t.Fatalf("unexpected error: %#v", reqErr)
}
}
func TestResolverClassifiesVideoFile(t *testing.T) {
root := t.TempDir()
filePath := filepath.Join(root, "Movie (2024).mkv")
internal/taskmanager/tasks/autoscan_poll.go
+62
View File
@@ -0,0 +1,62 @@
package tasks
import (
"context"
"fmt"
"github.com/Silo-Server/silo-server/internal/taskmanager"
)
type AutoscanPoller interface {
PollOnce(ctx context.Context) error
}
// defaultAutoscanPollIntervalMs is the fallback poll cadence (10 minutes) used
// when no positive interval is supplied.
const defaultAutoscanPollIntervalMs int64 = 10 * 60 * 1000
type AutoscanPollTask struct {
poller AutoscanPoller
intervalMs int64
}
// NewAutoscanPollTask builds the poll task with an interval expressed in
// MILLISECONDS. This matches the reschedule path (HandleUpdateSettings seeds
// seconds*1000 ms), so the startup-seeded schedule and a settings-driven
// reschedule agree for sub-minute and non-60-multiple intervals — previously the
// startup path integer-divided seconds by 60 (minutes) and diverged.
func NewAutoscanPollTask(poller AutoscanPoller, intervalMs int64) *AutoscanPollTask {
if intervalMs <= 0 {
intervalMs = defaultAutoscanPollIntervalMs
}
return &AutoscanPollTask{poller: poller, intervalMs: intervalMs}
}
func (t *AutoscanPollTask) Key() string { return "autoscan_poll" }
func (t *AutoscanPollTask) Name() string { return "Autoscan poll" }
func (t *AutoscanPollTask) Description() string {
return "Poll installed scan-source providers for changes and enqueue rescans"
}
func (t *AutoscanPollTask) Category() taskmanager.TaskCategory {
return taskmanager.TaskCategoryLibrary
}
func (t *AutoscanPollTask) IsHidden() bool { return false }
func (t *AutoscanPollTask) DefaultTriggers() []taskmanager.TriggerConfig {
return []taskmanager.TriggerConfig{
{Type: taskmanager.TriggerTypeInterval, IntervalMs: t.intervalMs},
}
}
func (t *AutoscanPollTask) Execute(ctx context.Context, progress taskmanager.ProgressReporter) error {
progress.Report(0, "Polling scan-source providers")
if t.poller == nil {
progress.Report(100, "Autoscan unavailable")
return nil
}
if err := t.poller.PollOnce(ctx); err != nil {
return fmt.Errorf("autoscan poll: %w", err)
}
progress.Report(100, "Autoscan poll complete")
return nil
}
migrations/171_autoscan.down.sql
+3
View File
@@ -0,0 +1,3 @@
DROP TABLE IF EXISTS public.autoscan_sources;
DROP TABLE IF EXISTS public.autoscan_connections;
DROP TABLE IF EXISTS public.autoscan_settings;
migrations/171_autoscan.up.sql
+62
View File
@@ -0,0 +1,62 @@
-- Autoscan: connections-based scan-source model. The host pulls changed paths
-- from installed scan_source.v1 plugins (Sonarr/Radarr, etc.) on a timer,
-- applies per-source path rewrites, and enqueues targeted library rescans.
--
-- Three tables:
-- * autoscan_settings - singleton global config (enable, poll cadence, debounce)
-- * autoscan_connections - reachable upstreams: own credentials, OR a live link
-- to a Requests integration (reused arr server)
-- * autoscan_sources - operator-created rows, each binding one installed
-- scan_source capability to a connection + rewrites +
-- cadence (many sources may share one capability)
CREATE TABLE public.autoscan_settings (
id boolean PRIMARY KEY DEFAULT true,
enabled boolean NOT NULL DEFAULT false,
default_poll_interval_seconds integer NOT NULL DEFAULT 600,
debounce_seconds integer NOT NULL DEFAULT 60,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
CONSTRAINT autoscan_settings_singleton CHECK (id),
CONSTRAINT autoscan_settings_interval_pos CHECK (default_poll_interval_seconds > 0),
CONSTRAINT autoscan_settings_debounce_nonneg CHECK (debounce_seconds >= 0)
);
INSERT INTO public.autoscan_settings (id) VALUES (true) ON CONFLICT (id) DO NOTHING;
CREATE TABLE public.autoscan_connections (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
name text NOT NULL,
kind text NOT NULL,
base_url text,
api_key_ref text,
request_integration_id text REFERENCES public.request_integrations(id) ON DELETE SET NULL,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now()
-- Creation-time validity (own creds OR a link) is enforced at the application
-- layer. A linked connection may become orphaned (both null) when its Requests
-- integration is deleted (ON DELETE SET NULL), surfacing as "needs attention".
);
CREATE TABLE public.autoscan_sources (
id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
installation_id integer NOT NULL,
capability_id text NOT NULL,
-- Nullable: a freshly-discovered source has no connection until the operator
-- binds one. The engine skips an enabled source with no connection.
connection_id uuid REFERENCES public.autoscan_connections(id) ON DELETE RESTRICT,
enabled boolean NOT NULL DEFAULT false,
poll_interval_seconds integer,
-- Host-owned per-source prefix rewrites: [{"from": "...", "to": "..."}].
-- Applied to the raw source-namespace paths the scan_source plugin returns,
-- before resolving/enqueueing.
path_rewrites jsonb NOT NULL DEFAULT '[]'::jsonb,
marker text,
last_run_at timestamptz,
last_error text,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
CONSTRAINT autoscan_sources_interval_pos CHECK (poll_interval_seconds IS NULL OR poll_interval_seconds > 0)
-- No UNIQUE(installation_id, capability_id): one installed scan_source plugin
-- capability backs MANY sources, each bound to a different connection (e.g. 4
-- arr servers from one plugin install). Operators create sources explicitly.
);
migrations/172_autoscan_source_config.down.sql
+2
View File
@@ -0,0 +1,2 @@
ALTER TABLE public.autoscan_sources
DROP COLUMN IF EXISTS source_config;
migrations/172_autoscan_source_config.up.sql
+23
View File
@@ -0,0 +1,23 @@
ALTER TABLE public.autoscan_sources
ADD COLUMN source_config jsonb NOT NULL DEFAULT '{}'::jsonb;
-- Move CephFS source-specific settings out of generic plugin runtime config
-- when upgrading dev/early installs. The plugin may still keep runtime-only
-- state settings there, but watch roots and exclusions belong to autoscan
-- source rows so operators manage them from Admin > Autoscan.
UPDATE public.autoscan_sources s
SET source_config = jsonb_strip_nulls(jsonb_build_object(
'movie_flat_paths', cfg.config_value->>'movie_flat_paths',
'tv_flat_paths', cfg.config_value->>'tv_flat_paths',
'movie_nested_paths', cfg.config_value->>'movie_nested_paths',
'tv_nested_paths', cfg.config_value->>'tv_nested_paths',
'exclusions', cfg.config_value->>'exclusions'
))
FROM public.plugin_installations pi
JOIN public.plugin_runtime_configs cfg
ON cfg.plugin_installation_id = pi.id
AND cfg.config_key = 'cephfs_monitor'
WHERE s.installation_id = pi.id
AND pi.plugin_id = 'silo.autoscan.cephfs'
AND s.capability_id = 'cephfs'
AND s.source_config = '{}'::jsonb;
migrations/173_autoscan_events.down.sql
+8
View File
@@ -0,0 +1,8 @@
DROP INDEX IF EXISTS public.idx_scan_runs_autoscan_event;
ALTER TABLE public.scan_runs
DROP COLUMN IF EXISTS autoscan_event_id;
DROP INDEX IF EXISTS public.idx_autoscan_events_completed;
DROP INDEX IF EXISTS public.idx_autoscan_events_source_completed;
DROP TABLE IF EXISTS public.autoscan_events;
migrations/173_autoscan_events.up.sql
+33
View File
@@ -0,0 +1,33 @@
CREATE TABLE public.autoscan_events (
id BIGSERIAL PRIMARY KEY,
source_id uuid REFERENCES public.autoscan_sources(id) ON DELETE SET NULL,
installation_id integer NOT NULL,
capability_id text NOT NULL,
started_at timestamptz NOT NULL,
completed_at timestamptz NOT NULL,
duration_ms bigint NOT NULL DEFAULT 0,
status text NOT NULL,
changes_returned integer NOT NULL DEFAULT 0,
changes_resolved integer NOT NULL DEFAULT 0,
targets_claimed integer NOT NULL DEFAULT 0,
scans_created integer NOT NULL DEFAULT 0,
scans_reused integer NOT NULL DEFAULT 0,
scans_suppressed integer NOT NULL DEFAULT 0,
error_message text NOT NULL DEFAULT '',
marker_before text,
marker_after text,
created_at timestamptz NOT NULL DEFAULT now(),
CONSTRAINT autoscan_events_status_check CHECK (status = ANY (ARRAY['success'::text, 'error'::text, 'unresolved'::text]))
);
CREATE INDEX idx_autoscan_events_source_completed
ON public.autoscan_events (source_id, completed_at DESC);
CREATE INDEX idx_autoscan_events_completed
ON public.autoscan_events (completed_at DESC);
ALTER TABLE public.scan_runs
ADD COLUMN autoscan_event_id bigint REFERENCES public.autoscan_events(id) ON DELETE SET NULL;
CREATE INDEX idx_scan_runs_autoscan_event
ON public.scan_runs (autoscan_event_id);
migrations/174_autoscan_source_label.down.sql
+1
View File
@@ -0,0 +1 @@
ALTER TABLE public.autoscan_sources DROP COLUMN label;
migrations/174_autoscan_source_label.up.sql
+3
View File
@@ -0,0 +1,3 @@
-- Operator-editable display label for an autoscan source. Empty string = unset;
-- the admin UI falls back to connection name / plugin display_name / capability.
ALTER TABLE public.autoscan_sources ADD COLUMN label text NOT NULL DEFAULT '';
migrations/174_movie_release_date_desc_index.down.sql → migrations/175_movie_release_date_desc_index.down.sql
View File
migrations/174_movie_release_date_desc_index.up.sql → migrations/175_movie_release_date_desc_index.up.sql
View File
web/src/App.tsx
+2
View File
@@ -40,6 +40,7 @@ import AdminActivity from "@/pages/AdminActivity";
import AdminLogs from "@/pages/AdminLogs";
import AdminUsers from "@/pages/AdminUsers";
import AdminRequests from "@/pages/AdminRequests";
import AdminAutoscan from "@/pages/AdminAutoscan";
import AdminDevices from "@/pages/AdminDevices";
import AdminLibraries from "@/pages/AdminLibraries";
import AdminSettingsLayout from "@/pages/admin-settings/AdminSettingsLayout";
@@ -372,6 +373,7 @@ function AppRoutes() {
<Route path="collections/new" element={<AdminCollectionEditor />} />
<Route path="collections/:id/edit" element={<AdminCollectionEditor />} />
<Route path="requests" element={<AdminRequests />} />
<Route path="autoscan" element={<AdminAutoscan />} />
<Route path="history" element={<AdminPlaybackHistory />} />
<Route path="history-import" element={<AdminHistoryImport />} />
<Route path="users" element={<AdminUsers />} />
web/src/api/types.ts
+202
View File
@@ -1660,6 +1660,208 @@ export interface RequestIntegrationsResponse {
integrations: RequestIntegration[];
}
// --- Autoscan v2 types (matched to autoscan.go handler DTOs) ---
export interface AutoscanSettings {
enabled: boolean;
default_poll_interval_seconds: number;
debounce_seconds: number;
}
export interface AutoscanConnection {
id: string;
name: string;
kind: string;
base_url?: string;
request_integration_id?: string | null;
has_api_key: boolean;
}
export interface AutoscanConnectionInput {
name: string;
kind: string;
base_url?: string;
api_key_ref?: string;
request_integration_id?: string | null;
}
export interface AutoscanConnectionsResponse {
connections: AutoscanConnection[];
}
export interface AutoscanPathRewrite {
from: string;
to: string;
}
export interface AutoscanSource {
id: string;
installation_id: number;
capability_id: string;
connection_id: string | null;
enabled: boolean;
poll_interval_seconds: number | null;
last_run_at: string | null;
last_error: string | null;
path_rewrites: AutoscanPathRewrite[];
source_config: Record<string, string>;
label: string;
}
export interface AutoscanSourceInput {
connection_id: string | null;
enabled: boolean;
poll_interval_seconds: number | null;
path_rewrites?: AutoscanPathRewrite[];
source_config?: Record<string, string>;
label?: string;
}
export interface AutoscanSourcesResponse {
sources: AutoscanSource[];
}
export interface AutoscanAvailableSource {
installation_id: number;
capability_id: string;
plugin_id: string;
display_name: string;
}
export interface AutoscanAvailableSourcesResponse {
plugins: AutoscanAvailableSource[];
}
export interface AutoscanSourceCreateInput {
installation_id: number;
capability_id: string;
connection_id?: string | null;
enabled: boolean;
poll_interval_seconds?: number | null;
path_rewrites: AutoscanPathRewrite[];
source_config?: Record<string, string>;
}
export interface AutoscanConnectionTestInput {
connection_id?: string;
base_url?: string;
api_key_ref?: string;
request_integration_id?: string | null;
}
export interface AutoscanConnectionTestResult {
ok: boolean;
version?: string;
error?: string;
}
export interface AutoscanProposedRewrite {
from: string;
to: string;
match_depth: number;
}
export interface AutoscanAmbiguousRoot {
root: string;
candidates: string[];
}
export interface AutoscanRewriteSuggestions {
proposed: AutoscanProposedRewrite[];
unmatched: string[];
ambiguous: AutoscanAmbiguousRoot[];
covered: string[];
}
export interface AutoscanStatusSource {
id: string;
installation_id: number;
capability_id: string;
connection_id: string | null;
enabled: boolean;
label: string;
last_run_at: string | null;
last_error: string | null;
}
export interface AutoscanStatus {
enabled: boolean;
sources: AutoscanStatusSource[];
active_scans: number;
accepted_scans: number;
running_scans: number;
latest_event_at?: string;
}
export type AutoscanEventStatus = "success" | "error" | "unresolved";
export interface AutoscanEventScanRun {
id: string;
library_id: number;
mode: "library" | "subtree" | "file";
path?: string;
trigger: string;
status: "accepted" | "running" | "completed" | "failed" | "cancelled";
requested_at?: string;
started_at?: string;
completed_at?: string;
error_message?: string;
}
export interface AutoscanEvent {
id: number;
source_id: string | null;
installation_id: number;
capability_id: string;
started_at: string;
completed_at: string;
duration_ms: number;
status: AutoscanEventStatus;
changes_returned: number;
changes_resolved: number;
targets_claimed: number;
scans_created: number;
scans_reused: number;
scans_suppressed: number;
error_message?: string;
scan_runs: AutoscanEventScanRun[];
}
export interface AutoscanEventsResponse {
events: AutoscanEvent[];
total: number;
limit: number;
offset: number;
}
export type AutoscanScanStatus = "accepted" | "running" | "completed" | "failed" | "cancelled";
export interface AutoscanScan {
id: string;
library_id: number;
mode: "library" | "subtree" | "file";
path?: string;
trigger: string;
status: AutoscanScanStatus;
error_message?: string;
requested_at?: string;
started_at?: string;
completed_at?: string;
autoscan_event_id?: number;
source_id?: string;
installation_id?: number;
capability_id?: string;
event_status?: AutoscanEventStatus;
event_completed_at?: string;
}
export interface AutoscanScansResponse {
scans: AutoscanScan[];
total: number;
limit: number;
offset: number;
}
export interface RequestListParams {
status?: MediaRequestStatus | "all";
outcome?: MediaRequestOutcome | "all";
web/src/components/AdminSidebar.tsx
+6
View File
@@ -21,6 +21,7 @@ import {
Blocks,
Puzzle,
Send,
RefreshCw,
} from "lucide-react";
import type { ReactNode } from "react";
import { SiloBrand } from "@/components/SiloBrand";
@@ -113,6 +114,11 @@ export default function AdminSidebar({ onNavigate }: AdminSidebarProps) {
icon: <Send className="h-[18px] w-[18px]" />,
href: "/admin/requests",
},
{
label: "Autoscan",
icon: <RefreshCw className="h-[18px] w-[18px]" />,
href: "/admin/autoscan",
},
{
label: "Sections",
icon: <PanelsTopLeft className="h-[18px] w-[18px]" />,
web/src/components/ui/pagination.test.ts
+40
View File
@@ -0,0 +1,40 @@
import { describe, expect, it } from "vitest";
import { pageWindow } from "./pagination.utils";
describe("pageWindow", () => {
it("lists every page without ellipsis when the count is small", () => {
expect(pageWindow(0, 5)).toEqual([0, 1, 2, 3, 4]);
expect(pageWindow(3, 7)).toEqual([0, 1, 2, 3, 4, 5, 6]);
});
it("anchors first/last and the current neighbours in the middle", () => {
expect(pageWindow(10, 20)).toEqual([0, "ellipsis", 9, 10, 11, "ellipsis", 19]);
});
it("collapses only the trailing gap near the start", () => {
expect(pageWindow(0, 20)).toEqual([0, 1, "ellipsis", 19]);
expect(pageWindow(1, 20)).toEqual([0, 1, 2, "ellipsis", 19]);
});
it("collapses only the leading gap near the end", () => {
expect(pageWindow(19, 20)).toEqual([0, "ellipsis", 18, 19]);
expect(pageWindow(18, 20)).toEqual([0, "ellipsis", 17, 18, 19]);
});
it("never emits duplicate or out-of-range indices", () => {
for (let count = 1; count <= 30; count++) {
for (let current = 0; current < count; current++) {
const window = pageWindow(current, count);
const pages = window.filter((entry): entry is number => entry !== "ellipsis");
// Strictly increasing, in-range, and always includes the current page.
expect(pages).toEqual([...pages].sort((a, b) => a - b));
expect(new Set(pages).size).toBe(pages.length);
expect(pages.every((p) => p >= 0 && p < count)).toBe(true);
expect(pages).toContain(current);
expect(pages).toContain(0);
expect(pages).toContain(count - 1);
}
}
});
});
web/src/components/ui/pagination.tsx
+166
View File
@@ -0,0 +1,166 @@
import { ChevronLeft, ChevronRight, ChevronsLeft, ChevronsRight } from "lucide-react";
import { pageWindow } from "@/components/ui/pagination.utils";
import { Button } from "@/components/ui/button";
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select";
import { cn } from "@/lib/utils";
const DEFAULT_PAGE_SIZE_OPTIONS = [25, 50, 100];
export interface TablePaginationProps {
/** Zero-indexed current page. */
page: number;
pageSize: number;
/** Total number of matching rows across all pages. */
total: number;
onPageChange: (page: number) => void;
/** When provided, renders a rows-per-page selector. */
onPageSizeChange?: (size: number) => void;
pageSizeOptions?: number[];
/** Singular noun for the summary, e.g. "scan" → "Showing 1–25 of 240 scans". */
itemNoun?: string;
/** Dims the summary while a background refetch is in flight. */
isFetching?: boolean;
className?: string;
}
export function TablePagination({
page,
pageSize,
total,
onPageChange,
onPageSizeChange,
pageSizeOptions = DEFAULT_PAGE_SIZE_OPTIONS,
itemNoun = "item",
isFetching = false,
className,
}: TablePaginationProps) {
if (total <= 0) return null;
const pageCount = Math.max(1, Math.ceil(total / pageSize));
const safePage = Math.min(Math.max(page, 0), pageCount - 1);
const from = safePage * pageSize + 1;
const to = Math.min((safePage + 1) * pageSize, total);
const window = pageWindow(safePage, pageCount);
const noun = total === 1 ? itemNoun : `${itemNoun}s`;
const goTo = (next: number) => {
const clamped = Math.min(Math.max(next, 0), pageCount - 1);
if (clamped !== safePage) onPageChange(clamped);
};
return (
<div
className={cn(
"flex flex-col gap-3 sm:flex-row sm:items-center sm:justify-between",
className,
)}
>
<div className="text-muted-foreground flex flex-wrap items-center gap-x-4 gap-y-2 text-xs">
<span className={cn("transition-opacity", isFetching && "opacity-60")}>
Showing <span className="text-foreground font-medium tabular-nums">{from}</span>–
<span className="text-foreground font-medium tabular-nums">{to}</span> of{" "}
<span className="text-foreground font-medium tabular-nums">{total}</span> {noun}
</span>
{onPageSizeChange ? (
<label className="flex items-center gap-2">
<span className="hidden sm:inline">Rows</span>
<Select
value={String(pageSize)}
onValueChange={(value) => onPageSizeChange(Number(value))}
>
<SelectTrigger size="sm" className="h-8 w-[4.5rem] tabular-nums">
<SelectValue />
</SelectTrigger>
<SelectContent>
{pageSizeOptions.map((size) => (
<SelectItem key={size} value={String(size)} className="tabular-nums">
{size}
</SelectItem>
))}
</SelectContent>
</Select>
</label>
) : null}
</div>
{pageCount > 1 ? (
<nav aria-label="Pagination" className="flex items-center gap-1">
<Button
variant="outline"
size="icon-sm"
onClick={() => goTo(0)}
disabled={safePage === 0}
aria-label="First page"
>
<ChevronsLeft />
</Button>
<Button
variant="outline"
size="icon-sm"
onClick={() => goTo(safePage - 1)}
disabled={safePage === 0}
aria-label="Previous page"
>
<ChevronLeft />
</Button>
{/* Numbered window on roomy viewports, compact counter on phones. */}
<div className="hidden items-center gap-1 sm:flex">
{window.map((entry, index) =>
entry === "ellipsis" ? (
<span
key={`ellipsis-${index}`}
className="text-muted-foreground/60 px-1 text-sm select-none"
aria-hidden="true"
>
…
</span>
) : (
<Button
key={entry}
variant={entry === safePage ? "default" : "ghost"}
size="icon-sm"
className="tabular-nums"
onClick={() => goTo(entry)}
aria-label={`Page ${entry + 1}`}
aria-current={entry === safePage ? "page" : undefined}
>
{entry + 1}
</Button>
),
)}
</div>
<span className="text-muted-foreground px-2 text-xs tabular-nums sm:hidden">
{safePage + 1} / {pageCount}
</span>
<Button
variant="outline"
size="icon-sm"
onClick={() => goTo(safePage + 1)}
disabled={safePage >= pageCount - 1}
aria-label="Next page"
>
<ChevronRight />
</Button>
<Button
variant="outline"
size="icon-sm"
onClick={() => goTo(pageCount - 1)}
disabled={safePage >= pageCount - 1}
aria-label="Last page"
>
<ChevronsRight />
</Button>
</nav>
) : null}
</div>
);
}
web/src/components/ui/pagination.utils.ts
+21
View File
@@ -0,0 +1,21 @@
/**
* Build a compact run of page indices with ellipsis gaps, always anchoring the
* first and last page plus the current page and its immediate neighbours, e.g.
* `[0, "ellipsis", 4, 5, 6, "ellipsis", 19]`. Pages are zero-indexed.
*/
export function pageWindow(current: number, pageCount: number): (number | "ellipsis")[] {
const last = pageCount - 1;
if (pageCount <= 7) {
return Array.from({ length: pageCount }, (_, i) => i);
}
const anchors = [0, last, current, current - 1, current + 1].filter((p) => p >= 0 && p <= last);
const unique = [...new Set(anchors)].sort((a, b) => a - b);
const out: (number | "ellipsis")[] = [];
let prev: number | null = null;
for (const p of unique) {
if (prev !== null && p - prev > 1) out.push("ellipsis");
out.push(p);
prev = p;
}
return out;
}
web/src/hooks/queries/keys.ts
+9
View File
@@ -393,4 +393,13 @@ export const adminKeys = {
itemImages: (id: string) => ["admin", "items", id, "images"] as const,
buildInfo: () => ["admin", "system", "buildInfo"] as const,
hwAccel: () => ["admin", "system", "hwAccel"] as const,
autoscanSettings: () => ["admin", "autoscan", "settings"] as const,
autoscanConnections: () => ["admin", "autoscan", "connections"] as const,
autoscanSources: () => ["admin", "autoscan", "sources"] as const,
autoscanScanSourcePlugins: () => ["admin", "autoscan", "scan-source-plugins"] as const,
autoscanStatus: () => ["admin", "autoscan", "status"] as const,
autoscanScans: (params?: Record<string, unknown>) =>
["admin", "autoscan", "scans", params ?? {}] as const,
autoscanEvents: (params?: Record<string, unknown>) =>
["admin", "autoscan", "events", params ?? {}] as const,
};
web/src/hooks/queries/useAutoscan.ts
+327
View File
@@ -0,0 +1,327 @@
import { keepPreviousData, useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { toast } from "sonner";
import { api } from "@/api/client";
import type {
AutoscanAvailableSourcesResponse,
AutoscanConnection,
AutoscanConnectionInput,
AutoscanConnectionsResponse,
AutoscanConnectionTestInput,
AutoscanConnectionTestResult,
AutoscanEvent,
AutoscanEventsResponse,
AutoscanEventStatus,
AutoscanScan,
AutoscanScansResponse,
AutoscanScanStatus,
AutoscanRewriteSuggestions,
AutoscanSettings,
AutoscanSource,
AutoscanSourceCreateInput,
AutoscanSourceInput,
AutoscanSourcesResponse,
AutoscanStatus,
} from "@/api/types";
import { adminKeys } from "./keys";
const AUTOSCAN_STALE_TIME = 30_000;
const AUTOSCAN_ACTIVITY_REFRESH_MS = 15_000;
// --- Settings ---
export function useAutoscanSettings() {
return useQuery({
queryKey: adminKeys.autoscanSettings(),
queryFn: () => api<AutoscanSettings>("/admin/autoscan/settings"),
staleTime: AUTOSCAN_STALE_TIME,
});
}
export function useUpdateAutoscanSettings() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: (body: AutoscanSettings) =>
api<AutoscanSettings>("/admin/autoscan/settings", {
method: "PUT",
body: JSON.stringify(body),
}),
onSuccess: () => {
toast.success("Autoscan settings saved");
queryClient.invalidateQueries({ queryKey: adminKeys.autoscanSettings() });
},
onError: (err) => {
toast.error(err instanceof Error ? err.message : "Failed to save autoscan settings");
},
});
}
// --- Connections ---
export function useAutoscanConnections() {
return useQuery({
queryKey: adminKeys.autoscanConnections(),
queryFn: () =>
api<AutoscanConnectionsResponse>("/admin/autoscan/connections").then(
(data) => data.connections ?? [],
),
staleTime: AUTOSCAN_STALE_TIME,
});
}
export function useCreateAutoscanConnection() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: (body: AutoscanConnectionInput) =>
api<AutoscanConnection>("/admin/autoscan/connections", {
method: "POST",
body: JSON.stringify(body),
}),
onSuccess: () => {
toast.success("Autoscan connection created");
queryClient.invalidateQueries({ queryKey: adminKeys.autoscanConnections() });
},
onError: (err) => {
toast.error(err instanceof Error ? err.message : "Failed to create autoscan connection");
},
});
}
export function useUpdateAutoscanConnection() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: ({ id, body }: { id: string; body: AutoscanConnectionInput }) =>
api<AutoscanConnection>(`/admin/autoscan/connections/${encodeURIComponent(id)}`, {
method: "PUT",
body: JSON.stringify(body),
}),
onSuccess: () => {
toast.success("Autoscan connection updated");
queryClient.invalidateQueries({ queryKey: adminKeys.autoscanConnections() });
},
onError: (err) => {
toast.error(err instanceof Error ? err.message : "Failed to update autoscan connection");
},
});
}
export function useDeleteAutoscanConnection() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: (id: string) =>
api<void>(`/admin/autoscan/connections/${encodeURIComponent(id)}`, {
method: "DELETE",
}),
onSuccess: () => {
toast.success("Autoscan connection deleted");
queryClient.invalidateQueries({ queryKey: adminKeys.autoscanConnections() });
// Sources may have lost their connection binding; invalidate them too.
queryClient.invalidateQueries({ queryKey: adminKeys.autoscanSources() });
},
onError: (err) => {
toast.error(err instanceof Error ? err.message : "Failed to delete autoscan connection");
},
});
}
// --- Sources ---
export function useAutoscanSources() {
return useQuery({
queryKey: adminKeys.autoscanSources(),
queryFn: () =>
api<AutoscanSourcesResponse>("/admin/autoscan/sources").then((data) => data.sources ?? []),
staleTime: AUTOSCAN_STALE_TIME,
});
}
export function useAvailableScanSources() {
return useQuery({
queryKey: adminKeys.autoscanScanSourcePlugins(),
queryFn: () =>
api<AutoscanAvailableSourcesResponse>("/admin/autoscan/scan-source-plugins").then(
(data) => data.plugins ?? [],
),
staleTime: AUTOSCAN_STALE_TIME,
});
}
export function useCreateAutoscanSource() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: (body: AutoscanSourceCreateInput) =>
api<AutoscanSource>("/admin/autoscan/sources", {
method: "POST",
body: JSON.stringify(body),
}),
onSuccess: () => {
toast.success("Autoscan source created");
queryClient.invalidateQueries({ queryKey: adminKeys.autoscanSources() });
},
onError: (err) => {
toast.error(err instanceof Error ? err.message : "Failed to create autoscan source");
},
});
}
export function useUpdateAutoscanSource() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: ({ id, body }: { id: string; body: AutoscanSourceInput }) =>
api<AutoscanSource>(`/admin/autoscan/sources/${encodeURIComponent(id)}`, {
method: "PUT",
body: JSON.stringify(body),
}),
onSuccess: () => {
toast.success("Autoscan source saved");
queryClient.invalidateQueries({ queryKey: adminKeys.autoscanSources() });
},
onError: (err) => {
toast.error(err instanceof Error ? err.message : "Failed to save autoscan source");
},
});
}
export function useDeleteAutoscanSource() {
const queryClient = useQueryClient();
return useMutation({
mutationFn: (id: string) =>
api<void>(`/admin/autoscan/sources/${encodeURIComponent(id)}`, {
method: "DELETE",
}),
onSuccess: () => {
toast.success("Autoscan source deleted");
queryClient.invalidateQueries({ queryKey: adminKeys.autoscanSources() });
},
onError: (err) => {
toast.error(err instanceof Error ? err.message : "Failed to delete autoscan source");
},
});
}
/**
* Test an arr connection. Accepts either an existing connection id, or raw
* credentials (base_url + api_key_ref) / a request integration id for an
* unsaved dialog. Returns the result so the caller can render it inline;
* errors are surfaced via the returned result, not a toast (advisory only).
*/
export function useTestAutoscanConnection() {
return useMutation({
mutationFn: (body: AutoscanConnectionTestInput) =>
api<AutoscanConnectionTestResult>("/admin/autoscan/connections/test", {
method: "POST",
body: JSON.stringify(body),
}),
});
}
/**
* Lazily fetch rewrite suggestions for a single source. Triggered on demand
* (per source) so it is modelled as a mutation rather than a query. Returns
* the suggestions; the 400 (no bound connection) surfaces as a toast.
*/
export function useAutoscanRewriteSuggestions() {
return useMutation({
mutationFn: (id: string) =>
api<AutoscanRewriteSuggestions>(
`/admin/autoscan/sources/${encodeURIComponent(id)}/rewrite-suggestions`,
),
onError: (err) =>
toast.error(
err instanceof Error ? err.message : "Could not sync rewrites from the arr instance",
),
});
}
// --- Status ---
export function useAutoscanStatus() {
return useQuery({
queryKey: adminKeys.autoscanStatus(),
queryFn: () => api<AutoscanStatus>("/admin/autoscan/status"),
staleTime: AUTOSCAN_STALE_TIME,
refetchInterval: AUTOSCAN_ACTIVITY_REFRESH_MS,
});
}
/** A single page of history rows plus the total matching count for pagination. */
export interface AutoscanPage<T> {
rows: T[];
total: number;
}
export function useAutoscanEvents(params?: {
sourceId?: string;
status?: AutoscanEventStatus;
query?: string;
limit?: number;
offset?: number;
enabled?: boolean;
}) {
const queryParams = new URLSearchParams();
if (params?.sourceId) queryParams.set("source_id", params.sourceId);
if (params?.status) queryParams.set("status", params.status);
if (params?.query) queryParams.set("q", params.query);
if (params?.limit != null) queryParams.set("limit", String(params.limit));
if (params?.offset != null) queryParams.set("offset", String(params.offset));
const suffix = queryParams.toString();
const path = suffix ? `/admin/autoscan/events?${suffix}` : "/admin/autoscan/events";
return useQuery({
queryKey: adminKeys.autoscanEvents(params ?? {}),
queryFn: (): Promise<AutoscanPage<AutoscanEvent>> =>
api<AutoscanEventsResponse>(path).then((data) => ({
rows: data.events ?? [],
total: data.total ?? data.events?.length ?? 0,
})),
staleTime: AUTOSCAN_ACTIVITY_REFRESH_MS,
refetchInterval: AUTOSCAN_ACTIVITY_REFRESH_MS,
// Hold the prior page on screen while the next one loads so paging through
// history never flashes an empty/loading state.
placeholderData: keepPreviousData,
enabled: params?.enabled ?? true,
});
}
export function useAutoscanScans(params?: {
status?: AutoscanScanStatus;
query?: string;
limit?: number;
offset?: number;
enabled?: boolean;
}) {
const queryParams = new URLSearchParams();
if (params?.status) queryParams.set("status", params.status);
if (params?.query) queryParams.set("q", params.query);
if (params?.limit != null) queryParams.set("limit", String(params.limit));
if (params?.offset != null) queryParams.set("offset", String(params.offset));
const suffix = queryParams.toString();
const path = suffix ? `/admin/autoscan/scans?${suffix}` : "/admin/autoscan/scans";
return useQuery({
queryKey: adminKeys.autoscanScans(params ?? {}),
queryFn: (): Promise<AutoscanPage<AutoscanScan>> =>
api<AutoscanScansResponse>(path).then((data) => ({
rows: data.scans ?? [],
total: data.total ?? data.scans?.length ?? 0,
})),
staleTime: AUTOSCAN_ACTIVITY_REFRESH_MS,
refetchInterval: AUTOSCAN_ACTIVITY_REFRESH_MS,
placeholderData: keepPreviousData,
enabled: params?.enabled ?? true,
});
}
// --- Trigger ---
export function useTriggerAutoscan() {
return useMutation({
mutationFn: () =>
api<{ status: string }>("/admin/autoscan/trigger", {
method: "POST",
}),
onSuccess: () => {
toast.success("Autoscan triggered");
},
onError: (err) => {
toast.error(err instanceof Error ? err.message : "Failed to trigger autoscan");
},
});
}
web/src/lib/autoscanLabels.test.ts
+114
View File
@@ -0,0 +1,114 @@
import { describe, expect, it } from "vitest";
import type { AutoscanSource } from "@/api/types";
import {
composeSourceLabel,
resolveEventSourceName,
type SourceLabelLookups,
} from "./autoscanLabels";
describe("composeSourceLabel", () => {
const base = { capabilityId: "arr", installationId: 4 };
it("uses the operator label first, demoting connection to detail", () => {
expect(
composeSourceLabel({ ...base, operatorLabel: "4K Movies", connectionName: "Radarr4k" }),
).toEqual({ name: "4K Movies", detail: "Radarr4k · plugin #4" });
});
it("uses the connection name when no operator label", () => {
expect(
composeSourceLabel({ ...base, connectionName: "Radarr4k", displayName: "Arr Watcher" }),
).toEqual({ name: "Radarr4k", detail: "Arr Watcher · plugin #4" });
});
it("uses the manifest display name when no connection", () => {
expect(
composeSourceLabel({
capabilityId: "cephfs",
installationId: 5,
displayName: "CephFS Watcher",
}),
).toEqual({ name: "CephFS Watcher", detail: "plugin #5" });
});
it("falls back to capability id when nothing else is set", () => {
expect(composeSourceLabel(base)).toEqual({ name: "arr", detail: "plugin #4" });
});
it("ignores whitespace-only rungs", () => {
expect(composeSourceLabel({ ...base, operatorLabel: " ", connectionName: " " })).toEqual({
name: "arr",
detail: "plugin #4",
});
});
});
describe("resolveEventSourceName", () => {
const source: AutoscanSource = {
id: "src-1",
installation_id: 4,
capability_id: "arr",
connection_id: "conn-1",
enabled: true,
poll_interval_seconds: null,
last_run_at: null,
last_error: null,
path_rewrites: [],
source_config: {},
label: "",
};
const lookups: SourceLabelLookups = {
sourceByID: new Map([["src-1", source]]),
connectionByID: new Map([["conn-1", "Radarr4k"]]),
displayNames: new Map([["4:arr", "Arr Watcher"]]),
};
it("resolves the connection name via the source reference", () => {
expect(
resolveEventSourceName(
{ source_id: "src-1", capability_id: "arr", installation_id: 4 },
lookups,
),
).toBe("Radarr4k");
});
it("prefers the operator label on the source", () => {
const withLabel: SourceLabelLookups = {
...lookups,
sourceByID: new Map([["src-1", { ...source, label: "4K Movies" }]]),
};
expect(
resolveEventSourceName(
{ source_id: "src-1", capability_id: "arr", installation_id: 4 },
withLabel,
),
).toBe("4K Movies");
});
it("falls back to display name when the source was deleted (null source_id)", () => {
expect(
resolveEventSourceName(
{ source_id: null, capability_id: "arr", installation_id: 4 },
lookups,
),
).toBe("Arr Watcher");
});
it("returns empty string when the reference has no capability", () => {
expect(resolveEventSourceName({ source_id: null }, lookups)).toBe("");
});
it("falls back to display name when the bound connection is missing (deleted)", () => {
const orphaned: SourceLabelLookups = {
...lookups,
sourceByID: new Map([["src-1", { ...source, connection_id: "conn-gone" }]]),
};
expect(
resolveEventSourceName(
{ source_id: "src-1", capability_id: "arr", installation_id: 4 },
orphaned,
),
).toBe("Arr Watcher");
});
});
web/src/lib/autoscanLabels.ts
+85
View File
@@ -0,0 +1,85 @@
import type { AutoscanAvailableSource, AutoscanSource } from "@/api/types";
export interface SourceLabelParts {
operatorLabel?: string | null;
connectionName?: string | null;
displayName?: string | null;
capabilityId: string;
installationId: number;
}
export interface SourceLabel {
name: string;
detail: string;
}
/**
* Resolve a scan source's display label through four rungs, most-specific first:
* operator label -> connection name -> manifest display_name -> capability_id.
* The winning rung is `name`; the remaining plugin identity is `detail`.
*/
export function composeSourceLabel(parts: SourceLabelParts): SourceLabel {
const operator = parts.operatorLabel?.trim() ?? "";
const connection = parts.connectionName?.trim() ?? "";
const display = parts.displayName?.trim() ?? "";
const pluginSuffix = `plugin #${parts.installationId}`;
const pluginIdentity = display || parts.capabilityId;
if (operator) {
return { name: operator, detail: `${connection || pluginIdentity} · ${pluginSuffix}` };
}
if (connection) {
return { name: connection, detail: `${pluginIdentity} · ${pluginSuffix}` };
}
if (display) {
return { name: display, detail: pluginSuffix };
}
return { name: parts.capabilityId, detail: pluginSuffix };
}
/** Stable key for the (installation, capability) -> manifest display_name map. */
export function pluginDisplayNameKey(installationId: number, capabilityId: string): string {
return `${installationId}:${capabilityId}`;
}
/** Build the (installation, capability) -> display_name lookup from the picker list. */
export function buildPluginDisplayNames(available: AutoscanAvailableSource[]): Map<string, string> {
const map = new Map<string, string>();
for (const a of available) {
map.set(pluginDisplayNameKey(a.installation_id, a.capability_id), a.display_name);
}
return map;
}
/** Maps the Activity panel uses to resolve an event/scan's source label. */
export interface SourceLabelLookups {
sourceByID: Map<string, AutoscanSource>;
connectionByID: Map<string, string>;
displayNames: Map<string, string>;
}
/**
* Resolve the `name` rung for an Activity event/scan that references a source by
* id. Returns "" when the reference carries no capability (caller supplies its
* own fallback, e.g. "Autoscan").
*/
export function resolveEventSourceName(
ref: { source_id?: string | null; capability_id?: string; installation_id?: number | null },
lookups: SourceLabelLookups,
): string {
if (!ref.capability_id || ref.installation_id == null) return "";
const source = ref.source_id ? lookups.sourceByID.get(ref.source_id) : undefined;
const connectionName = source?.connection_id
? lookups.connectionByID.get(source.connection_id)
: undefined;
const displayName = lookups.displayNames.get(
pluginDisplayNameKey(ref.installation_id, ref.capability_id),
);
return composeSourceLabel({
operatorLabel: source?.label,
connectionName,
displayName,
capabilityId: ref.capability_id,
installationId: ref.installation_id,
}).name;
}
web/src/lib/scanRuns.ts
+88
View File
@@ -0,0 +1,88 @@
import type { ScanRun } from "@/api/types";
export function compareActiveScans(left: ScanRun, right: ScanRun) {
const leftRank = left.status === "running" ? 0 : 1;
const rightRank = right.status === "running" ? 0 : 1;
if (leftRank !== rightRank) {
return leftRank - rightRank;
}
const modeCompare = formatActiveScanMode(left).localeCompare(formatActiveScanMode(right));
if (modeCompare !== 0) {
return modeCompare;
}
return (left.path ?? "").localeCompare(right.path ?? "");
}
export function formatActiveScanMode(scan: Pick<ScanRun, "mode">) {
switch (scan.mode) {
case "library":
return "Full library scan";
case "subtree":
return "Subtree scan";
case "file":
return "Single file scan";
default:
return scan.mode;
}
}
export function formatActiveScanTrigger(trigger: string) {
switch (trigger) {
case "autoscan":
return "Autoscan";
case "library_created":
return "Created";
case "library_paths_changed":
case "library_updated":
return "Updated";
case "library_id":
case "manual":
return "Manual";
case "task:scan_libraries":
case "task_scan_libraries":
return "Scheduled";
default:
return trigger.replace(/_/g, " ");
}
}
export function formatActiveScanTime(iso: string | undefined, prefix: string) {
if (!iso) {
return prefix;
}
const date = new Date(iso);
if (Number.isNaN(date.getTime())) {
return prefix;
}
const seconds = Math.max(0, Math.floor((Date.now() - date.getTime()) / 1000));
if (seconds < 60) {
return `${prefix} just now`;
}
if (seconds < 3600) {
const minutes = Math.floor(seconds / 60);
return `${prefix} ${minutes}m ago`;
}
if (seconds < 86400) {
const hours = Math.floor(seconds / 3600);
return `${prefix} ${hours}h ago`;
}
const days = Math.floor(seconds / 86400);
return `${prefix} ${days}d ago`;
}
export function formatActiveScanProgress(scan: ScanRun) {
const result = scan.result;
if (!result) {
return "";
}
if (result.total_files && result.files_processed) {
const percent = Math.max(
0,
Math.min(100, Math.round((result.files_processed / result.total_files) * 100)),
);
return `${result.message ?? "Processing files"} · ${result.files_processed.toLocaleString()} / ${result.total_files.toLocaleString()} (${percent}%)`;
}
return result.message ?? "";
}
web/src/pages/AdminAutoscan.tsx
+220
View File
@@ -0,0 +1,220 @@
import { useState } from "react";
import { useSearchParams } from "react-router";
import { Play } from "lucide-react";
import type { AutoscanSettings } from "@/api/types";
import { Badge } from "@/components/ui/badge";
import { Button } from "@/components/ui/button";
import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label";
import { Switch } from "@/components/ui/switch";
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs";
import {
useAutoscanSettings,
useTriggerAutoscan,
useUpdateAutoscanSettings,
} from "@/hooks/queries/useAutoscan";
import ConnectionsPanel from "@/pages/admin/autoscan/ConnectionsPanel";
import ActivityPanel from "@/pages/admin/autoscan/ActivityPanel";
import SourcesPanel from "@/pages/admin/autoscan/SourcesPanel";
// ---------------------------------------------------------------------------
// Tab routing helpers
// ---------------------------------------------------------------------------
const AUTOSCAN_TABS = ["sources", "activity", "connections", "settings"] as const;
type AutoscanTab = (typeof AUTOSCAN_TABS)[number];
function normalizeTab(value: string | null): AutoscanTab {
return AUTOSCAN_TABS.includes(value as AutoscanTab) ? (value as AutoscanTab) : "sources";
}
// ---------------------------------------------------------------------------
// Settings tab
// ---------------------------------------------------------------------------
function SettingsTab() {
const settings = useAutoscanSettings();
const updateSettings = useUpdateAutoscanSettings();
// Local form state — initialised from server data; reflected immediately on
// every mutation so the UI stays responsive without waiting for refetch.
const [form, setForm] = useState<AutoscanSettings | null>(null);
// Merge server data into local form on first load (and after invalidation).
const serverData = settings.data;
const effective: AutoscanSettings = form ??
serverData ?? { enabled: false, default_poll_interval_seconds: 300, debounce_seconds: 10 };
function patch(delta: Partial<AutoscanSettings>) {
setForm((prev) => ({
...(prev ?? effective),
...delta,
}));
}
function save(override?: Partial<AutoscanSettings>) {
const body: AutoscanSettings = { ...effective, ...override };
updateSettings.mutate(body, {
onSuccess: () => setForm(null), // reset to server truth after save
});
}
if (settings.isLoading) {
return <p className="text-muted-foreground py-4 text-sm">Loading settings…</p>;
}
return (
<div className="max-w-lg space-y-6">
<p className="text-muted-foreground text-sm">
Autoscan can be enabled or disabled from the toggle in the page header. Tune polling
behavior below.
</p>
{/* Default poll interval */}
<div className="space-y-1.5">
<Label htmlFor="default-poll-interval">Default poll interval (seconds)</Label>
<div className="flex items-center gap-2">
<Input
id="default-poll-interval"
className="w-32"
type="number"
min={1}
value={effective.default_poll_interval_seconds}
onChange={(e) =>
patch({ default_poll_interval_seconds: Number(e.target.value) || 300 })
}
onBlur={() => save()}
/>
<span className="text-muted-foreground text-sm">sec</span>
</div>
<p className="text-muted-foreground text-xs">
Used for sources that have no per-source interval set.
</p>
</div>
{/* Debounce */}
<div className="space-y-1.5">
<Label htmlFor="debounce-seconds">Debounce (seconds)</Label>
<div className="flex items-center gap-2">
<Input
id="debounce-seconds"
className="w-32"
type="number"
min={0}
value={effective.debounce_seconds}
onChange={(e) => patch({ debounce_seconds: Number(e.target.value) || 0 })}
onBlur={() => save()}
/>
<span className="text-muted-foreground text-sm">sec</span>
</div>
<p className="text-muted-foreground text-xs">
Coalesces rapid change events before triggering a scan.
</p>
</div>
</div>
);
}
// ---------------------------------------------------------------------------
// Page
// ---------------------------------------------------------------------------
export default function AdminAutoscan() {
const [searchParams, setSearchParams] = useSearchParams();
const activeTab = normalizeTab(searchParams.get("tab"));
const trigger = useTriggerAutoscan();
const settings = useAutoscanSettings();
const updateSettings = useUpdateAutoscanSettings();
const enabled = settings.data?.enabled ?? false;
function toggleEnabled(checked: boolean) {
if (!settings.data) return;
updateSettings.mutate({ ...settings.data, enabled: checked });
}
function setActiveTab(value: string) {
const next = new URLSearchParams(searchParams);
if (value === "sources") {
next.delete("tab");
} else {
next.set("tab", value);
}
setSearchParams(next, { replace: true });
}
return (
<div className="space-y-6">
<div className="page-header">
<div className="space-y-2">
<div className="flex items-center gap-2.5">
<h1 className="text-3xl font-semibold tracking-normal text-balance sm:text-4xl">
Autoscan
</h1>
{settings.data &&
(enabled ? (
<Badge variant="secondary">Enabled</Badge>
) : (
<Badge variant="outline" className="text-muted-foreground">
Disabled
</Badge>
))}
</div>
<p className="text-muted-foreground max-w-2xl text-sm leading-6">
Automatically detect library changes from installed scan-source plugins. Install them
from the Plugins page, configure each source, bind a connection if it needs one, and
tune polling intervals.
</p>
</div>
<div className="flex items-center gap-3">
<div className="flex items-center gap-2">
<Label htmlFor="autoscan-enabled" className="text-muted-foreground text-sm">
Autoscan
</Label>
<Switch
id="autoscan-enabled"
checked={enabled}
onCheckedChange={toggleEnabled}
disabled={!settings.data || updateSettings.isPending}
aria-label="Enable autoscan"
/>
</div>
<Button
variant="outline"
size="sm"
disabled={trigger.isPending}
onClick={() => trigger.mutate()}
>
<Play />
{trigger.isPending ? "Triggering…" : "Run now"}
</Button>
</div>
</div>
<Tabs value={activeTab} onValueChange={setActiveTab} className="gap-5">
<TabsList variant="line" className="border-border w-full justify-start border-b">
<TabsTrigger value="sources">Sources</TabsTrigger>
<TabsTrigger value="activity">Activity</TabsTrigger>
<TabsTrigger value="connections">Connections</TabsTrigger>
<TabsTrigger value="settings">Settings</TabsTrigger>
</TabsList>
<TabsContent value="sources">
<SourcesPanel />
</TabsContent>
<TabsContent value="activity">
<ActivityPanel />
</TabsContent>
<TabsContent value="connections">
<ConnectionsPanel />
</TabsContent>
<TabsContent value="settings">
<SettingsTab />
</TabsContent>
</Tabs>
</div>
);
}
web/src/pages/AdminLibraries.tsx
+7 -85
View File
@@ -101,6 +101,13 @@ import { ConfirmDialog } from "@/components/ConfirmDialog";
import FolderBrowser from "@/components/FolderBrowser";
import PathAutocompleteInput from "@/components/PathAutocompleteInput";
import { cn } from "@/lib/utils";
import {
compareActiveScans,
formatActiveScanMode,
formatActiveScanProgress,
formatActiveScanTime,
formatActiveScanTrigger,
} from "@/lib/scanRuns";
import {
DndContext,
DragOverlay,
@@ -802,91 +809,6 @@ function getLibraryScanGroupName(library: Library | null, libraryID: number) {
return library?.name ?? `Library #${libraryID}`;
}
function compareActiveScans(left: ScanRun, right: ScanRun) {
const leftRank = left.status === "running" ? 0 : 1;
const rightRank = right.status === "running" ? 0 : 1;
if (leftRank !== rightRank) {
return leftRank - rightRank;
}
const modeCompare = formatActiveScanMode(left).localeCompare(formatActiveScanMode(right));
if (modeCompare !== 0) {
return modeCompare;
}
return (left.path ?? "").localeCompare(right.path ?? "");
}
function formatActiveScanMode(scan: Pick<ScanRun, "mode">) {
switch (scan.mode) {
case "library":
return "Full library scan";
case "subtree":
return "Subtree scan";
case "file":
return "Single file scan";
default:
return scan.mode;
}
}
function formatActiveScanTrigger(trigger: string) {
switch (trigger) {
case "library_created":
return "Created";
case "library_paths_changed":
case "library_updated":
return "Updated";
case "library_id":
case "manual":
return "Manual";
case "task:scan_libraries":
case "task_scan_libraries":
return "Scheduled";
default:
return trigger.replace(/_/g, " ");
}
}
function formatActiveScanTime(iso: string | undefined, prefix: string) {
if (!iso) {
return prefix;
}
const date = new Date(iso);
if (Number.isNaN(date.getTime())) {
return prefix;
}
const seconds = Math.max(0, Math.floor((Date.now() - date.getTime()) / 1000));
if (seconds < 60) {
return `${prefix} just now`;
}
if (seconds < 3600) {
const minutes = Math.floor(seconds / 60);
return `${prefix} ${minutes}m ago`;
}
if (seconds < 86400) {
const hours = Math.floor(seconds / 3600);
return `${prefix} ${hours}h ago`;
}
const days = Math.floor(seconds / 86400);
return `${prefix} ${days}d ago`;
}
function formatActiveScanProgress(scan: ScanRun) {
const result = scan.result;
if (!result) {
return "";
}
if (result.total_files && result.files_processed) {
const percent = Math.max(
0,
Math.min(100, Math.round((result.files_processed / result.total_files) * 100)),
);
return `${result.message ?? "Processing files"} · ${result.files_processed.toLocaleString()} / ${result.total_files.toLocaleString()} (${percent}%)`;
}
return result.message ?? "";
}
function getLibraryRefreshLibraryID(job: AdminJob): number | null {
const value = (job.request_payload as { library_id?: unknown }).library_id;
if (typeof value === "number" && Number.isFinite(value)) {
web/src/pages/admin/autoscan/ActivityPanel.tsx
+924
View File
@@ -0,0 +1,924 @@
import { useMemo, useState, type ReactNode } from "react";
import {
AlertTriangle,
CheckCircle2,
Clock,
History,
ListChecks,
RefreshCw,
ScanLine,
Search,
Square,
X,
} from "lucide-react";
import type {
AutoscanEvent,
AutoscanEventScanRun,
AutoscanEventStatus,
AutoscanScan,
AutoscanScanStatus,
Library,
ScanRun,
} from "@/api/types";
import { Badge } from "@/components/ui/badge";
import { Button } from "@/components/ui/button";
import { Input } from "@/components/ui/input";
import { TablePagination } from "@/components/ui/pagination";
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select";
import { Skeleton } from "@/components/ui/skeleton";
import {
Table,
TableBody,
TableCell,
TableHead,
TableHeader,
TableRow,
} from "@/components/ui/table";
import { useEventChannel } from "@/components/realtimeEventsContext";
import { useDebounce } from "@/hooks/useDebounce";
import {
useAutoscanConnections,
useAutoscanEvents,
useAutoscanScans,
useAutoscanSources,
useAutoscanStatus,
useAvailableScanSources,
} from "@/hooks/queries/useAutoscan";
import { useActiveScans } from "@/hooks/queries/admin/scans";
import { useAdminLibraries, useCancelLibraryScans } from "@/hooks/queries/admin/libraries";
import {
buildPluginDisplayNames,
resolveEventSourceName,
type SourceLabelLookups,
} from "@/lib/autoscanLabels";
import {
compareActiveScans,
formatActiveScanMode,
formatActiveScanProgress,
formatActiveScanTime,
formatActiveScanTrigger,
} from "@/lib/scanRuns";
import { cn } from "@/lib/utils";
const HISTORY_PAGE_SIZE_OPTIONS = [25, 50, 100];
const DEFAULT_HISTORY_PAGE_SIZE = 25;
const QUEUE_PAGE_SIZE = 8;
type HistoryView = "scans" | "polls";
function formatTime(value: string | undefined): string {
if (!value) return "-";
return new Date(value).toLocaleTimeString([], { hour: "2-digit", minute: "2-digit" });
}
function formatTimestamp(value: string | undefined): string {
if (!value) return "-";
return new Date(value).toLocaleString([], {
month: "short",
day: "numeric",
hour: "2-digit",
minute: "2-digit",
});
}
function formatDuration(ms: number): string {
if (ms < 1000) return `${ms}ms`;
if (ms < 60_000) return `${Math.round(ms / 100) / 10}s`;
const minutes = Math.floor(ms / 60_000);
const seconds = Math.round((ms % 60_000) / 1000);
return `${minutes}m ${seconds}s`;
}
function eventStatusTone(status: AutoscanEventStatus): {
label: string;
icon: typeof CheckCircle2;
className: string;
} {
switch (status) {
case "success":
return {
label: "Success",
icon: CheckCircle2,
className: "border-emerald-500/30 bg-emerald-500/10 text-emerald-500",
};
case "unresolved":
return {
label: "Unresolved",
icon: AlertTriangle,
className: "border-amber-500/30 bg-amber-500/10 text-amber-500",
};
case "error":
return {
label: "Error",
icon: AlertTriangle,
className: "border-destructive/30 bg-destructive/10 text-destructive",
};
}
}
function scanStatusClass(status: AutoscanEventScanRun["status"] | ScanRun["status"]): string {
switch (status) {
case "completed":
return "border-emerald-500/30 bg-emerald-500/10 text-emerald-500";
case "failed":
return "border-destructive/30 bg-destructive/10 text-destructive";
case "running":
return "border-sky-500/30 bg-sky-500/10 text-sky-500";
case "accepted":
return "border-muted-foreground/25 bg-muted/60 text-muted-foreground";
case "cancelled":
return "border-amber-500/30 bg-amber-500/10 text-amber-500";
}
}
function scanStatusLabel(status: AutoscanScanStatus | ScanRun["status"]) {
return status === "accepted" ? "queued" : status;
}
// arr-plugin sources fan out one-per-connection under a single generic
// capability, so resolve every Activity row through the shared label chain:
// operator label -> connection name -> manifest display_name -> capability_id.
function pollSourceName(event: AutoscanEvent, lookups: SourceLabelLookups): string {
return (
resolveEventSourceName(event, lookups) || `${event.capability_id} #${event.installation_id}`
);
}
function scanSourceName(scan: AutoscanScan, lookups: SourceLabelLookups): string {
return resolveEventSourceName(scan, lookups) || "Autoscan";
}
function libraryName(librariesByID: Map<number, Library>, libraryID: number): string {
return librariesByID.get(libraryID)?.name ?? `Library #${libraryID}`;
}
function PollStatusBadge({ status }: { status: AutoscanEventStatus }) {
const tone = eventStatusTone(status);
const Icon = tone.icon;
return (
<Badge variant="outline" className={tone.className}>
<Icon className="h-3.5 w-3.5" />
{tone.label}
</Badge>
);
}
function ScanStatusBadge({ status }: { status: AutoscanScanStatus | ScanRun["status"] }) {
return (
<Badge variant="outline" className={cn("capitalize tabular-nums", scanStatusClass(status))}>
{scanStatusLabel(status)}
</Badge>
);
}
// Shared desktop table chrome so the queue, scan history, and poll log read as
// one family: clipped rounded border, muted sticky-feeling header band.
function DataTable({ head, children }: { head: ReactNode; children: ReactNode }) {
return (
<div className="hidden overflow-hidden rounded-lg border lg:block">
<Table>
<TableHeader className="bg-muted/40">
<TableRow className="hover:bg-transparent">{head}</TableRow>
</TableHeader>
<TableBody>{children}</TableBody>
</Table>
</div>
);
}
function RunList({ runs }: { runs: AutoscanEventScanRun[] }) {
if (runs.length === 0) {
return <span className="text-muted-foreground text-xs">No new scan rows were created.</span>;
}
return (
<div className="space-y-2">
{runs.map((run) => (
<div
key={run.id}
className="border-border/70 bg-background/40 grid gap-1 rounded-md border px-3 py-2 text-xs sm:grid-cols-[auto_1fr_auto] sm:items-center"
>
<ScanStatusBadge status={run.status} />
<div className="min-w-0">
<div className="font-medium">{run.mode}</div>
<div className="text-muted-foreground [overflow-wrap:anywhere]">
{run.path || "Entire library"}
</div>
</div>
<div className="text-muted-foreground whitespace-nowrap tabular-nums">
{formatTime(run.completed_at ?? run.started_at ?? run.requested_at)}
</div>
</div>
))}
</div>
);
}
function PollMetricStrip({ event }: { event: AutoscanEvent }) {
return (
<div className="text-muted-foreground flex flex-wrap gap-x-4 gap-y-1 text-xs tabular-nums">
<span>{event.changes_returned} changes</span>
<span>{event.targets_claimed} targets</span>
<span>{event.scans_created} created</span>
<span>{event.scans_reused} reused</span>
<span>{event.scans_suppressed} suppressed</span>
</div>
);
}
function QueueCard({
scan,
librariesByID,
cancellingLibraryID,
onCancel,
}: {
scan: ScanRun;
librariesByID: Map<number, Library>;
cancellingLibraryID: number | null;
onCancel: (libraryID: number) => void;
}) {
const progress = formatActiveScanProgress(scan);
return (
<div className="border-border rounded-lg border p-4">
<div className="flex flex-wrap items-start justify-between gap-3">
<div className="min-w-0 space-y-1">
<div className="flex flex-wrap items-center gap-2">
<ScanStatusBadge status={scan.status} />
<span className="font-medium">{libraryName(librariesByID, scan.library_id)}</span>
</div>
<div className="text-muted-foreground flex flex-wrap gap-x-3 gap-y-1 text-xs">
<span>{formatActiveScanMode(scan)}</span>
<span>{formatActiveScanTrigger(scan.trigger)}</span>
<span>
{scan.status === "running"
? formatActiveScanTime(scan.started_at, "Started")
: "Waiting for capacity"}
</span>
</div>
</div>
<Button
variant="ghost"
size="icon"
className="text-destructive h-8 w-8"
disabled={cancellingLibraryID === scan.library_id}
onClick={() => onCancel(scan.library_id)}
aria-label="Cancel scans for this library"
>
<Square className="h-3.5 w-3.5" />
</Button>
</div>
<div className="text-muted-foreground mt-3 font-mono text-xs [overflow-wrap:anywhere]">
{scan.path || "Entire library"}
</div>
{progress ? (
<div className="text-muted-foreground mt-2 text-xs [overflow-wrap:anywhere]">
{progress}
</div>
) : null}
</div>
);
}
function AutoscanQueue({
scans,
statusActiveCount,
librariesByID,
cancellingLibraryID,
onCancel,
}: {
scans: ScanRun[];
statusActiveCount: number;
librariesByID: Map<number, Library>;
cancellingLibraryID: number | null;
onCancel: (libraryID: number) => void;
}) {
const [page, setPage] = useState(0);
const pageCount = Math.max(1, Math.ceil(scans.length / QUEUE_PAGE_SIZE));
// Live scans complete out from under us; the active set length is known
// synchronously, so derive the in-range page during render (no effect needed)
// and the view self-heals when the queue shrinks below the current offset.
const safePage = Math.min(page, pageCount - 1);
const rows = scans.slice(safePage * QUEUE_PAGE_SIZE, (safePage + 1) * QUEUE_PAGE_SIZE);
if (scans.length === 0) {
return (
<div className="border-border rounded-lg border border-dashed p-6">
<div className="flex items-center gap-2 text-sm font-medium">
<ScanLine className="text-muted-foreground h-4 w-4" />
Autoscan queue
</div>
<p className="text-muted-foreground mt-2 text-sm">
{statusActiveCount > 0
? "Queue counts are active, but live scan details have not arrived yet."
: "No autoscan scans are queued or running."}
</p>
</div>
);
}
return (
<section className="space-y-3">
<div className="flex flex-wrap items-center justify-between gap-2">
<div>
<div className="flex items-center gap-2 text-sm font-semibold">
<ScanLine className="text-primary h-4 w-4" />
Autoscan queue
</div>
<p className="text-muted-foreground mt-1 text-xs">
Live scans created by autoscan sources.
</p>
</div>
<Badge variant="secondary" className="tabular-nums">
{scans.length} active
</Badge>
</div>
<div className="space-y-3 lg:hidden">
{rows.map((scan) => (
<QueueCard
key={scan.id}
scan={scan}
librariesByID={librariesByID}
cancellingLibraryID={cancellingLibraryID}
onCancel={onCancel}
/>
))}
</div>
<DataTable
head={
<>
<TableHead>Status</TableHead>
<TableHead>Library</TableHead>
<TableHead>Scope</TableHead>
<TableHead>Progress</TableHead>
<TableHead className="w-20 text-right">Action</TableHead>
</>
}
>
{rows.map((scan) => {
const progress = formatActiveScanProgress(scan);
return (
<TableRow key={scan.id}>
<TableCell>
<ScanStatusBadge status={scan.status} />
</TableCell>
<TableCell className="font-medium">
{libraryName(librariesByID, scan.library_id)}
<div className="text-muted-foreground mt-1 text-xs">
{scan.status === "running"
? formatActiveScanTime(scan.started_at, "Started")
: "Waiting for capacity"}
</div>
</TableCell>
<TableCell className="max-w-xl">
<div className="text-sm">{formatActiveScanMode(scan)}</div>
<div className="text-muted-foreground mt-1 font-mono text-xs [overflow-wrap:anywhere]">
{scan.path || "Entire library"}
</div>
</TableCell>
<TableCell className="text-muted-foreground text-xs [overflow-wrap:anywhere]">
{progress || formatActiveScanTrigger(scan.trigger)}
</TableCell>
<TableCell className="text-right">
<Button
variant="ghost"
size="icon"
className="text-destructive h-8 w-8"
disabled={cancellingLibraryID === scan.library_id}
onClick={() => onCancel(scan.library_id)}
aria-label="Cancel scans for this library"
>
<Square className="h-3.5 w-3.5" />
</Button>
</TableCell>
</TableRow>
);
})}
</DataTable>
{scans.length > QUEUE_PAGE_SIZE ? (
<TablePagination
page={safePage}
pageSize={QUEUE_PAGE_SIZE}
total={scans.length}
onPageChange={setPage}
itemNoun="scan"
/>
) : null}
</section>
);
}
function ScanHistoryCard({
scan,
librariesByID,
lookups,
}: {
scan: AutoscanScan;
librariesByID: Map<number, Library>;
lookups: SourceLabelLookups;
}) {
return (
<div className="border-border rounded-lg border p-4">
<div className="flex flex-wrap items-start justify-between gap-3">
<div className="min-w-0 space-y-1">
<div className="flex flex-wrap items-center gap-2">
<ScanStatusBadge status={scan.status} />
<span className="font-medium">{libraryName(librariesByID, scan.library_id)}</span>
</div>
<div className="text-muted-foreground flex flex-wrap gap-x-3 gap-y-1 text-xs">
<span>{formatActiveScanMode(scan)}</span>
<span>{scanSourceName(scan, lookups)}</span>
<span>
{formatTimestamp(scan.completed_at ?? scan.started_at ?? scan.requested_at)}
</span>
</div>
</div>
{scan.event_status ? <PollStatusBadge status={scan.event_status} /> : null}
</div>
<div className="text-muted-foreground mt-3 font-mono text-xs [overflow-wrap:anywhere]">
{scan.path || "Entire library"}
</div>
{scan.error_message ? (
<div className="text-destructive mt-2 text-xs [overflow-wrap:anywhere]">
{scan.error_message}
</div>
) : null}
<div className="text-muted-foreground mt-3 font-mono text-[11px] [overflow-wrap:anywhere]">
{scan.id}
</div>
</div>
);
}
function ScanHistoryTable({
scans,
librariesByID,
lookups,
}: {
scans: AutoscanScan[];
librariesByID: Map<number, Library>;
lookups: SourceLabelLookups;
}) {
return (
<>
<div className="space-y-3 lg:hidden">
{scans.map((scan) => (
<ScanHistoryCard
key={scan.id}
scan={scan}
librariesByID={librariesByID}
lookups={lookups}
/>
))}
</div>
<DataTable
head={
<>
<TableHead>Status</TableHead>
<TableHead>Library</TableHead>
<TableHead>Source</TableHead>
<TableHead>Scope</TableHead>
<TableHead>Time</TableHead>
<TableHead>Poll</TableHead>
</>
}
>
{scans.map((scan) => (
<TableRow key={scan.id}>
<TableCell>
<ScanStatusBadge status={scan.status} />
</TableCell>
<TableCell className="font-medium">
{libraryName(librariesByID, scan.library_id)}
<div className="text-muted-foreground mt-1 font-mono text-[11px]">{scan.id}</div>
</TableCell>
<TableCell className="whitespace-nowrap">{scanSourceName(scan, lookups)}</TableCell>
<TableCell className="max-w-xl">
<div className="text-sm">{formatActiveScanMode(scan)}</div>
<div className="text-muted-foreground mt-1 font-mono text-xs [overflow-wrap:anywhere]">
{scan.path || "Entire library"}
</div>
{scan.error_message ? (
<div className="text-destructive mt-1 text-xs [overflow-wrap:anywhere]">
{scan.error_message}
</div>
) : null}
</TableCell>
<TableCell className="text-muted-foreground whitespace-nowrap tabular-nums">
{formatTimestamp(scan.completed_at ?? scan.started_at ?? scan.requested_at)}
</TableCell>
<TableCell>
{scan.event_status ? <PollStatusBadge status={scan.event_status} /> : "-"}
</TableCell>
</TableRow>
))}
</DataTable>
</>
);
}
function PollEventCard({ event, lookups }: { event: AutoscanEvent; lookups: SourceLabelLookups }) {
return (
<div className="border-border rounded-lg border p-4">
<div className="flex flex-wrap items-start justify-between gap-3">
<div className="min-w-0 space-y-1">
<div className="flex flex-wrap items-center gap-2">
<PollStatusBadge status={event.status} />
<span className="font-medium">{pollSourceName(event, lookups)}</span>
</div>
<PollMetricStrip event={event} />
</div>
<div className="text-muted-foreground text-right text-xs tabular-nums">
<div>{formatTimestamp(event.completed_at)}</div>
<div>{formatDuration(event.duration_ms)}</div>
</div>
</div>
{event.error_message ? (
<p className="text-destructive mt-3 text-xs [overflow-wrap:anywhere]">
{event.error_message}
</p>
) : null}
<details className="mt-3">
<summary className="text-muted-foreground cursor-pointer text-xs">
{event.scan_runs.length} linked scan {event.scan_runs.length === 1 ? "run" : "runs"}
</summary>
<div className="mt-2">
<RunList runs={event.scan_runs} />
</div>
</details>
</div>
);
}
function PollEventTable({
events,
lookups,
}: {
events: AutoscanEvent[];
lookups: SourceLabelLookups;
}) {
return (
<>
<div className="space-y-3 lg:hidden">
{events.map((event) => (
<PollEventCard key={event.id} event={event} lookups={lookups} />
))}
</div>
<DataTable
head={
<>
<TableHead>Status</TableHead>
<TableHead>Source</TableHead>
<TableHead>Counts</TableHead>
<TableHead>Scans</TableHead>
<TableHead>Duration</TableHead>
<TableHead>Completed</TableHead>
</>
}
>
{events.map((event) => (
<TableRow key={event.id}>
<TableCell>
<PollStatusBadge status={event.status} />
</TableCell>
<TableCell className="font-medium">{pollSourceName(event, lookups)}</TableCell>
<TableCell>
<PollMetricStrip event={event} />
{event.error_message ? (
<div className="text-destructive mt-1 max-w-md text-xs [overflow-wrap:anywhere]">
{event.error_message}
</div>
) : null}
</TableCell>
<TableCell>
<details>
<summary className="text-muted-foreground cursor-pointer text-xs">
{event.scan_runs.length} linked
</summary>
<div className="mt-2 w-[min(42rem,70vw)]">
<RunList runs={event.scan_runs} />
</div>
</details>
</TableCell>
<TableCell className="whitespace-nowrap tabular-nums">
{formatDuration(event.duration_ms)}
</TableCell>
<TableCell className="text-muted-foreground whitespace-nowrap tabular-nums">
{formatTimestamp(event.completed_at)}
</TableCell>
</TableRow>
))}
</DataTable>
</>
);
}
function HistorySkeleton() {
return (
<div className="space-y-3" aria-hidden="true">
{Array.from({ length: 6 }).map((_, i) => (
<Skeleton key={i} className="h-16 w-full rounded-lg" />
))}
</div>
);
}
function StatTile({
label,
value,
icon: Icon,
}: {
label: string;
value: ReactNode;
icon?: typeof ScanLine;
}) {
return (
<div>
<div className="text-muted-foreground flex items-center gap-2 text-xs">
{Icon ? <Icon className="h-3.5 w-3.5" /> : null}
{label}
</div>
<div className="mt-1 text-2xl font-semibold tabular-nums">{value}</div>
</div>
);
}
export default function ActivityPanel() {
useEventChannel("scans");
const [historyView, setHistoryView] = useState<HistoryView>("scans");
const [historyQuery, setHistoryQuery] = useState("");
const [scanStatus, setScanStatus] = useState<AutoscanScanStatus | "all">("all");
const [pollStatus, setPollStatus] = useState<AutoscanEventStatus | "all">("all");
const [page, setPage] = useState(0);
const [pageSize, setPageSize] = useState(DEFAULT_HISTORY_PAGE_SIZE);
const debouncedHistoryQuery = useDebounce(historyQuery.trim(), 300);
const scanHistory = useAutoscanScans({
limit: pageSize,
offset: page * pageSize,
query: debouncedHistoryQuery || undefined,
status: scanStatus === "all" ? undefined : scanStatus,
enabled: historyView === "scans",
});
const pollEvents = useAutoscanEvents({
limit: pageSize,
offset: page * pageSize,
query: debouncedHistoryQuery || undefined,
status: pollStatus === "all" ? undefined : pollStatus,
enabled: historyView === "polls",
});
const status = useAutoscanStatus();
const { data: activeScans = [] } = useActiveScans();
const { data: libraries = [] } = useAdminLibraries();
const { data: autoscanSources = [] } = useAutoscanSources();
const { data: autoscanConnections = [] } = useAutoscanConnections();
const available = useAvailableScanSources();
const cancelScans = useCancelLibraryScans();
const queue = status.data;
const librariesByID = useMemo(
() => new Map(libraries.map((library) => [library.id, library])),
[libraries],
);
const labelLookups: SourceLabelLookups = useMemo(
() => ({
sourceByID: new Map(autoscanSources.map((s) => [s.id, s])),
connectionByID: new Map(autoscanConnections.map((c) => [c.id, c.name])),
displayNames: buildPluginDisplayNames(available.data ?? []),
}),
[autoscanSources, autoscanConnections, available.data],
);
const autoscanQueue = useMemo(
() =>
activeScans
.filter((scan) => scan.trigger === "autoscan")
.slice()
.sort(compareActiveScans),
[activeScans],
);
const activeQuery = historyView === "scans" ? scanHistory : pollEvents;
const activeRows = activeQuery.data?.rows ?? [];
const activeTotal = activeQuery.data?.total ?? 0;
const activeStatus = historyView === "scans" ? scanStatus : pollStatus;
const hasHistoryFilters = debouncedHistoryQuery !== "" || activeStatus !== "all";
const isRefreshing = status.isFetching || activeQuery.isFetching;
const itemNoun = historyView === "scans" ? "scan" : "poll";
// If a background refresh shrinks the result set below the current page,
// pull the viewer back to the last page that still has rows. Adjusting state
// during render (React's escape hatch) converges in one guarded step and
// avoids an extra effect pass; the offset query then refetches the valid page.
const historyPageCount = Math.max(1, Math.ceil(activeTotal / pageSize));
if (page > 0 && page >= historyPageCount && !activeQuery.isFetching) {
setPage(historyPageCount - 1);
}
function refresh() {
void status.refetch();
void activeQuery.refetch();
}
function resetHistoryFilters() {
setHistoryQuery("");
setScanStatus("all");
setPollStatus("all");
setPage(0);
}
function switchHistoryView(nextView: HistoryView) {
setHistoryView(nextView);
setPage(0);
}
return (
<div className="space-y-6">
<div className="border-border grid gap-4 rounded-lg border p-4 sm:grid-cols-4">
<StatTile label="Active" value={queue?.active_scans ?? 0} icon={ScanLine} />
<StatTile label="Queued" value={queue?.accepted_scans ?? 0} />
<StatTile label="Running" value={queue?.running_scans ?? 0} />
<div className="flex items-start justify-between gap-3 sm:block">
<StatTile label="Latest poll" value={formatTime(queue?.latest_event_at)} icon={Clock} />
<Button
variant="outline"
size="icon"
className="sm:mt-2"
onClick={refresh}
disabled={isRefreshing}
aria-label="Refresh autoscan activity"
>
<RefreshCw className={cn(isRefreshing && "animate-spin")} />
</Button>
</div>
</div>
<AutoscanQueue
scans={autoscanQueue}
statusActiveCount={queue?.active_scans ?? 0}
librariesByID={librariesByID}
cancellingLibraryID={cancelScans.variables ?? null}
onCancel={(libraryID) => cancelScans.mutate(libraryID)}
/>
<section className="space-y-3">
<div className="flex flex-col gap-3 lg:flex-row lg:items-end lg:justify-between">
<div>
<div className="flex items-center gap-2 text-sm font-semibold">
{historyView === "scans" ? (
<ListChecks className="text-primary h-4 w-4" />
) : (
<History className="text-primary h-4 w-4" />
)}
{historyView === "scans" ? "Scan history" : "Poll log"}
</div>
<p className="text-muted-foreground mt-1 text-xs">
{historyView === "scans"
? "Real scan rows created by autoscan, searchable by path, library, source, status, or scan id."
: "Diagnostic poll records from scan-source plugins."}
</p>
</div>
<div className="grid gap-2 sm:grid-cols-[auto_minmax(0,1fr)_10rem_auto] lg:w-[50rem]">
<div className="border-border bg-background flex rounded-md border p-1">
<Button
type="button"
variant={historyView === "scans" ? "secondary" : "ghost"}
size="sm"
className="h-7"
onClick={() => switchHistoryView("scans")}
>
Scans
</Button>
<Button
type="button"
variant={historyView === "polls" ? "secondary" : "ghost"}
size="sm"
className="h-7"
onClick={() => switchHistoryView("polls")}
>
Polls
</Button>
</div>
<div className="relative min-w-0">
<Search className="text-muted-foreground pointer-events-none absolute top-1/2 left-3 h-4 w-4 -translate-y-1/2" />
<Input
value={historyQuery}
onChange={(event) => {
setHistoryQuery(event.target.value);
setPage(0);
}}
placeholder={historyView === "scans" ? "Search scan history" : "Search poll log"}
className="pr-9 pl-9"
/>
{historyQuery ? (
<button
type="button"
className="text-muted-foreground hover:text-foreground absolute top-1/2 right-2 rounded p-1 transition-colors"
onClick={() => setHistoryQuery("")}
aria-label="Clear history search"
>
<X className="h-3.5 w-3.5" />
</button>
) : null}
</div>
{historyView === "scans" ? (
<Select
value={scanStatus}
onValueChange={(value) => {
setScanStatus(value as AutoscanScanStatus | "all");
setPage(0);
}}
>
<SelectTrigger className="w-full">
<SelectValue />
</SelectTrigger>
<SelectContent>
<SelectItem value="all">All statuses</SelectItem>
<SelectItem value="accepted">Queued</SelectItem>
<SelectItem value="running">Running</SelectItem>
<SelectItem value="completed">Completed</SelectItem>
<SelectItem value="failed">Failed</SelectItem>
<SelectItem value="cancelled">Cancelled</SelectItem>
</SelectContent>
</Select>
) : (
<Select
value={pollStatus}
onValueChange={(value) => {
setPollStatus(value as AutoscanEventStatus | "all");
setPage(0);
}}
>
<SelectTrigger className="w-full">
<SelectValue />
</SelectTrigger>
<SelectContent>
<SelectItem value="all">All statuses</SelectItem>
<SelectItem value="success">Success</SelectItem>
<SelectItem value="unresolved">Unresolved</SelectItem>
<SelectItem value="error">Error</SelectItem>
</SelectContent>
</Select>
)}
<Button variant="outline" onClick={resetHistoryFilters} disabled={!hasHistoryFilters}>
Reset
</Button>
</div>
</div>
{activeQuery.isLoading ? (
<HistorySkeleton />
) : activeQuery.isError ? (
<div className="border-destructive/30 bg-destructive/5 rounded-lg border p-8 text-center">
<p className="text-destructive text-sm">Failed to load autoscan activity.</p>
<Button variant="outline" size="sm" className="mt-3" onClick={refresh}>
Try again
</Button>
</div>
) : activeRows.length === 0 ? (
<div className="rounded-lg border border-dashed p-8 text-center">
<p className="text-muted-foreground text-sm">
{hasHistoryFilters
? "No autoscan activity matches those filters."
: historyView === "scans"
? "No autoscan scans have been created yet."
: "No autoscan polls have been recorded yet."}
</p>
</div>
) : (
<div className="space-y-4">
{historyView === "scans" ? (
<ScanHistoryTable
scans={activeRows as AutoscanScan[]}
librariesByID={librariesByID}
lookups={labelLookups}
/>
) : (
<PollEventTable events={activeRows as AutoscanEvent[]} lookups={labelLookups} />
)}
<TablePagination
page={page}
pageSize={pageSize}
total={activeTotal}
onPageChange={setPage}
onPageSizeChange={(size) => {
setPageSize(size);
setPage(0);
}}
pageSizeOptions={HISTORY_PAGE_SIZE_OPTIONS}
itemNoun={itemNoun}
isFetching={activeQuery.isFetching}
/>
</div>
)}
</section>
</div>
);
}
web/src/pages/admin/autoscan/ConnectionsPanel.tsx
+532
View File
@@ -0,0 +1,532 @@
import { useState } from "react";
import { CheckCircle2, Pencil, Plus, Trash2, XCircle } from "lucide-react";
import type {
AutoscanConnection,
AutoscanConnectionInput,
AutoscanConnectionTestInput,
AutoscanConnectionTestResult,
} from "@/api/types";
import type { RequestIntegration } from "@/api/types";
import {
AlertDialog,
AlertDialogAction,
AlertDialogCancel,
AlertDialogContent,
AlertDialogDescription,
AlertDialogFooter,
AlertDialogHeader,
AlertDialogTitle,
} from "@/components/ui/alert-dialog";
import { Badge } from "@/components/ui/badge";
import { Button } from "@/components/ui/button";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle,
} from "@/components/ui/dialog";
import { Input } from "@/components/ui/input";
import { Label } from "@/components/ui/label";
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select";
import {
Table,
TableBody,
TableCell,
TableHead,
TableHeader,
TableRow,
} from "@/components/ui/table";
import {
useAutoscanConnections,
useCreateAutoscanConnection,
useDeleteAutoscanConnection,
useTestAutoscanConnection,
useUpdateAutoscanConnection,
} from "@/hooks/queries/useAutoscan";
import { useRequestIntegrations } from "@/hooks/queries/useRequests";
// ---------------------------------------------------------------------------
// Dialog mode types
// ---------------------------------------------------------------------------
type DialogMode = "reuse" | "own";
interface DialogState {
open: boolean;
editing: AutoscanConnection | null;
mode: DialogMode;
// "reuse" fields
requestIntegrationId: string;
// "own" fields
name: string;
kind: string;
baseUrl: string;
apiKey: string;
}
const BLANK_DIALOG: DialogState = {
open: false,
editing: null,
mode: "own",
requestIntegrationId: "",
name: "",
kind: "sonarr",
baseUrl: "",
apiKey: "",
};
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
function connectionKindLabel(kind: string): string {
if (kind === "sonarr") return "Sonarr";
if (kind === "radarr") return "Radarr";
return kind;
}
function isArrKind(integration: RequestIntegration): boolean {
return integration.kind === "sonarr" || integration.kind === "radarr";
}
// ---------------------------------------------------------------------------
// Main panel
// ---------------------------------------------------------------------------
export default function ConnectionsPanel() {
const connections = useAutoscanConnections();
const requestIntegrations = useRequestIntegrations();
const createConnection = useCreateAutoscanConnection();
const updateConnection = useUpdateAutoscanConnection();
const deleteConnection = useDeleteAutoscanConnection();
const testConnection = useTestAutoscanConnection();
const [dialog, setDialog] = useState<DialogState>(BLANK_DIALOG);
const [deleteTarget, setDeleteTarget] = useState<AutoscanConnection | null>(null);
const [testResult, setTestResult] = useState<AutoscanConnectionTestResult | null>(null);
const arrIntegrations = (requestIntegrations.data ?? []).filter(isArrKind);
// -------------------------------------------------------------------------
// Dialog helpers
// -------------------------------------------------------------------------
function openAddDialog() {
setTestResult(null);
setDialog({ ...BLANK_DIALOG, open: true });
}
function openEditDialog(conn: AutoscanConnection) {
setTestResult(null);
if (conn.request_integration_id) {
setDialog({
open: true,
editing: conn,
mode: "reuse",
requestIntegrationId: conn.request_integration_id,
name: conn.name,
kind: conn.kind,
baseUrl: "",
apiKey: "",
});
} else {
setDialog({
open: true,
editing: conn,
mode: "own",
requestIntegrationId: "",
name: conn.name,
kind: conn.kind,
baseUrl: conn.base_url ?? "",
apiKey: "",
});
}
}
function closeDialog() {
setTestResult(null);
setDialog(BLANK_DIALOG);
}
// -------------------------------------------------------------------------
// Test connection (advisory — never blocks save)
// -------------------------------------------------------------------------
function handleTest() {
setTestResult(null);
let body: AutoscanConnectionTestInput;
if (dialog.editing && dialog.mode === "own") {
// Editing an existing own-credentials connection: test the saved record
// unless the operator typed new credentials in this dialog session.
if (dialog.baseUrl.trim() && dialog.apiKey.trim()) {
body = { base_url: dialog.baseUrl.trim(), api_key_ref: dialog.apiKey.trim() };
} else {
body = { connection_id: dialog.editing.id };
}
} else if (dialog.editing && dialog.mode === "reuse") {
body = { connection_id: dialog.editing.id };
} else if (dialog.mode === "reuse") {
body = { request_integration_id: dialog.requestIntegrationId };
} else {
body = {
base_url: dialog.baseUrl.trim(),
...(dialog.apiKey.trim() ? { api_key_ref: dialog.apiKey.trim() } : {}),
};
}
testConnection.mutate(body, {
onSuccess: (result) => setTestResult(result),
onError: (err) =>
setTestResult({
ok: false,
error: err instanceof Error ? err.message : "Connection test failed",
}),
});
}
const canTest =
dialog.mode === "reuse"
? Boolean(dialog.requestIntegrationId) || Boolean(dialog.editing)
: Boolean(dialog.baseUrl.trim()) || Boolean(dialog.editing);
// -------------------------------------------------------------------------
// Save
// -------------------------------------------------------------------------
function handleSave() {
let body: AutoscanConnectionInput;
if (dialog.mode === "reuse") {
if (!dialog.requestIntegrationId) return;
// Resolve name + kind from the selected integration for display purposes;
// the backend can also derive them from the linked integration.
const linked = arrIntegrations.find((i) => i.id === dialog.requestIntegrationId);
body = {
name: linked?.name ?? dialog.requestIntegrationId,
kind: linked?.kind ?? "sonarr",
request_integration_id: dialog.requestIntegrationId,
};
} else {
if (!dialog.name.trim() || !dialog.baseUrl.trim()) return;
body = {
name: dialog.name.trim(),
kind: dialog.kind,
base_url: dialog.baseUrl.trim(),
// Only send api_key_ref when the operator has typed something;
// on edit with a blank field the backend keeps the stored key.
...(dialog.apiKey.trim() ? { api_key_ref: dialog.apiKey.trim() } : {}),
};
}
if (dialog.editing) {
updateConnection.mutate({ id: dialog.editing.id, body }, { onSuccess: closeDialog });
} else {
createConnection.mutate(body, { onSuccess: closeDialog });
}
}
const isSaving = createConnection.isPending || updateConnection.isPending;
const canSave =
dialog.mode === "reuse"
? Boolean(dialog.requestIntegrationId)
: Boolean(dialog.name.trim()) &&
Boolean(dialog.baseUrl.trim()) &&
(Boolean(dialog.apiKey.trim()) || Boolean(dialog.editing?.has_api_key));
// -------------------------------------------------------------------------
// Render
// -------------------------------------------------------------------------
const list = connections.data ?? [];
return (
<>
<Card>
<CardHeader>
<div className="flex items-center justify-between gap-3">
<CardTitle>Connections</CardTitle>
<Button variant="outline" size="sm" onClick={openAddDialog}>
<Plus />
Add connection
</Button>
</div>
</CardHeader>
<CardContent className="px-0">
{connections.isLoading ? (
<p className="text-muted-foreground px-6 py-4 text-sm">Loading…</p>
) : list.length === 0 ? (
<p className="text-muted-foreground px-6 py-6 text-sm">
No connections yet. A connection holds credentials for a server-based source (e.g.
Sonarr/Radarr). Add one if your source needs to reach a server.
</p>
) : (
<Table>
<TableHeader>
<TableRow>
<TableHead>Name</TableHead>
<TableHead>Kind</TableHead>
<TableHead>Source</TableHead>
<TableHead>URL</TableHead>
<TableHead className="w-0" />
</TableRow>
</TableHeader>
<TableBody>
{list.map((conn) => (
<TableRow key={conn.id}>
<TableCell className="font-medium">{conn.name}</TableCell>
<TableCell>{connectionKindLabel(conn.kind)}</TableCell>
<TableCell>
{conn.request_integration_id ? (
<Badge variant="secondary">Reused from Requests</Badge>
) : (
<Badge variant="outline">Own</Badge>
)}
</TableCell>
<TableCell className="text-muted-foreground max-w-[240px] truncate">
{conn.base_url ?? "—"}
</TableCell>
<TableCell>
<div className="flex items-center gap-1">
<Button
variant="ghost"
size="icon-sm"
aria-label="Edit connection"
onClick={() => openEditDialog(conn)}
>
<Pencil />
</Button>
<Button
variant="ghost"
size="icon-sm"
aria-label="Delete connection"
onClick={() => setDeleteTarget(conn)}
>
<Trash2 className="text-destructive" />
</Button>
</div>
</TableCell>
</TableRow>
))}
</TableBody>
</Table>
)}
</CardContent>
</Card>
{/* Add / Edit Dialog */}
<Dialog open={dialog.open} onOpenChange={(open) => !open && closeDialog()}>
<DialogContent>
<DialogHeader>
<DialogTitle>{dialog.editing ? "Edit connection" : "Add connection"}</DialogTitle>
<DialogDescription>
Reuse a server you already configured in Requests (Sonarr/Radarr), or enter your own
credentials.
</DialogDescription>
</DialogHeader>
<div className="space-y-4">
{/* Mode picker (only when creating) */}
{!dialog.editing && (
<div className="space-y-1.5">
<Label>Connection type</Label>
<Select
value={dialog.mode}
onValueChange={(v) => setDialog((d) => ({ ...d, mode: v as DialogMode }))}
>
<SelectTrigger className="w-full">
<SelectValue />
</SelectTrigger>
<SelectContent>
<SelectItem value="own">Enter own credentials</SelectItem>
<SelectItem value="reuse">Reuse from Requests</SelectItem>
</SelectContent>
</Select>
</div>
)}
{/* Reuse mode */}
{dialog.mode === "reuse" && (
<div className="space-y-1.5">
<Label>Requests integration</Label>
{requestIntegrations.isLoading ? (
<p className="text-muted-foreground text-sm">Loading integrations…</p>
) : arrIntegrations.length === 0 ? (
<p className="text-muted-foreground text-sm">
No Sonarr/Radarr integrations found. Add one in the Requests page first.
</p>
) : (
<Select
value={dialog.requestIntegrationId}
onValueChange={(v) => {
setTestResult(null);
setDialog((d) => ({ ...d, requestIntegrationId: v }));
}}
>
<SelectTrigger className="w-full">
<SelectValue placeholder="Select an integration…" />
</SelectTrigger>
<SelectContent>
{arrIntegrations.map((integration) => (
<SelectItem key={integration.id} value={integration.id}>
{integration.name} ({connectionKindLabel(integration.kind)})
</SelectItem>
))}
</SelectContent>
</Select>
)}
</div>
)}
{/* Own credentials mode */}
{dialog.mode === "own" && (
<div className="space-y-4">
<div className="space-y-1.5">
<Label htmlFor="conn-name">Name</Label>
<Input
id="conn-name"
placeholder="My Sonarr"
value={dialog.name}
onChange={(e) => setDialog((d) => ({ ...d, name: e.target.value }))}
/>
</div>
<div className="space-y-1.5">
<Label htmlFor="conn-kind">Kind</Label>
<Select
value={dialog.kind}
onValueChange={(v) => setDialog((d) => ({ ...d, kind: v }))}
>
<SelectTrigger id="conn-kind" className="w-full">
<SelectValue />
</SelectTrigger>
<SelectContent>
<SelectItem value="sonarr">Sonarr</SelectItem>
<SelectItem value="radarr">Radarr</SelectItem>
</SelectContent>
</Select>
</div>
<div className="space-y-1.5">
<Label htmlFor="conn-url">Base URL</Label>
<Input
id="conn-url"
placeholder="http://localhost:8989"
value={dialog.baseUrl}
onChange={(e) => {
setTestResult(null);
setDialog((d) => ({ ...d, baseUrl: e.target.value }));
}}
/>
</div>
<div className="space-y-1.5">
<Label htmlFor="conn-key">
API key
{dialog.editing?.has_api_key && (
<span className="text-muted-foreground ml-1 font-normal">
(leave blank to keep existing)
</span>
)}
</Label>
<Input
id="conn-key"
type="password"
placeholder={dialog.editing?.has_api_key ? "••••••••" : "Enter API key"}
value={dialog.apiKey}
onChange={(e) => {
setTestResult(null);
setDialog((d) => ({ ...d, apiKey: e.target.value }));
}}
autoComplete="new-password"
/>
</div>
</div>
)}
</div>
{/* Inline test result (advisory) */}
{testResult && (
<div
className={`flex items-start gap-2 rounded-md border p-2.5 text-sm ${
testResult.ok
? "border-green-500/30 bg-green-500/10"
: "border-destructive/30 bg-destructive/10"
}`}
role="status"
>
{testResult.ok ? (
<CheckCircle2 className="mt-0.5 size-4 shrink-0 text-green-500" />
) : (
<XCircle className="text-destructive mt-0.5 size-4 shrink-0" />
)}
<span
className={
testResult.ok ? "text-green-600 dark:text-green-400" : "text-destructive"
}
>
{testResult.ok
? `Connected${testResult.version ? ` (v${testResult.version})` : ""}`
: (testResult.error ?? "Connection failed")}
</span>
</div>
)}
<DialogFooter className="sm:justify-between">
<Button
type="button"
variant="ghost"
onClick={handleTest}
disabled={!canTest || testConnection.isPending}
>
{testConnection.isPending ? "Testing…" : "Test connection"}
</Button>
<div className="flex gap-2">
<Button variant="outline" onClick={closeDialog} disabled={isSaving}>
Cancel
</Button>
<Button onClick={handleSave} disabled={!canSave || isSaving}>
{isSaving ? "Saving…" : dialog.editing ? "Save changes" : "Add connection"}
</Button>
</div>
</DialogFooter>
</DialogContent>
</Dialog>
{/* Delete confirmation */}
<AlertDialog
open={deleteTarget !== null}
onOpenChange={(open) => !open && setDeleteTarget(null)}
>
<AlertDialogContent>
<AlertDialogHeader>
<AlertDialogTitle>Delete connection?</AlertDialogTitle>
<AlertDialogDescription>
&ldquo;{deleteTarget?.name}&rdquo; will be removed. Any scan sources bound to it will
lose their connection and will need to be reconfigured.
</AlertDialogDescription>
</AlertDialogHeader>
<AlertDialogFooter>
<AlertDialogCancel>Cancel</AlertDialogCancel>
<AlertDialogAction
variant="destructive"
onClick={() => {
if (deleteTarget) {
deleteConnection.mutate(deleteTarget.id);
setDeleteTarget(null);
}
}}
>
Delete
</AlertDialogAction>
</AlertDialogFooter>
</AlertDialogContent>
</AlertDialog>
</>
);
}
web/src/pages/admin/autoscan/SourcesPanel.tsx
+1426
View File
File diff suppressed because it is too large Load Diff