Files
silo-server/internal/plugins/auto_update.go
T
91e1164090 feat(metadata): local NFO metadata and sidecar artwork (builtin chain provider) (#390)
* feat(metadata): register builtin NFO provider and broaden parsing

Phases A and B of the #216 local-NFO work, implemented test-first.

Registration & hint-first identity (Phase A):
- Migration seeds a reserved kind='builtin' silo.builtin installation
  and an 'nfo' metadata capability (default_enabled=false, priority 1
  for movie/series) with a partial unique index and documented Down.
- In-process builtin provider registry (internal/metadata/builtin.go);
  buildProviders returns the registered provider for builtin rows.
- Guard rails keep the reserved row out of every plugin surface (user
  plugin-settings, installations list, image resolvers, preload,
  auto-update, store Delete, mutation handlers -> 409); silo.builtin is
  a reserved manifest id.
- Startup sync materializes legacy content_level='' chains per level,
  then appends builtin capabilities disabled via
  AppendProviderToAllChains (idempotent); resolveEnabledProvidersBy
  priority now respects default_enabled=false.
- NFO uniqueids seed the trusted-hint machinery via IdentityHintProvider
  with per-mode conflict policy (stored IDs win on scheduled refresh,
  NFO wins on manual refresh, Identify skips NFO); ID-less candidates
  are excluded from provider-priority tie-breaks and nfo never counts
  as corroboration.
- Web chain-editor empty-state gate is now server-derived so builtin
  providers are reachable on plugin-less servers.

Parser breadth & sidecar hardening (Phase B):
- Parser covers the practical Kodi/Jellyfin field set for <movie> and
  <tvshow>: original title, tagline, runtime, dates, content rating,
  genres/studios/countries/tags, multi-source ratings with scale
  normalization, cast with roles/order, director/credits. Empty
  collections stay nil so merge early-returns apply.
- findNFO parses candidates and falls through on read/parse failure or
  root-type mismatch, so a stray movie.nfo cannot shadow tvshow.nfo;
  GetMetadata gains the same ContentType guard Search has.
- New FieldReleaseDates lock gates Year/ReleaseDate/First+LastAirDate
  in merge (Go) and the edit-metadata dialog (web), closing the gap
  where a manual refresh re-applied NFO dates over admin corrections.
- Merge-contract tests pin NFO fill semantics, genres whole-list
  first-provider-wins, and NFO edits propagating on manual refresh only.
- Docs: new admin wiki page (supported fields, merge semantics,
  naming-supplies-structure contract), index bullet, sidecar wording
  revision, v1-scope feature-detection note.

Zero behavior change while the provider is disabled (default); pinned
by CI-mode and DB-gated test suites.

Part of #216

AI-use disclosure: implemented with Claude Code (Fable 5) via
spec-driven TDD and agent-assisted implementation.

* feat(metadata): ingest local sidecar artwork and read series-depth NFO

Phases C and D of the #216 local-NFO work, implemented test-first, plus
the mixed-library use-case pins. Together these deliver the headline
case: a series absent from every remote database (e.g. a fitness
library) scans into a fully presented show -> named seasons -> titled
episodes tree from NFO files and sidecar art alone.

Local sidecar artwork through the S3 image cache (Phase C):
- The NFO provider implements ImageProvider: poster/backdrop/logo
  sidecar discovery with a fixed precedence map, symlink/non-regular
  rejection, an 8 MiB cap, and file:// source URLs at rating 0. Generic
  filenames apply only via the sidecar search paths, so a shared
  folder.jpg in a flat multi-movie directory applies to none.
- file:// becomes a live local source scheme: routed into *_source_path
  (never *_path), accepted by every image enqueue gate, attributed as
  provider "local", excluded from cached-path detection.
- The image-cache processor caches local files with lexical-on-logical
  confinement to the library roots, open-handle reads with re-checks,
  the same variant widths as remote art, and stable (7-day) failure
  classification. Keys land under
  local/{contentType}/{contentID}/{hash8}/{imageType}; superseded
  prefixes are cleaned on re-cache and item deletion.
- applyIfBetter gains a local exemption so rating-0 local art can fill
  matched items without being stickily displaced; ImageRequest carries
  additive sidecar path context.

Series depth (Phase D):
- SeasonsRequest/EpisodesRequest carry additive local path context
  (series roots, per-season directories, per-episode file paths),
  derived from naming at match time and reconstructed on refresh.
- season.nfo supplies season name/plot; NFO season numbers are advisory
  (directory-derived number wins with a Warn - naming owns structure).
  <episodedetails> gains aired/runtime/ratings; <basename>.nfo titles
  episodes and <basename>-thumb.ext supplies thumbs; filename SxxEyy
  wins over NFO numbers.
- Episode NFOs work without a season.nfo (provider seasons unioned with
  on-disk seasons); SynthesizeFallbackEpisodes always runs after persist
  so NFO-less episodes keep synthesized rows. Season/episode file:// art
  rides the Phase C pipeline unchanged.
- Migration adds season:1/episode:1 to the builtin NFO capability's
  default_priority (still default_enabled=false).

Mixed sports-library use case (tests only, no product change):
- Pins the classification contract for one library holding movie-shaped
  and show-shaped content (WWE PPV events as movies next to a "WWE
  SmackDown" show, NASCAR/F1/FIFA with partial TVDB/TMDB data): naming
  decides movie-vs-series per file before any provider runs; the NFO
  supplies metadata/identity but never flips type (ContentType guard);
  the per-root Type override is the correction path.
- NFO-driven type classification at scan time is recorded as an explicit
  deferred open question.

Part of #216

AI-use disclosure: implemented with Claude Code (Fable 5) via
spec-driven TDD and agent-assisted implementation.

* docs(metadata): document local NFO metadata architecture

Add a single as-built architecture page
(docs/architecture/local-nfo-metadata.md) for the #216 local-NFO
feature: the builtin registration model, hint-first identity semantics,
the file:// -> S3 artwork pipeline and its deployment constraint, series
depth, the mixed-library classification contract, and known limitations.

This replaces the working implementation plan, the per-phase specs, and
the narrow sidecar-artwork note, which were planning drafts and are left
untracked; admin-facing behavior remains in the wiki.

Part of #216

AI-use disclosure: planned, drafted, and consolidated with Claude Code
(Fable 5) using multi-agent exploration and adversarial review.

* fix(metadata): address PR review findings on NFO builtin provider

Fold in the valid, low-risk fixes surfaced by automated review on #390:

- imagecache: extract validateCacheRequest so CacheBytes (the local
  sidecar season/episode path) enforces the same episode-requires-season
  guard as Cache, preventing distinct episodes' art from colliding under
  one S3 key.
- image_cache_processor: close the sidecar symlink-swap window by
  rejecting the opened handle unless os.SameFile matches the Lstat'd
  file, so a leaf swapped to a symlink can't pull an out-of-root target
  into the public cache.
- plugins: guard the reserved builtin installation row in the store's
  Update, matching Delete, so its version/enabled/capabilities can never
  be rewritten even if a mutation slips past the HTTP layer.
- cmd/silo: bound SyncBuiltinProviderChains with a 30s timeout so a stuck
  DB round-trip fails fast at startup instead of hanging.
- metadata: panic instead of silently no-op'ing on an invalid
  RegisterBuiltinProvider call (init-time programmer error).
- docs: correct the media-folder-and-naming NFO paragraph to state
  season/episode NFOs and sidecar artwork are actively read.

---------

Co-authored-by: Quick104 <31828688+Quick104@users.noreply.github.com>
2026-07-16 17:55:36 -04:00

504 lines
15 KiB
Go

package plugins
import (
"context"
"errors"
"fmt"
"log/slog"
"slices"
"strconv"
"strings"
"github.com/Silo-Server/silo-server/internal/pluginhost"
)
// compareVersions compares two dot-separated version strings numerically.
// Returns -1 if a < b, 0 if a == b, 1 if a > b.
// Non-numeric segments fall back to lexicographic comparison.
func compareVersions(a, b string) int {
partsA := strings.Split(a, ".")
partsB := strings.Split(b, ".")
maxLen := len(partsA)
if len(partsB) > maxLen {
maxLen = len(partsB)
}
for i := 0; i < maxLen; i++ {
var segA, segB string
if i < len(partsA) {
segA = partsA[i]
}
if i < len(partsB) {
segB = partsB[i]
}
numA, errA := strconv.Atoi(segA)
numB, errB := strconv.Atoi(segB)
if errA == nil && errB == nil {
if numA < numB {
return -1
}
if numA > numB {
return 1
}
} else {
if segA < segB {
return -1
}
if segA > segB {
return 1
}
}
}
return 0
}
var defaultPluginIDs = []string{"silo.tmdb", "silo.tvdb"}
type autoUpdateRepositoryStore interface {
List(ctx context.Context) ([]*Repository, error)
Create(ctx context.Context, input CreateRepositoryInput) (*Repository, error)
}
type managedRepositoryReconciler interface {
ReconcileManaged(ctx context.Context) (ManagedRepositoryReconcileResult, error)
}
type autoUpdateInstallationStore interface {
List(ctx context.Context) ([]*Installation, error)
Update(ctx context.Context, id int, input UpdateInstallationInput) error
Delete(ctx context.Context, id int) error
}
type autoUpdateCatalog interface {
Fetch(ctx context.Context) ([]CatalogEntry, error)
ResolveInstall(ctx context.Context, req InstallCatalogRequest) (*ResolvedCatalogInstall, error)
}
type autoUpdateInstaller interface {
InstallRemote(ctx context.Context, req InstallArchiveRequest) (*InstallResult, error)
InstallBinary(ctx context.Context, req InstallBinaryRequest) (*InstallResult, error)
ReplaceRemote(ctx context.Context, existing *Installation, req InstallArchiveRequest) (*InstallResult, error)
ReplaceBinary(ctx context.Context, existing *Installation, req InstallBinaryRequest) (*InstallResult, error)
}
type autoUpdateHost interface {
Stop(installationID int) error
}
type AutoUpdateOptions struct {
SeedDefaultRepository bool
AutoInstallDefaults bool
}
type AutoUpdateSummary struct {
RepositoriesSeeded int `json:"repositories_seeded"`
CatalogEntries int `json:"catalog_entries"`
InstalledPlugins int `json:"installed_plugins"`
DefaultPluginsInstalled int `json:"default_plugins_installed"`
UpdatesApplied int `json:"updates_applied"`
UpdatesAvailable int `json:"updates_available"`
FailedOperations int `json:"failed_operations"`
Failures []string `json:"failures,omitempty"`
}
// AutoUpdateService reconciles managed plugin repositories, auto-installs
// default plugins, and auto-updates installed plugins at server startup.
type AutoUpdateService struct {
repositories autoUpdateRepositoryStore
installations autoUpdateInstallationStore
catalog autoUpdateCatalog
installer autoUpdateInstaller
host autoUpdateHost
logger *slog.Logger
// onChange is fired after a run mutates any plugin_installations row so
// that peers sharing the same store (notably plugins.Service and its
// installation cache) can invalidate their memoized state. It is optional:
// when nil, no notification is sent. Pass plugins.Service.OnLifecycleChange
// here to keep that service's installation cache consistent with the
// version-specific InstallPath/Version this service writes.
onChange func(context.Context)
}
// NewAutoUpdateService creates a new AutoUpdateService. onChange is optional and
// nil-safe: when non-nil it is invoked once after any run that mutates an
// installation row (auto-update applied, default plugin installed, or an
// available version recorded) so peers can invalidate cached installation state.
func NewAutoUpdateService(
repositories autoUpdateRepositoryStore,
installations autoUpdateInstallationStore,
catalog autoUpdateCatalog,
installer autoUpdateInstaller,
host autoUpdateHost,
logger *slog.Logger,
onChange func(context.Context),
) *AutoUpdateService {
if logger == nil {
logger = slog.Default()
}
return &AutoUpdateService{
repositories: repositories,
installations: installations,
catalog: catalog,
installer: installer,
host: host,
logger: logger,
onChange: onChange,
}
}
// Check runs a plugin update pass. It can be used by startup, scheduled tasks,
// and manual admin actions.
func (s *AutoUpdateService) Check(ctx context.Context, opts AutoUpdateOptions) (AutoUpdateSummary, error) {
var summary AutoUpdateSummary
if opts.SeedDefaultRepository {
seeded, err := s.ensureManagedRepositoryRows(ctx)
if err != nil {
return summary, err
}
summary.RepositoriesSeeded += seeded
}
entries, err := s.catalog.Fetch(ctx)
if err != nil {
return summary, err
}
summary.CatalogEntries = len(entries)
installed, err := s.installations.List(ctx)
if err != nil {
return summary, err
}
summary.InstalledPlugins = len(installed)
installedPluginIDs := make(map[string]struct{}, len(installed))
for _, inst := range installed {
if inst == nil {
continue
}
installedPluginIDs[inst.PluginID] = struct{}{}
}
latestByRepositoryPlugin := latestCatalogEntriesByRepository(entries)
for _, existing := range installed {
if existing == nil || existing.RepositoryID == nil {
continue
}
// The reserved builtin row must never be matched against catalog
// entries: a catalog plugin named after it could otherwise rewrite its
// version/install_path and convert it into a launchable plugin.
// update_policy='manual' on the row and the reserved-plugin-id install
// rejection are the other layers.
if existing.IsBuiltin() {
continue
}
entry, ok := latestByRepositoryPlugin[repositoryPluginKey{
RepositoryID: *existing.RepositoryID,
PluginID: existing.PluginID,
}]
if !ok {
continue
}
outcome, err := s.handleExistingPlugin(ctx, existing, entry)
if err != nil {
summary.recordFailure("process plugin update %s: %v", existing.PluginID, err)
continue
}
switch outcome {
case autoUpdateOutcomeUpdated:
summary.UpdatesApplied++
case autoUpdateOutcomeNotified:
summary.UpdatesAvailable++
}
}
if opts.AutoInstallDefaults {
latestOfficial := latestCatalogEntriesForSource(entries, RepositorySourceSilo)
for _, pluginID := range defaultPluginIDs {
if _, installed := installedPluginIDs[pluginID]; installed {
continue
}
entry, ok := latestOfficial[pluginID]
if !ok {
continue
}
installedDefault, err := s.handleNewPlugin(ctx, pluginID, entry)
if err != nil {
summary.recordFailure("auto-install default plugin %s: %v", pluginID, err)
} else if installedDefault {
summary.DefaultPluginsInstalled++
}
}
}
// Any of these outcomes wrote to a plugin_installations row: installing a
// default plugin creates one, an applied auto-update rewrites the version-
// specific InstallPath/Version (and deletes the old install dir), and a
// notify records available_version. Fire onChange once per run so peers such
// as plugins.Service invalidate their installation cache; otherwise stale
// rows (old InstallPath/Version) would make later plugin RPCs fail against a
// re-extracted, newer archive.
if summary.DefaultPluginsInstalled > 0 || summary.UpdatesApplied > 0 || summary.UpdatesAvailable > 0 {
s.notifyChanged(ctx)
}
return summary, nil
}
// notifyChanged fires the optional onChange hook. It is nil-safe and
// best-effort: OnLifecycleChange already recovers hook panics internally, so a
// direct call cannot fail the update pass.
func (s *AutoUpdateService) notifyChanged(ctx context.Context) {
if s.onChange == nil {
return
}
s.onChange(ctx)
}
// Run reconciles managed repositories, fetches the catalog, auto-installs
// default plugins that are not yet installed, and processes updates for
// installed plugins according to their update policy. All errors are logged
// rather than returned so that startup is never blocked.
func (s *AutoUpdateService) Run(ctx context.Context) error {
summary, err := s.Check(ctx, AutoUpdateOptions{
SeedDefaultRepository: true,
AutoInstallDefaults: true,
})
if err != nil {
s.logger.WarnContext(ctx, "failed to run plugin auto-update", "error", err)
return nil
}
for _, failure := range summary.Failures {
s.logger.WarnContext(ctx, "plugin auto-update operation failed", "error", failure)
}
return nil
}
// ensureManagedRepositoryRows reconciles built-in repositories. The fallback
// preserves the legacy fake-store contract used by isolated unit tests.
func (s *AutoUpdateService) ensureManagedRepositoryRows(ctx context.Context) (int, error) {
if reconciler, ok := s.repositories.(managedRepositoryReconciler); ok {
result, err := reconciler.ReconcileManaged(ctx)
if err != nil {
return 0, err
}
if result.RepositoriesCreated > 0 {
s.logger.InfoContext(ctx, "reconciled managed plugin repositories",
"repositories_created", result.RepositoriesCreated,
)
}
return result.RepositoriesCreated, nil
}
repos, err := s.repositories.List(ctx)
if err != nil {
return 0, err
}
if len(repos) > 0 {
return 0, nil
}
enabled := true
_, err = s.repositories.Create(ctx, CreateRepositoryInput{
URL: DefaultRepositoryURL,
DisplayName: DefaultRepositoryName,
Enabled: &enabled,
})
if err != nil {
return 0, err
}
s.logger.InfoContext(ctx, "seeded default plugin repository",
"url", DefaultRepositoryURL,
"name", DefaultRepositoryName,
)
return 1, nil
}
// handleNewPlugin auto-installs a plugin if it is in the default plugin list.
func (s *AutoUpdateService) handleNewPlugin(ctx context.Context, pluginID string, entry CatalogEntry) (bool, error) {
if !slices.Contains(defaultPluginIDs, pluginID) {
return false, nil
}
version := entry.Manifest.GetVersion()
s.logger.InfoContext(ctx, "auto-installing default plugin",
"plugin_id", pluginID,
"version", version,
)
repoID := entry.RepositoryID
target, err := s.catalog.ResolveInstall(ctx, InstallCatalogRequest{
RepositoryID: repoID,
PluginID: pluginID,
Version: version,
})
if err == nil {
_, err = s.installResolvedCatalogTarget(ctx, target)
}
if err != nil {
return false, err
}
return true, nil
}
// handleExistingPlugin checks for version updates and applies the installation's
// update policy.
func (s *AutoUpdateService) handleExistingPlugin(ctx context.Context, existing *Installation, entry CatalogEntry) (autoUpdateOutcome, error) {
catalogVersion := entry.Manifest.GetVersion()
if compareVersions(catalogVersion, existing.Version) <= 0 {
return autoUpdateOutcomeNone, nil
}
switch existing.UpdatePolicy {
case "auto":
return autoUpdateOutcomeUpdated, s.autoUpdatePlugin(ctx, existing, entry)
case "notify":
return autoUpdateOutcomeNotified, s.notifyPluginUpdate(ctx, existing, entry)
default:
// "off" or any unrecognized policy: do nothing.
return autoUpdateOutcomeNone, nil
}
}
// autoUpdatePlugin stops the running plugin and replaces it in-place so the
// installation ID and dependent configuration rows remain stable.
func (s *AutoUpdateService) autoUpdatePlugin(ctx context.Context, existing *Installation, entry CatalogEntry) error {
pluginID := existing.PluginID
oldVersion := existing.Version
newVersion := entry.Manifest.GetVersion()
// Stop the running plugin if a host is available.
if s.host != nil {
if err := s.host.Stop(existing.ID); err != nil && !errors.Is(err, pluginhost.ErrClientNotFound) {
return fmt.Errorf("stop plugin %s: %w", pluginID, err)
}
}
// Install the new version.
target, err := s.catalog.ResolveInstall(ctx, InstallCatalogRequest{
RepositoryID: entry.RepositoryID,
PluginID: pluginID,
Version: newVersion,
})
if err == nil {
repositoryID := target.RepositoryID
if target.LegacyArchive {
_, err = s.installer.ReplaceRemote(ctx, existing, InstallArchiveRequest{
ArchiveURL: target.ArchiveURL,
RepositoryID: &repositoryID,
})
} else {
_, err = s.installer.ReplaceBinary(ctx, existing, InstallBinaryRequest{
BinaryURL: target.ArchiveURL,
Checksum: target.Checksum,
RepositoryID: &repositoryID,
})
}
}
if err != nil {
return fmt.Errorf("install updated plugin %s from %s to %s: %w", pluginID, oldVersion, newVersion, err)
}
s.logger.InfoContext(ctx, "auto-updated plugin",
"plugin_id", pluginID,
"old_version", oldVersion,
"new_version", newVersion,
)
return nil
}
// notifyPluginUpdate records the available version on the installation so the
// user can be informed through the UI.
func (s *AutoUpdateService) notifyPluginUpdate(ctx context.Context, existing *Installation, entry CatalogEntry) error {
newVersion := entry.Manifest.GetVersion()
if err := s.installations.Update(ctx, existing.ID, UpdateInstallationInput{
AvailableVersion: &newVersion,
}); err != nil {
return fmt.Errorf("record available version for plugin %s: %w", existing.PluginID, err)
}
s.logger.InfoContext(ctx, "update available for plugin",
"plugin_id", existing.PluginID,
"installed_version", existing.Version,
"available_version", newVersion,
)
return nil
}
type repositoryPluginKey struct {
RepositoryID int
PluginID string
}
func latestCatalogEntriesByRepository(entries []CatalogEntry) map[repositoryPluginKey]CatalogEntry {
latest := make(map[repositoryPluginKey]CatalogEntry, len(entries))
for _, entry := range entries {
if entry.Manifest == nil {
continue
}
pluginID := entry.Manifest.GetPluginId()
key := repositoryPluginKey{RepositoryID: entry.RepositoryID, PluginID: pluginID}
if existing, ok := latest[key]; ok {
if compareVersions(entry.Manifest.GetVersion(), existing.Manifest.GetVersion()) <= 0 {
continue
}
}
latest[key] = entry
}
return latest
}
func latestCatalogEntriesForSource(entries []CatalogEntry, sourceKind string) map[string]CatalogEntry {
latest := make(map[string]CatalogEntry, len(entries))
for _, entry := range entries {
if entry.Manifest == nil || entry.SourceKind != sourceKind {
continue
}
pluginID := entry.Manifest.GetPluginId()
if existing, ok := latest[pluginID]; ok && compareVersions(entry.Manifest.GetVersion(), existing.Manifest.GetVersion()) <= 0 {
continue
}
latest[pluginID] = entry
}
return latest
}
func (s *AutoUpdateService) installResolvedCatalogTarget(ctx context.Context, target *ResolvedCatalogInstall) (*InstallResult, error) {
if target == nil {
return nil, fmt.Errorf("catalog install target is required")
}
repositoryID := target.RepositoryID
if target.LegacyArchive {
return s.installer.InstallRemote(ctx, InstallArchiveRequest{
ArchiveURL: target.ArchiveURL,
RepositoryID: &repositoryID,
})
}
return s.installer.InstallBinary(ctx, InstallBinaryRequest{
BinaryURL: target.ArchiveURL,
Checksum: target.Checksum,
RepositoryID: &repositoryID,
})
}
type autoUpdateOutcome int
const (
autoUpdateOutcomeNone autoUpdateOutcome = iota
autoUpdateOutcomeUpdated
autoUpdateOutcomeNotified
)
func (s *AutoUpdateSummary) recordFailure(format string, args ...any) {
s.FailedOperations++
s.Failures = append(s.Failures, fmt.Sprintf(format, args...))
}