* feat(metadata): register builtin NFO provider and broaden parsing Phases A and B of the #216 local-NFO work, implemented test-first. Registration & hint-first identity (Phase A): - Migration seeds a reserved kind='builtin' silo.builtin installation and an 'nfo' metadata capability (default_enabled=false, priority 1 for movie/series) with a partial unique index and documented Down. - In-process builtin provider registry (internal/metadata/builtin.go); buildProviders returns the registered provider for builtin rows. - Guard rails keep the reserved row out of every plugin surface (user plugin-settings, installations list, image resolvers, preload, auto-update, store Delete, mutation handlers -> 409); silo.builtin is a reserved manifest id. - Startup sync materializes legacy content_level='' chains per level, then appends builtin capabilities disabled via AppendProviderToAllChains (idempotent); resolveEnabledProvidersBy priority now respects default_enabled=false. - NFO uniqueids seed the trusted-hint machinery via IdentityHintProvider with per-mode conflict policy (stored IDs win on scheduled refresh, NFO wins on manual refresh, Identify skips NFO); ID-less candidates are excluded from provider-priority tie-breaks and nfo never counts as corroboration. - Web chain-editor empty-state gate is now server-derived so builtin providers are reachable on plugin-less servers. Parser breadth & sidecar hardening (Phase B): - Parser covers the practical Kodi/Jellyfin field set for <movie> and <tvshow>: original title, tagline, runtime, dates, content rating, genres/studios/countries/tags, multi-source ratings with scale normalization, cast with roles/order, director/credits. Empty collections stay nil so merge early-returns apply. - findNFO parses candidates and falls through on read/parse failure or root-type mismatch, so a stray movie.nfo cannot shadow tvshow.nfo; GetMetadata gains the same ContentType guard Search has. - New FieldReleaseDates lock gates Year/ReleaseDate/First+LastAirDate in merge (Go) and the edit-metadata dialog (web), closing the gap where a manual refresh re-applied NFO dates over admin corrections. - Merge-contract tests pin NFO fill semantics, genres whole-list first-provider-wins, and NFO edits propagating on manual refresh only. - Docs: new admin wiki page (supported fields, merge semantics, naming-supplies-structure contract), index bullet, sidecar wording revision, v1-scope feature-detection note. Zero behavior change while the provider is disabled (default); pinned by CI-mode and DB-gated test suites. Part of #216 AI-use disclosure: implemented with Claude Code (Fable 5) via spec-driven TDD and agent-assisted implementation. * feat(metadata): ingest local sidecar artwork and read series-depth NFO Phases C and D of the #216 local-NFO work, implemented test-first, plus the mixed-library use-case pins. Together these deliver the headline case: a series absent from every remote database (e.g. a fitness library) scans into a fully presented show -> named seasons -> titled episodes tree from NFO files and sidecar art alone. Local sidecar artwork through the S3 image cache (Phase C): - The NFO provider implements ImageProvider: poster/backdrop/logo sidecar discovery with a fixed precedence map, symlink/non-regular rejection, an 8 MiB cap, and file:// source URLs at rating 0. Generic filenames apply only via the sidecar search paths, so a shared folder.jpg in a flat multi-movie directory applies to none. - file:// becomes a live local source scheme: routed into *_source_path (never *_path), accepted by every image enqueue gate, attributed as provider "local", excluded from cached-path detection. - The image-cache processor caches local files with lexical-on-logical confinement to the library roots, open-handle reads with re-checks, the same variant widths as remote art, and stable (7-day) failure classification. Keys land under local/{contentType}/{contentID}/{hash8}/{imageType}; superseded prefixes are cleaned on re-cache and item deletion. - applyIfBetter gains a local exemption so rating-0 local art can fill matched items without being stickily displaced; ImageRequest carries additive sidecar path context. Series depth (Phase D): - SeasonsRequest/EpisodesRequest carry additive local path context (series roots, per-season directories, per-episode file paths), derived from naming at match time and reconstructed on refresh. - season.nfo supplies season name/plot; NFO season numbers are advisory (directory-derived number wins with a Warn - naming owns structure). <episodedetails> gains aired/runtime/ratings; <basename>.nfo titles episodes and <basename>-thumb.ext supplies thumbs; filename SxxEyy wins over NFO numbers. - Episode NFOs work without a season.nfo (provider seasons unioned with on-disk seasons); SynthesizeFallbackEpisodes always runs after persist so NFO-less episodes keep synthesized rows. Season/episode file:// art rides the Phase C pipeline unchanged. - Migration adds season:1/episode:1 to the builtin NFO capability's default_priority (still default_enabled=false). Mixed sports-library use case (tests only, no product change): - Pins the classification contract for one library holding movie-shaped and show-shaped content (WWE PPV events as movies next to a "WWE SmackDown" show, NASCAR/F1/FIFA with partial TVDB/TMDB data): naming decides movie-vs-series per file before any provider runs; the NFO supplies metadata/identity but never flips type (ContentType guard); the per-root Type override is the correction path. - NFO-driven type classification at scan time is recorded as an explicit deferred open question. Part of #216 AI-use disclosure: implemented with Claude Code (Fable 5) via spec-driven TDD and agent-assisted implementation. * docs(metadata): document local NFO metadata architecture Add a single as-built architecture page (docs/architecture/local-nfo-metadata.md) for the #216 local-NFO feature: the builtin registration model, hint-first identity semantics, the file:// -> S3 artwork pipeline and its deployment constraint, series depth, the mixed-library classification contract, and known limitations. This replaces the working implementation plan, the per-phase specs, and the narrow sidecar-artwork note, which were planning drafts and are left untracked; admin-facing behavior remains in the wiki. Part of #216 AI-use disclosure: planned, drafted, and consolidated with Claude Code (Fable 5) using multi-agent exploration and adversarial review. * fix(metadata): address PR review findings on NFO builtin provider Fold in the valid, low-risk fixes surfaced by automated review on #390: - imagecache: extract validateCacheRequest so CacheBytes (the local sidecar season/episode path) enforces the same episode-requires-season guard as Cache, preventing distinct episodes' art from colliding under one S3 key. - image_cache_processor: close the sidecar symlink-swap window by rejecting the opened handle unless os.SameFile matches the Lstat'd file, so a leaf swapped to a symlink can't pull an out-of-root target into the public cache. - plugins: guard the reserved builtin installation row in the store's Update, matching Delete, so its version/enabled/capabilities can never be rewritten even if a mutation slips past the HTTP layer. - cmd/silo: bound SyncBuiltinProviderChains with a 30s timeout so a stuck DB round-trip fails fast at startup instead of hanging. - metadata: panic instead of silently no-op'ing on an invalid RegisterBuiltinProvider call (init-time programmer error). - docs: correct the media-folder-and-naming NFO paragraph to state season/episode NFOs and sidecar artwork are actively read. --------- Co-authored-by: Quick104 <31828688+Quick104@users.noreply.github.com>
788 lines
28 KiB
Go
788 lines
28 KiB
Go
package metadata
|
|
|
|
import (
|
|
"context"
|
|
"crypto/sha256"
|
|
"encoding/hex"
|
|
"errors"
|
|
"fmt"
|
|
"io"
|
|
"io/fs"
|
|
"log/slog"
|
|
"os"
|
|
"path/filepath"
|
|
"strconv"
|
|
"strings"
|
|
"sync"
|
|
"sync/atomic"
|
|
"time"
|
|
|
|
"github.com/Silo-Server/silo-server/internal/models"
|
|
)
|
|
|
|
// maxLocalImageSourceBytes caps local sidecar artwork reads at 8 MiB,
|
|
// matching the NFO provider's discovery guard.
|
|
const maxLocalImageSourceBytes = 8 << 20
|
|
|
|
// imageCacheDiscoveryInterval throttles the full-catalog backfill sweep so an
|
|
// idle installation does not re-scan every entity table on every task tick.
|
|
// Draining of already-queued jobs is unaffected and stays responsive.
|
|
const imageCacheDiscoveryInterval = 15 * time.Minute
|
|
|
|
type ImageCacheJobClaimer interface {
|
|
ClaimDue(ctx context.Context, workerID string, limit int) ([]*models.MetadataImageCacheJob, error)
|
|
MarkSucceeded(ctx context.Context, id int64, lockedBy string) error
|
|
MarkFailed(ctx context.Context, id int64, attemptCount int, lockedBy string, errText string) error
|
|
RequeueClaimed(ctx context.Context, ids []int64, workerID string) error
|
|
CurrentTargetSourcePath(ctx context.Context, job *models.MetadataImageCacheJob) (string, error)
|
|
EnqueueExistingProviderArtwork(ctx context.Context, limit int) (int, error)
|
|
DeleteSucceededBefore(ctx context.Context, before time.Time, limit int) (int, error)
|
|
}
|
|
|
|
// imageCacheTargetCachedPathReader is optionally implemented by the job store
|
|
// (ImageCacheJobRepository does) to expose the target's currently stored
|
|
// cached path for the local-artwork unchanged-skip and stale-prefix cleanup.
|
|
type imageCacheTargetCachedPathReader interface {
|
|
CurrentTargetCachedPath(ctx context.Context, job *models.MetadataImageCacheJob) (string, error)
|
|
}
|
|
|
|
// LibraryRootResolver reports the media folder root paths a piece of content
|
|
// belongs to (media_folders.Paths via media_item_libraries). The processor
|
|
// confines local file:// sources to these roots before reading them.
|
|
type LibraryRootResolver interface {
|
|
LibraryRootsForContent(ctx context.Context, contentID string) ([]string, error)
|
|
}
|
|
|
|
// ImagePrefixDeleter deletes cached image objects under a key prefix; used to
|
|
// sweep the previous hashed local/ prefix after a successful re-cache.
|
|
type ImagePrefixDeleter interface {
|
|
DeletePrefix(ctx context.Context, bucket, prefix string) (int, error)
|
|
Bucket() string
|
|
}
|
|
|
|
type SeasonArtworkUpdater interface {
|
|
UpdateArtworkIfSourceMatches(ctx context.Context, contentID, sourcePath, cachedPath, thumbhash string) (bool, error)
|
|
}
|
|
|
|
type EpisodeStillUpdater interface {
|
|
UpdateStillIfSourceMatches(ctx context.Context, contentID, sourcePath, cachedPath, thumbhash string) (bool, error)
|
|
}
|
|
|
|
type ItemArtworkUpdater interface {
|
|
UpdateArtworkIfSourceMatches(ctx context.Context, contentID, imageType, sourcePath, cachedPath, thumbhash string) (bool, error)
|
|
}
|
|
|
|
type ItemLocalizationArtworkUpdater interface {
|
|
UpdateArtworkIfSourceMatches(ctx context.Context, contentID, language, imageType, sourcePath, cachedPath, thumbhash string) (bool, error)
|
|
}
|
|
|
|
type SeasonLocalizationArtworkUpdater interface {
|
|
UpdateArtworkIfSourceMatches(ctx context.Context, contentID, language, sourcePath, cachedPath, thumbhash string) (bool, error)
|
|
}
|
|
|
|
type PersonPhotoUpdater interface {
|
|
UpdatePhotoIfSourceMatches(ctx context.Context, personID int64, sourcePath, cachedPath, thumbhash string) (bool, error)
|
|
}
|
|
|
|
type ImageCacheProcessorTargets struct {
|
|
Items ItemArtworkUpdater
|
|
Seasons SeasonArtworkUpdater
|
|
Episodes EpisodeStillUpdater
|
|
ItemLocalizations ItemLocalizationArtworkUpdater
|
|
SeasonLocalizations SeasonLocalizationArtworkUpdater
|
|
People PersonPhotoUpdater
|
|
}
|
|
|
|
type ImageCacheProcessor struct {
|
|
jobs ImageCacheJobClaimer
|
|
cacher ImageCacher
|
|
resolver interface {
|
|
ResolveImageURL(ctx context.Context, path string, variant string) string
|
|
}
|
|
targets ImageCacheProcessorTargets
|
|
logger *slog.Logger
|
|
|
|
// Local file:// artwork support. libraryRoots confines reads to the
|
|
// owning library's roots; prefixDeleter sweeps stale hashed prefixes.
|
|
// The processor host must mount the libraries, like the metadata worker.
|
|
libraryRoots LibraryRootResolver
|
|
prefixDeleter ImagePrefixDeleter
|
|
|
|
enabled atomic.Bool
|
|
|
|
discoveryInterval time.Duration
|
|
discoveryMu sync.Mutex
|
|
lastDiscovery time.Time
|
|
}
|
|
|
|
// SetLibraryRootResolver wires the folder repository used to confine local
|
|
// file:// sources to their owning library's roots. Without it, local jobs
|
|
// fail (and retry on the normal backoff) rather than reading arbitrary paths.
|
|
func (p *ImageCacheProcessor) SetLibraryRootResolver(resolver LibraryRootResolver) {
|
|
if p == nil {
|
|
return
|
|
}
|
|
p.libraryRoots = resolver
|
|
}
|
|
|
|
// SetImagePrefixDeleter wires the object store used to delete the previous
|
|
// hashed local/ prefix after a successful local re-cache. Optional: without
|
|
// it, stale prefixes are left behind (item deletion still sweeps them).
|
|
func (p *ImageCacheProcessor) SetImagePrefixDeleter(deleter ImagePrefixDeleter) {
|
|
if p == nil {
|
|
return
|
|
}
|
|
p.prefixDeleter = deleter
|
|
}
|
|
|
|
// SetEnabled toggles background caching. When disabled the processor performs
|
|
// no discovery, claiming, or uploading, honoring metadata.cache_images so that
|
|
// merely configuring object storage does not download the whole catalog.
|
|
func (p *ImageCacheProcessor) SetEnabled(enabled bool) {
|
|
if p == nil {
|
|
return
|
|
}
|
|
p.enabled.Store(enabled)
|
|
}
|
|
|
|
func NewImageCacheProcessor(
|
|
jobs ImageCacheJobClaimer,
|
|
cacher ImageCacher,
|
|
resolver interface {
|
|
ResolveImageURL(ctx context.Context, path string, variant string) string
|
|
},
|
|
seasons SeasonArtworkUpdater,
|
|
episodes EpisodeStillUpdater,
|
|
) *ImageCacheProcessor {
|
|
return NewImageCacheProcessorWithTargets(jobs, cacher, resolver, ImageCacheProcessorTargets{
|
|
Seasons: seasons,
|
|
Episodes: episodes,
|
|
})
|
|
}
|
|
|
|
func NewImageCacheProcessorWithTargets(
|
|
jobs ImageCacheJobClaimer,
|
|
cacher ImageCacher,
|
|
resolver interface {
|
|
ResolveImageURL(ctx context.Context, path string, variant string) string
|
|
},
|
|
targets ImageCacheProcessorTargets,
|
|
) *ImageCacheProcessor {
|
|
p := &ImageCacheProcessor{
|
|
jobs: jobs,
|
|
cacher: cacher,
|
|
resolver: resolver,
|
|
targets: targets,
|
|
logger: slog.Default(),
|
|
discoveryInterval: imageCacheDiscoveryInterval,
|
|
}
|
|
// Default to enabled; callers gate on metadata.cache_images via SetEnabled.
|
|
p.enabled.Store(true)
|
|
return p
|
|
}
|
|
|
|
type ImageCacheRunStats struct {
|
|
Batches int
|
|
EnqueuedExisting int
|
|
Claimed int
|
|
Succeeded int
|
|
Failed int
|
|
Skipped int
|
|
DeletedSucceeded int
|
|
UploadedVariants int
|
|
ExistingVariants int
|
|
RuntimeLimited bool
|
|
}
|
|
|
|
func (s *ImageCacheRunStats) add(other ImageCacheRunStats) {
|
|
s.EnqueuedExisting += other.EnqueuedExisting
|
|
s.Claimed += other.Claimed
|
|
s.Succeeded += other.Succeeded
|
|
s.Failed += other.Failed
|
|
s.Skipped += other.Skipped
|
|
s.DeletedSucceeded += other.DeletedSucceeded
|
|
s.UploadedVariants += other.UploadedVariants
|
|
s.ExistingVariants += other.ExistingVariants
|
|
}
|
|
|
|
// RunOnce claims and processes one batch of already-queued jobs. It does not
|
|
// run catalog discovery; callers (RunUntilIdle) drive discovery on a throttled
|
|
// cadence so backlog draining stays decoupled from full-table sweeps.
|
|
func (p *ImageCacheProcessor) RunOnce(ctx context.Context, workerID string, claimLimit int, concurrency int) (ImageCacheRunStats, error) {
|
|
var stats ImageCacheRunStats
|
|
if p == nil || p.jobs == nil || p.cacher == nil || !p.enabled.Load() {
|
|
return stats, nil
|
|
}
|
|
if claimLimit <= 0 {
|
|
claimLimit = 100
|
|
}
|
|
if concurrency <= 0 {
|
|
concurrency = 4
|
|
}
|
|
|
|
jobs, err := p.jobs.ClaimDue(ctx, workerID, claimLimit)
|
|
if err != nil {
|
|
return stats, err
|
|
}
|
|
stats.Claimed = len(jobs)
|
|
if len(jobs) == 0 {
|
|
p.cleanupSucceeded(ctx, &stats)
|
|
return stats, nil
|
|
}
|
|
|
|
sem := make(chan struct{}, concurrency)
|
|
var wg sync.WaitGroup
|
|
var mu sync.Mutex
|
|
var unstarted []int64
|
|
loop:
|
|
for i, job := range jobs {
|
|
// Acquire the semaphore before spawning so cancellation is observed here
|
|
// rather than inside a goroutine that already holds a claimed job. Jobs we
|
|
// never start are requeued below instead of being left locked until the
|
|
// lease expires.
|
|
select {
|
|
case sem <- struct{}{}:
|
|
case <-ctx.Done():
|
|
for _, rem := range jobs[i:] {
|
|
unstarted = append(unstarted, rem.ID)
|
|
}
|
|
break loop
|
|
}
|
|
wg.Add(1)
|
|
go func(job *models.MetadataImageCacheJob) {
|
|
defer wg.Done()
|
|
defer func() { <-sem }()
|
|
result := p.processOne(ctx, job)
|
|
mu.Lock()
|
|
switch result.outcome {
|
|
case "succeeded":
|
|
stats.Succeeded++
|
|
case "skipped":
|
|
stats.Skipped++
|
|
default:
|
|
stats.Failed++
|
|
}
|
|
stats.UploadedVariants += result.uploadedVariants
|
|
stats.ExistingVariants += result.existingVariants
|
|
mu.Unlock()
|
|
}(job)
|
|
}
|
|
wg.Wait()
|
|
|
|
if len(unstarted) > 0 {
|
|
requeueCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
|
|
if err := p.jobs.RequeueClaimed(requeueCtx, unstarted, workerID); err != nil {
|
|
p.logger.WarnContext(ctx, "metadata image cache: failed to requeue unstarted jobs", "count", len(unstarted), "error", err)
|
|
}
|
|
cancel()
|
|
}
|
|
|
|
p.cleanupSucceeded(ctx, &stats)
|
|
if ctxErr := ctx.Err(); ctxErr != nil && stats.Claimed == 0 {
|
|
return stats, ctxErr
|
|
}
|
|
return stats, nil
|
|
}
|
|
|
|
func (p *ImageCacheProcessor) RunUntilIdle(ctx context.Context, workerID string, claimLimit int, concurrency int, maxRuntime time.Duration) (ImageCacheRunStats, error) {
|
|
var total ImageCacheRunStats
|
|
if p == nil || p.jobs == nil || p.cacher == nil || !p.enabled.Load() {
|
|
return total, nil
|
|
}
|
|
|
|
if maxRuntime <= 0 {
|
|
enqueued, derr := p.discoverExisting(ctx, claimLimit)
|
|
total.EnqueuedExisting += enqueued
|
|
if derr != nil {
|
|
return total, derr
|
|
}
|
|
stats, err := p.RunOnce(ctx, workerID, claimLimit, concurrency)
|
|
total.add(stats)
|
|
total.Batches = 1
|
|
return total, err
|
|
}
|
|
|
|
// Decide once per run whether a full-catalog backfill sweep is due. Within a
|
|
// due run we keep sweeping until the catalog is exhausted; otherwise we only
|
|
// drain the existing queue.
|
|
sweep := p.discoveryDue()
|
|
deadline := time.Now().Add(maxRuntime)
|
|
for {
|
|
if err := ctx.Err(); err != nil {
|
|
return total, err
|
|
}
|
|
if !time.Now().Before(deadline) {
|
|
total.RuntimeLimited = true
|
|
return total, nil
|
|
}
|
|
|
|
stats, err := p.RunOnce(ctx, workerID, claimLimit, concurrency)
|
|
total.Batches++
|
|
total.add(stats)
|
|
if err != nil {
|
|
return total, err
|
|
}
|
|
if stats.Claimed > 0 {
|
|
// Keep draining the queue before spending a full-table sweep.
|
|
continue
|
|
}
|
|
if !sweep {
|
|
return total, nil
|
|
}
|
|
enqueued, err := p.jobs.EnqueueExistingProviderArtwork(ctx, claimLimit)
|
|
if err != nil {
|
|
return total, err
|
|
}
|
|
total.EnqueuedExisting += enqueued
|
|
if enqueued == 0 {
|
|
// Catalog fully swept; throttle the next sweep.
|
|
p.markDiscovered()
|
|
return total, nil
|
|
}
|
|
}
|
|
}
|
|
|
|
// discoveryDue reports whether enough time has elapsed since the last completed
|
|
// sweep to run another one.
|
|
func (p *ImageCacheProcessor) discoveryDue() bool {
|
|
if p.discoveryInterval <= 0 {
|
|
return true
|
|
}
|
|
p.discoveryMu.Lock()
|
|
defer p.discoveryMu.Unlock()
|
|
return p.lastDiscovery.IsZero() || time.Since(p.lastDiscovery) >= p.discoveryInterval
|
|
}
|
|
|
|
func (p *ImageCacheProcessor) markDiscovered() {
|
|
p.discoveryMu.Lock()
|
|
p.lastDiscovery = time.Now()
|
|
p.discoveryMu.Unlock()
|
|
}
|
|
|
|
// discoverExisting runs an unthrottled sweep (single-pass path) and records the
|
|
// time so the throttle applies to subsequent interval-driven runs.
|
|
func (p *ImageCacheProcessor) discoverExisting(ctx context.Context, limit int) (int, error) {
|
|
enqueued, err := p.jobs.EnqueueExistingProviderArtwork(ctx, limit)
|
|
if err == nil {
|
|
p.markDiscovered()
|
|
}
|
|
return enqueued, err
|
|
}
|
|
|
|
func (p *ImageCacheProcessor) cleanupSucceeded(ctx context.Context, stats *ImageCacheRunStats) {
|
|
cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
|
|
defer cancel()
|
|
deleted, err := p.jobs.DeleteSucceededBefore(cleanupCtx, time.Now().Add(-30*24*time.Hour), 1000)
|
|
if err != nil {
|
|
p.logger.WarnContext(ctx, "metadata image cache: failed to delete old succeeded jobs", "error", err)
|
|
} else {
|
|
stats.DeletedSucceeded = deleted
|
|
}
|
|
}
|
|
|
|
func terminalJobContext(parent context.Context) (context.Context, context.CancelFunc) {
|
|
return context.WithTimeout(context.WithoutCancel(parent), 10*time.Second)
|
|
}
|
|
|
|
func (p *ImageCacheProcessor) markFailed(parent context.Context, job *models.MetadataImageCacheJob, errText string) {
|
|
writeCtx, cancel := terminalJobContext(parent)
|
|
defer cancel()
|
|
if err := p.jobs.MarkFailed(writeCtx, job.ID, job.AttemptCount, job.LockedBy, errText); err != nil {
|
|
p.logger.WarnContext(parent, "metadata image cache: failed to mark job failed", "job_id", job.ID, "error", err)
|
|
}
|
|
}
|
|
|
|
func (p *ImageCacheProcessor) markSucceeded(parent context.Context, job *models.MetadataImageCacheJob) {
|
|
writeCtx, cancel := terminalJobContext(parent)
|
|
defer cancel()
|
|
if err := p.jobs.MarkSucceeded(writeCtx, job.ID, job.LockedBy); err != nil {
|
|
p.logger.WarnContext(parent, "metadata image cache: failed to mark job succeeded", "job_id", job.ID, "error", err)
|
|
}
|
|
}
|
|
|
|
type imageCacheProcessResult struct {
|
|
outcome string
|
|
uploadedVariants int
|
|
existingVariants int
|
|
}
|
|
|
|
func (p *ImageCacheProcessor) processOne(ctx context.Context, job *models.MetadataImageCacheJob) imageCacheProcessResult {
|
|
if job == nil {
|
|
return imageCacheProcessResult{outcome: "skipped"}
|
|
}
|
|
imageType, err := imageCacheJobImageType(job.ImageType)
|
|
if err != nil {
|
|
p.markFailed(ctx, job, err.Error())
|
|
return imageCacheProcessResult{outcome: "failed"}
|
|
}
|
|
|
|
// Confirm the target still references this job's source before doing the
|
|
// download and immutable upload. The conditional update below remains the
|
|
// final concurrency guard; this early check avoids work for an obsolete job.
|
|
current, err := p.jobs.CurrentTargetSourcePath(ctx, job)
|
|
if err != nil {
|
|
p.markFailed(ctx, job, err.Error())
|
|
return imageCacheProcessResult{outcome: "failed"}
|
|
}
|
|
if current != job.SourcePath {
|
|
p.markSucceeded(ctx, job)
|
|
return imageCacheProcessResult{outcome: "skipped"}
|
|
}
|
|
|
|
if isLocalImageSourcePath(job.SourcePath) {
|
|
return p.processLocalOne(ctx, job, imageType)
|
|
}
|
|
|
|
downloadURL := job.SourcePath
|
|
if isProviderImagePath(downloadURL) {
|
|
if p.resolver == nil {
|
|
p.markFailed(ctx, job, "missing image resolver")
|
|
return imageCacheProcessResult{outcome: "failed"}
|
|
}
|
|
downloadURL = p.resolver.ResolveImageURL(ctx, job.SourcePath, "original")
|
|
if downloadURL == "" {
|
|
p.markFailed(ctx, job, "image resolver returned empty URL")
|
|
return imageCacheProcessResult{outcome: "failed"}
|
|
}
|
|
}
|
|
|
|
result, err := p.cacher.CacheImage(ctx, CacheImageRequest{
|
|
SourceURL: downloadURL,
|
|
ProviderID: job.ProviderID,
|
|
ContentType: job.ContentType,
|
|
ContentID: job.ProviderContentID,
|
|
ImageType: imageType,
|
|
SeasonNumber: job.SeasonNumber,
|
|
EpisodeNumber: job.EpisodeNumber,
|
|
Language: job.TargetLanguage,
|
|
})
|
|
if err != nil {
|
|
p.markFailed(ctx, job, err.Error())
|
|
return imageCacheProcessResult{outcome: "failed"}
|
|
}
|
|
|
|
if result == nil {
|
|
p.markFailed(ctx, job, "image cache returned no result")
|
|
return imageCacheProcessResult{outcome: "failed"}
|
|
}
|
|
processResult := imageCacheProcessResult{
|
|
uploadedVariants: result.UploadedVariants,
|
|
existingVariants: result.ExistingVariants,
|
|
}
|
|
cachedPath := CachedImageOriginalPath(result)
|
|
if cachedPath == "" {
|
|
p.markFailed(ctx, job, "image cache returned empty stored path")
|
|
processResult.outcome = "failed"
|
|
return processResult
|
|
}
|
|
p.finishJobWithTargetUpdate(ctx, job, cachedPath, result.Thumbhash, &processResult)
|
|
return processResult
|
|
}
|
|
|
|
// finishJobWithTargetUpdate persists the cached path/thumbhash to the job's
|
|
// target row and marks the job terminal, setting processResult.outcome.
|
|
func (p *ImageCacheProcessor) finishJobWithTargetUpdate(ctx context.Context, job *models.MetadataImageCacheJob, cachedPath, thumbhash string, processResult *imageCacheProcessResult) {
|
|
updated, err := p.updateTargetArtwork(ctx, job, cachedPath, thumbhash)
|
|
if err != nil {
|
|
p.markFailed(ctx, job, err.Error())
|
|
processResult.outcome = "failed"
|
|
return
|
|
}
|
|
p.markSucceeded(ctx, job)
|
|
if updated {
|
|
processResult.outcome = "succeeded"
|
|
} else {
|
|
processResult.outcome = "skipped"
|
|
}
|
|
}
|
|
|
|
func (p *ImageCacheProcessor) updateTargetArtwork(ctx context.Context, job *models.MetadataImageCacheJob, cachedPath, thumbhash string) (bool, error) {
|
|
switch job.TargetType {
|
|
case ImageCacheTargetItem:
|
|
if p.targets.Items == nil {
|
|
return false, errors.New("missing item updater")
|
|
}
|
|
return p.targets.Items.UpdateArtworkIfSourceMatches(ctx, job.TargetContentID, job.ImageType, job.SourcePath, cachedPath, thumbhash)
|
|
case ImageCacheTargetItemLocalization:
|
|
if p.targets.ItemLocalizations == nil {
|
|
return false, errors.New("missing item localization updater")
|
|
}
|
|
return p.targets.ItemLocalizations.UpdateArtworkIfSourceMatches(ctx, job.TargetContentID, job.TargetLanguage, job.ImageType, job.SourcePath, cachedPath, thumbhash)
|
|
case ImageCacheTargetSeason:
|
|
if p.targets.Seasons == nil {
|
|
return false, errors.New("missing season updater")
|
|
}
|
|
return p.targets.Seasons.UpdateArtworkIfSourceMatches(ctx, job.TargetContentID, job.SourcePath, cachedPath, thumbhash)
|
|
case ImageCacheTargetSeasonLocalization:
|
|
if p.targets.SeasonLocalizations == nil {
|
|
return false, errors.New("missing season localization updater")
|
|
}
|
|
return p.targets.SeasonLocalizations.UpdateArtworkIfSourceMatches(ctx, job.TargetContentID, job.TargetLanguage, job.SourcePath, cachedPath, thumbhash)
|
|
case ImageCacheTargetEpisode:
|
|
if p.targets.Episodes == nil {
|
|
return false, errors.New("missing episode updater")
|
|
}
|
|
return p.targets.Episodes.UpdateStillIfSourceMatches(ctx, job.TargetContentID, job.SourcePath, cachedPath, thumbhash)
|
|
case ImageCacheTargetPerson:
|
|
if p.targets.People == nil {
|
|
return false, errors.New("missing person updater")
|
|
}
|
|
personID, parseErr := strconv.ParseInt(job.TargetContentID, 10, 64)
|
|
if parseErr != nil {
|
|
return false, fmt.Errorf("invalid person image cache target %q: %w", job.TargetContentID, parseErr)
|
|
}
|
|
return p.targets.People.UpdatePhotoIfSourceMatches(ctx, personID, job.SourcePath, cachedPath, thumbhash)
|
|
default:
|
|
return false, fmt.Errorf("unknown image cache target type %q", job.TargetType)
|
|
}
|
|
}
|
|
|
|
// processLocalOne caches a local sidecar image (file:// source). The file is
|
|
// confined to the owning library's roots by a lexical check on the logical path
|
|
// followed by a symlink-resolving re-check (both path and roots are resolved, so
|
|
// scanner-recorded logical paths under symlinked roots stay valid while an
|
|
// intermediate directory symlink escaping a root is rejected), then read through
|
|
// an opened handle with fstat re-checks and pushed to S3 under a content-hashed
|
|
// local/ key.
|
|
func (p *ImageCacheProcessor) processLocalOne(ctx context.Context, job *models.MetadataImageCacheJob, imageType ImageType) imageCacheProcessResult {
|
|
byteCacher, ok := p.cacher.(ImageByteCacher)
|
|
if !ok {
|
|
p.markFailed(ctx, job, "image cacher does not support local artwork")
|
|
return imageCacheProcessResult{outcome: "failed"}
|
|
}
|
|
if p.libraryRoots == nil {
|
|
p.markFailed(ctx, job, "missing library root resolver for local artwork")
|
|
return imageCacheProcessResult{outcome: "failed"}
|
|
}
|
|
|
|
localPath := filepath.Clean(strings.TrimSpace(job.SourcePath)[len("file://"):])
|
|
rootsContentID := firstNonEmpty(job.SeriesID, job.TargetContentID)
|
|
roots, err := p.libraryRoots.LibraryRootsForContent(ctx, rootsContentID)
|
|
if err != nil {
|
|
p.markFailed(ctx, job, fmt.Sprintf("resolving library roots: %v", err))
|
|
return imageCacheProcessResult{outcome: "failed"}
|
|
}
|
|
if !localImagePathWithinRoots(localPath, roots) {
|
|
p.markFailed(ctx, job, "local image path outside library roots: "+localPath)
|
|
return imageCacheProcessResult{outcome: "failed"}
|
|
}
|
|
// The lexical check above cannot see through symlinks. Resolve the path and
|
|
// roots and re-confine so an intermediate directory symlink planted inside a
|
|
// root (e.g. a link pointing out of the library) cannot pull an out-of-root
|
|
// file into the cache. EvalSymlinks resolves both sides, so a legitimately
|
|
// symlinked root still matches. This is a confinement GATE only: the read
|
|
// below still uses the logical path so readLocalImageFile's Lstat keeps
|
|
// rejecting a symlinked leaf. A not-yet-existent path (ErrNotExist) falls
|
|
// through to the reader, which classifies it as the stable "missing" failure.
|
|
if _, err := localImagePathResolvedWithinRoots(localPath, roots); err != nil && !errors.Is(err, fs.ErrNotExist) {
|
|
p.markFailed(ctx, job, "local image path outside library roots: "+localPath)
|
|
return imageCacheProcessResult{outcome: "failed"}
|
|
}
|
|
|
|
data, err := readLocalImageFile(localPath)
|
|
if err != nil {
|
|
p.markFailed(ctx, job, err.Error())
|
|
return imageCacheProcessResult{outcome: "failed"}
|
|
}
|
|
|
|
// The target's current cached path drives the unchanged-skip (same hash →
|
|
// same key → nothing to sweep) and the stale-prefix cleanup below.
|
|
previousCachedPath := ""
|
|
if reader, ok := p.jobs.(imageCacheTargetCachedPathReader); ok {
|
|
previousCachedPath, err = reader.CurrentTargetCachedPath(ctx, job)
|
|
if err != nil {
|
|
p.logger.WarnContext(ctx, "metadata image cache: failed to read current cached path", "job_id", job.ID, "error", err)
|
|
previousCachedPath = ""
|
|
}
|
|
}
|
|
|
|
digest := sha256.Sum256(data)
|
|
result, err := byteCacher.CacheImageBytes(ctx, data, CacheImageRequest{
|
|
SourceURL: job.SourcePath,
|
|
ProviderID: imageCacheLocalProviderID,
|
|
ContentType: job.ContentType,
|
|
ContentID: job.ProviderContentID,
|
|
ImageType: imageType,
|
|
SeasonNumber: job.SeasonNumber,
|
|
EpisodeNumber: job.EpisodeNumber,
|
|
Language: job.TargetLanguage,
|
|
KeyDiscriminator: hex.EncodeToString(digest[:4]),
|
|
})
|
|
if err != nil {
|
|
p.markFailed(ctx, job, err.Error())
|
|
return imageCacheProcessResult{outcome: "failed"}
|
|
}
|
|
if result == nil {
|
|
p.markFailed(ctx, job, "image cache returned no result")
|
|
return imageCacheProcessResult{outcome: "failed"}
|
|
}
|
|
processResult := imageCacheProcessResult{
|
|
uploadedVariants: result.UploadedVariants,
|
|
existingVariants: result.ExistingVariants,
|
|
}
|
|
cachedPath := CachedImageOriginalPath(result)
|
|
if cachedPath == "" {
|
|
p.markFailed(ctx, job, "image cache returned empty stored path")
|
|
processResult.outcome = "failed"
|
|
return processResult
|
|
}
|
|
p.finishJobWithTargetUpdate(ctx, job, cachedPath, result.Thumbhash, &processResult)
|
|
if processResult.outcome == "succeeded" {
|
|
p.deleteStaleLocalPrefix(ctx, previousCachedPath, cachedPath)
|
|
}
|
|
return processResult
|
|
}
|
|
|
|
// deleteStaleLocalPrefix removes the previous hashed local/ image prefix
|
|
// after a re-cache stored the artwork under a different key.
|
|
func (p *ImageCacheProcessor) deleteStaleLocalPrefix(ctx context.Context, previousCachedPath, cachedPath string) {
|
|
previousCachedPath = strings.TrimSpace(previousCachedPath)
|
|
if p.prefixDeleter == nil ||
|
|
previousCachedPath == "" ||
|
|
previousCachedPath == cachedPath ||
|
|
!strings.HasPrefix(previousCachedPath, "local/") {
|
|
return
|
|
}
|
|
lastSlash := strings.LastIndex(previousCachedPath, "/")
|
|
if lastSlash <= 0 {
|
|
return
|
|
}
|
|
prefix := previousCachedPath[:lastSlash+1]
|
|
if strings.HasPrefix(cachedPath, prefix) {
|
|
return
|
|
}
|
|
if _, err := p.prefixDeleter.DeletePrefix(ctx, p.prefixDeleter.Bucket(), prefix); err != nil {
|
|
p.logger.WarnContext(ctx, "metadata image cache: failed to delete stale local image prefix", "prefix", prefix, "error", err)
|
|
}
|
|
}
|
|
|
|
// errLocalImageOutsideRoots is returned by localImagePathResolvedWithinRoots
|
|
// when a fully symlink-resolved path escapes every resolved library root.
|
|
var errLocalImageOutsideRoots = errors.New("local image path outside library roots")
|
|
|
|
// localImagePathResolvedWithinRoots resolves symlinks on both the path and the
|
|
// library roots and returns the real path when it stays within a real root.
|
|
// This closes the gap the lexical check leaves open — an intermediate directory
|
|
// symlink planted inside a root that points outside it. Resolving both sides
|
|
// keeps a legitimately symlinked root valid. A path that does not exist yet
|
|
// surfaces as fs.ErrNotExist for the caller to treat as a stable "missing"
|
|
// failure via the reader; any other resolution failure fails closed.
|
|
func localImagePathResolvedWithinRoots(path string, roots []string) (string, error) {
|
|
resolvedPath, err := filepath.EvalSymlinks(path)
|
|
if err != nil {
|
|
return "", err
|
|
}
|
|
for _, root := range roots {
|
|
resolvedRoot, err := filepath.EvalSymlinks(strings.TrimSpace(root))
|
|
if err != nil {
|
|
continue
|
|
}
|
|
if localImagePathWithinRoots(resolvedPath, []string{resolvedRoot}) {
|
|
return resolvedPath, nil
|
|
}
|
|
}
|
|
return "", errLocalImageOutsideRoots
|
|
}
|
|
|
|
// localImagePathWithinRoots confines path (already cleaned, logical) to the
|
|
// given library roots with a separator-aware lexical prefix check.
|
|
func localImagePathWithinRoots(path string, roots []string) bool {
|
|
if !filepath.IsAbs(path) {
|
|
return false
|
|
}
|
|
for _, root := range roots {
|
|
root = filepath.Clean(strings.TrimSpace(root))
|
|
if root == "" || root == "." || !filepath.IsAbs(root) {
|
|
continue
|
|
}
|
|
if path == root {
|
|
return true
|
|
}
|
|
prefix := root
|
|
if prefix != string(filepath.Separator) {
|
|
prefix += string(filepath.Separator)
|
|
}
|
|
if strings.HasPrefix(path, prefix) {
|
|
return true
|
|
}
|
|
}
|
|
return false
|
|
}
|
|
|
|
// readLocalImageFile reads a sidecar image with the same guards as discovery:
|
|
// Lstat rejects symlinked leaves and non-regular files, the opened handle is
|
|
// fstat-re-checked, and reads cap at maxLocalImageSourceBytes. ENOENT/EPERM
|
|
// map to the stable-failure texts matched by isStableProviderImageFailure.
|
|
func readLocalImageFile(path string) ([]byte, error) {
|
|
classify := func(err error) error {
|
|
switch {
|
|
case errors.Is(err, fs.ErrNotExist):
|
|
return fmt.Errorf("local image missing: %s", path)
|
|
case errors.Is(err, fs.ErrPermission):
|
|
return fmt.Errorf("local image forbidden: %s", path)
|
|
default:
|
|
return fmt.Errorf("local image read failed: %w", err)
|
|
}
|
|
}
|
|
info, err := os.Lstat(path)
|
|
if err != nil {
|
|
return nil, classify(err)
|
|
}
|
|
if !info.Mode().IsRegular() {
|
|
return nil, fmt.Errorf("local image is not a regular file: %s", path)
|
|
}
|
|
if info.Size() > maxLocalImageSourceBytes {
|
|
return nil, fmt.Errorf("local image exceeds %d byte limit: %s", maxLocalImageSourceBytes, path)
|
|
}
|
|
file, err := os.Open(path)
|
|
if err != nil {
|
|
return nil, classify(err)
|
|
}
|
|
defer func() { _ = file.Close() }()
|
|
stat, err := file.Stat()
|
|
if err != nil {
|
|
return nil, classify(err)
|
|
}
|
|
// Close the symlink-swap window: os.Open follows symlinks, so a leaf swapped
|
|
// to a symlink between the Lstat above and this Open would be followed to its
|
|
// target. Reject unless the opened handle is the exact file Lstat inspected,
|
|
// so an out-of-root target can never be pulled into the public cache.
|
|
if !os.SameFile(info, stat) {
|
|
return nil, fmt.Errorf("local image is not a regular file: %s", path)
|
|
}
|
|
if !stat.Mode().IsRegular() {
|
|
return nil, fmt.Errorf("local image is not a regular file: %s", path)
|
|
}
|
|
if stat.Size() > maxLocalImageSourceBytes {
|
|
return nil, fmt.Errorf("local image exceeds %d byte limit: %s", maxLocalImageSourceBytes, path)
|
|
}
|
|
data, err := io.ReadAll(io.LimitReader(file, maxLocalImageSourceBytes+1))
|
|
if err != nil {
|
|
return nil, classify(err)
|
|
}
|
|
if len(data) > maxLocalImageSourceBytes {
|
|
return nil, fmt.Errorf("local image exceeds %d byte limit: %s", maxLocalImageSourceBytes, path)
|
|
}
|
|
if len(data) == 0 {
|
|
return nil, fmt.Errorf("local image read failed: empty file %s", path)
|
|
}
|
|
return data, nil
|
|
}
|
|
|
|
func imageCacheJobImageType(value string) (ImageType, error) {
|
|
switch value {
|
|
case ImageCacheImagePoster:
|
|
return ImagePoster, nil
|
|
case ImageCacheImageBackdrop:
|
|
return ImageBackdrop, nil
|
|
case ImageCacheImageLogo:
|
|
return ImageLogo, nil
|
|
case ImageCacheImageStill:
|
|
return ImageStill, nil
|
|
case ImageCacheImageProfile:
|
|
return ImageProfile, nil
|
|
default:
|
|
return ImagePoster, fmt.Errorf("unknown metadata image cache image type %q", value)
|
|
}
|
|
}
|