* feat(metadata): register builtin NFO provider and broaden parsing Phases A and B of the #216 local-NFO work, implemented test-first. Registration & hint-first identity (Phase A): - Migration seeds a reserved kind='builtin' silo.builtin installation and an 'nfo' metadata capability (default_enabled=false, priority 1 for movie/series) with a partial unique index and documented Down. - In-process builtin provider registry (internal/metadata/builtin.go); buildProviders returns the registered provider for builtin rows. - Guard rails keep the reserved row out of every plugin surface (user plugin-settings, installations list, image resolvers, preload, auto-update, store Delete, mutation handlers -> 409); silo.builtin is a reserved manifest id. - Startup sync materializes legacy content_level='' chains per level, then appends builtin capabilities disabled via AppendProviderToAllChains (idempotent); resolveEnabledProvidersBy priority now respects default_enabled=false. - NFO uniqueids seed the trusted-hint machinery via IdentityHintProvider with per-mode conflict policy (stored IDs win on scheduled refresh, NFO wins on manual refresh, Identify skips NFO); ID-less candidates are excluded from provider-priority tie-breaks and nfo never counts as corroboration. - Web chain-editor empty-state gate is now server-derived so builtin providers are reachable on plugin-less servers. Parser breadth & sidecar hardening (Phase B): - Parser covers the practical Kodi/Jellyfin field set for <movie> and <tvshow>: original title, tagline, runtime, dates, content rating, genres/studios/countries/tags, multi-source ratings with scale normalization, cast with roles/order, director/credits. Empty collections stay nil so merge early-returns apply. - findNFO parses candidates and falls through on read/parse failure or root-type mismatch, so a stray movie.nfo cannot shadow tvshow.nfo; GetMetadata gains the same ContentType guard Search has. - New FieldReleaseDates lock gates Year/ReleaseDate/First+LastAirDate in merge (Go) and the edit-metadata dialog (web), closing the gap where a manual refresh re-applied NFO dates over admin corrections. - Merge-contract tests pin NFO fill semantics, genres whole-list first-provider-wins, and NFO edits propagating on manual refresh only. - Docs: new admin wiki page (supported fields, merge semantics, naming-supplies-structure contract), index bullet, sidecar wording revision, v1-scope feature-detection note. Zero behavior change while the provider is disabled (default); pinned by CI-mode and DB-gated test suites. Part of #216 AI-use disclosure: implemented with Claude Code (Fable 5) via spec-driven TDD and agent-assisted implementation. * feat(metadata): ingest local sidecar artwork and read series-depth NFO Phases C and D of the #216 local-NFO work, implemented test-first, plus the mixed-library use-case pins. Together these deliver the headline case: a series absent from every remote database (e.g. a fitness library) scans into a fully presented show -> named seasons -> titled episodes tree from NFO files and sidecar art alone. Local sidecar artwork through the S3 image cache (Phase C): - The NFO provider implements ImageProvider: poster/backdrop/logo sidecar discovery with a fixed precedence map, symlink/non-regular rejection, an 8 MiB cap, and file:// source URLs at rating 0. Generic filenames apply only via the sidecar search paths, so a shared folder.jpg in a flat multi-movie directory applies to none. - file:// becomes a live local source scheme: routed into *_source_path (never *_path), accepted by every image enqueue gate, attributed as provider "local", excluded from cached-path detection. - The image-cache processor caches local files with lexical-on-logical confinement to the library roots, open-handle reads with re-checks, the same variant widths as remote art, and stable (7-day) failure classification. Keys land under local/{contentType}/{contentID}/{hash8}/{imageType}; superseded prefixes are cleaned on re-cache and item deletion. - applyIfBetter gains a local exemption so rating-0 local art can fill matched items without being stickily displaced; ImageRequest carries additive sidecar path context. Series depth (Phase D): - SeasonsRequest/EpisodesRequest carry additive local path context (series roots, per-season directories, per-episode file paths), derived from naming at match time and reconstructed on refresh. - season.nfo supplies season name/plot; NFO season numbers are advisory (directory-derived number wins with a Warn - naming owns structure). <episodedetails> gains aired/runtime/ratings; <basename>.nfo titles episodes and <basename>-thumb.ext supplies thumbs; filename SxxEyy wins over NFO numbers. - Episode NFOs work without a season.nfo (provider seasons unioned with on-disk seasons); SynthesizeFallbackEpisodes always runs after persist so NFO-less episodes keep synthesized rows. Season/episode file:// art rides the Phase C pipeline unchanged. - Migration adds season:1/episode:1 to the builtin NFO capability's default_priority (still default_enabled=false). Mixed sports-library use case (tests only, no product change): - Pins the classification contract for one library holding movie-shaped and show-shaped content (WWE PPV events as movies next to a "WWE SmackDown" show, NASCAR/F1/FIFA with partial TVDB/TMDB data): naming decides movie-vs-series per file before any provider runs; the NFO supplies metadata/identity but never flips type (ContentType guard); the per-root Type override is the correction path. - NFO-driven type classification at scan time is recorded as an explicit deferred open question. Part of #216 AI-use disclosure: implemented with Claude Code (Fable 5) via spec-driven TDD and agent-assisted implementation. * docs(metadata): document local NFO metadata architecture Add a single as-built architecture page (docs/architecture/local-nfo-metadata.md) for the #216 local-NFO feature: the builtin registration model, hint-first identity semantics, the file:// -> S3 artwork pipeline and its deployment constraint, series depth, the mixed-library classification contract, and known limitations. This replaces the working implementation plan, the per-phase specs, and the narrow sidecar-artwork note, which were planning drafts and are left untracked; admin-facing behavior remains in the wiki. Part of #216 AI-use disclosure: planned, drafted, and consolidated with Claude Code (Fable 5) using multi-agent exploration and adversarial review. * fix(metadata): address PR review findings on NFO builtin provider Fold in the valid, low-risk fixes surfaced by automated review on #390: - imagecache: extract validateCacheRequest so CacheBytes (the local sidecar season/episode path) enforces the same episode-requires-season guard as Cache, preventing distinct episodes' art from colliding under one S3 key. - image_cache_processor: close the sidecar symlink-swap window by rejecting the opened handle unless os.SameFile matches the Lstat'd file, so a leaf swapped to a symlink can't pull an out-of-root target into the public cache. - plugins: guard the reserved builtin installation row in the store's Update, matching Delete, so its version/enabled/capabilities can never be rewritten even if a mutation slips past the HTTP layer. - cmd/silo: bound SyncBuiltinProviderChains with a 30s timeout so a stuck DB round-trip fails fast at startup instead of hanging. - metadata: panic instead of silently no-op'ing on an invalid RegisterBuiltinProvider call (init-time programmer error). - docs: correct the media-folder-and-naming NFO paragraph to state season/episode NFOs and sidecar artwork are actively read. --------- Co-authored-by: Quick104 <31828688+Quick104@users.noreply.github.com>
323 lines
10 KiB
Go
323 lines
10 KiB
Go
package metadata
|
|
|
|
import (
|
|
"context"
|
|
"fmt"
|
|
"os"
|
|
"path/filepath"
|
|
"testing"
|
|
|
|
"github.com/Silo-Server/silo-server/internal/models"
|
|
)
|
|
|
|
// fakeLocalImageCacheJobs is a fakeImageCacheJobs that also reports the
|
|
// target's currently stored cached path, like the real repository.
|
|
type fakeLocalImageCacheJobs struct {
|
|
fakeImageCacheJobs
|
|
currentCached string
|
|
}
|
|
|
|
func (f *fakeLocalImageCacheJobs) CurrentTargetCachedPath(context.Context, *models.MetadataImageCacheJob) (string, error) {
|
|
return f.currentCached, nil
|
|
}
|
|
|
|
// fakeByteImageCacher supports both remote and byte caching. CacheImageBytes
|
|
// builds the base path the way the real cacher does so key assertions hold.
|
|
type fakeByteImageCacher struct {
|
|
fakeImageCacher
|
|
byteReqs [][]byte
|
|
bytesReq []CacheImageRequest
|
|
thumbhash string
|
|
}
|
|
|
|
func (f *fakeByteImageCacher) CacheImageBytes(_ context.Context, data []byte, req CacheImageRequest) (*CacheImageResult, error) {
|
|
f.byteReqs = append(f.byteReqs, data)
|
|
f.bytesReq = append(f.bytesReq, req)
|
|
if f.err != nil {
|
|
return nil, f.err
|
|
}
|
|
return &CacheImageResult{
|
|
BasePath: fmt.Sprintf("%s/%s/%s/%s/%s", req.ProviderID, req.ContentType, req.ContentID, req.KeyDiscriminator, ImageTypeToString(req.ImageType)),
|
|
Thumbhash: f.thumbhash,
|
|
Ext: ".webp",
|
|
}, nil
|
|
}
|
|
|
|
type fakeLibraryRootResolver struct {
|
|
roots []string
|
|
err error
|
|
}
|
|
|
|
func (f *fakeLibraryRootResolver) LibraryRootsForContent(context.Context, string) ([]string, error) {
|
|
return f.roots, f.err
|
|
}
|
|
|
|
type fakePrefixDeleter struct {
|
|
bucket string
|
|
prefixes []string
|
|
}
|
|
|
|
func (f *fakePrefixDeleter) DeletePrefix(_ context.Context, _ string, prefix string) (int, error) {
|
|
f.prefixes = append(f.prefixes, prefix)
|
|
return 1, nil
|
|
}
|
|
|
|
func (f *fakePrefixDeleter) Bucket() string { return f.bucket }
|
|
|
|
func localArtworkJob(sourcePath string) *models.MetadataImageCacheJob {
|
|
return &models.MetadataImageCacheJob{
|
|
ID: 7,
|
|
TargetType: ImageCacheTargetItem,
|
|
TargetContentID: "movie-1",
|
|
SeriesID: "movie-1",
|
|
SourcePath: sourcePath,
|
|
ProviderID: "local",
|
|
ProviderContentID: "movie-1",
|
|
ContentType: "movies",
|
|
ImageType: ImageCacheImagePoster,
|
|
}
|
|
}
|
|
|
|
func newLocalProcessorForTest(
|
|
job *models.MetadataImageCacheJob,
|
|
roots []string,
|
|
currentCached string,
|
|
) (*ImageCacheProcessor, *fakeLocalImageCacheJobs, *fakeByteImageCacher, *fakeItemArtworkUpdater, *fakePrefixDeleter) {
|
|
jobs := &fakeLocalImageCacheJobs{
|
|
fakeImageCacheJobs: fakeImageCacheJobs{claimed: []*models.MetadataImageCacheJob{job}},
|
|
currentCached: currentCached,
|
|
}
|
|
cacher := &fakeByteImageCacher{thumbhash: "th-new"}
|
|
items := &fakeItemArtworkUpdater{updated: true}
|
|
deleter := &fakePrefixDeleter{bucket: "images"}
|
|
p := NewImageCacheProcessorWithTargets(jobs, cacher, nil, ImageCacheProcessorTargets{Items: items})
|
|
p.SetLibraryRootResolver(&fakeLibraryRootResolver{roots: roots})
|
|
p.SetImagePrefixDeleter(deleter)
|
|
return p, jobs, cacher, items, deleter
|
|
}
|
|
|
|
func writeLocalPoster(t *testing.T, dir string) string {
|
|
t.Helper()
|
|
path := filepath.Join(dir, "poster.jpg")
|
|
if err := os.WriteFile(path, []byte("poster-bytes"), 0o644); err != nil {
|
|
t.Fatalf("writing poster: %v", err)
|
|
}
|
|
return path
|
|
}
|
|
|
|
func TestProcessLocalImageCachesWithinLibraryRoots(t *testing.T) {
|
|
root := t.TempDir()
|
|
posterPath := writeLocalPoster(t, root)
|
|
job := localArtworkJob("file://" + posterPath)
|
|
p, jobs, cacher, items, deleter := newLocalProcessorForTest(job, []string{root}, "")
|
|
|
|
stats, err := p.RunOnce(context.Background(), "w1", 10, 1)
|
|
if err != nil {
|
|
t.Fatalf("RunOnce: %v", err)
|
|
}
|
|
if stats.Succeeded != 1 {
|
|
t.Fatalf("stats = %+v, want 1 succeeded (failed text %q)", stats, jobs.failedText)
|
|
}
|
|
if len(cacher.bytesReq) != 1 {
|
|
t.Fatalf("CacheImageBytes calls = %d, want 1", len(cacher.bytesReq))
|
|
}
|
|
req := cacher.bytesReq[0]
|
|
if req.ProviderID != "local" || req.ContentType != "movies" || req.ContentID != "movie-1" {
|
|
t.Fatalf("cache request = %+v", req)
|
|
}
|
|
if len(req.KeyDiscriminator) != 8 {
|
|
t.Fatalf("KeyDiscriminator = %q, want 8-char content hash", req.KeyDiscriminator)
|
|
}
|
|
wantCached := "local/movies/movie-1/" + req.KeyDiscriminator + "/poster/original.webp"
|
|
if items.cachedPath != wantCached {
|
|
t.Fatalf("cachedPath = %q, want %q", items.cachedPath, wantCached)
|
|
}
|
|
if items.thumbhash != "th-new" {
|
|
t.Fatalf("thumbhash = %q", items.thumbhash)
|
|
}
|
|
if len(deleter.prefixes) != 0 {
|
|
t.Fatalf("no stale prefix to delete on first cache, got %v", deleter.prefixes)
|
|
}
|
|
}
|
|
|
|
func TestProcessLocalImageRejectsPathOutsideRoots(t *testing.T) {
|
|
root := t.TempDir()
|
|
other := t.TempDir()
|
|
posterPath := writeLocalPoster(t, other)
|
|
job := localArtworkJob("file://" + posterPath)
|
|
p, jobs, cacher, _, _ := newLocalProcessorForTest(job, []string{root}, "")
|
|
|
|
stats, err := p.RunOnce(context.Background(), "w1", 10, 1)
|
|
if err != nil {
|
|
t.Fatalf("RunOnce: %v", err)
|
|
}
|
|
if stats.Failed != 1 {
|
|
t.Fatalf("stats = %+v, want 1 failed", stats)
|
|
}
|
|
if !isStableProviderImageFailure(jobs.failedText) {
|
|
t.Fatalf("confinement failure %q must be a stable failure", jobs.failedText)
|
|
}
|
|
if len(cacher.bytesReq) != 0 {
|
|
t.Fatal("must not read or cache a file outside library roots")
|
|
}
|
|
}
|
|
|
|
func TestProcessLocalImageRejectsDotDotTraversal(t *testing.T) {
|
|
root := t.TempDir()
|
|
outside := t.TempDir()
|
|
posterPath := writeLocalPoster(t, outside)
|
|
traversal := root + "/../" + filepath.Base(outside) + "/" + filepath.Base(posterPath)
|
|
job := localArtworkJob("file://" + traversal)
|
|
p, jobs, cacher, _, _ := newLocalProcessorForTest(job, []string{root}, "")
|
|
|
|
stats, err := p.RunOnce(context.Background(), "w1", 10, 1)
|
|
if err != nil {
|
|
t.Fatalf("RunOnce: %v", err)
|
|
}
|
|
if stats.Failed != 1 {
|
|
t.Fatalf("stats = %+v, want 1 failed", stats)
|
|
}
|
|
if !isStableProviderImageFailure(jobs.failedText) {
|
|
t.Fatalf("traversal failure %q must be a stable failure", jobs.failedText)
|
|
}
|
|
if len(cacher.bytesReq) != 0 {
|
|
t.Fatal("must not cache a traversal path")
|
|
}
|
|
}
|
|
|
|
func TestProcessLocalImageAcceptsLogicalPathUnderSymlinkedRoot(t *testing.T) {
|
|
// The scanner records logical paths under symlinked roots; confinement is
|
|
// deliberately lexical-on-logical (no EvalSymlinks).
|
|
realRoot := t.TempDir()
|
|
writeLocalPoster(t, realRoot)
|
|
linkParent := t.TempDir()
|
|
linkedRoot := filepath.Join(linkParent, "library")
|
|
if err := os.Symlink(realRoot, linkedRoot); err != nil {
|
|
t.Skipf("symlinks unavailable: %v", err)
|
|
}
|
|
job := localArtworkJob("file://" + filepath.Join(linkedRoot, "poster.jpg"))
|
|
p, jobs, cacher, _, _ := newLocalProcessorForTest(job, []string{linkedRoot}, "")
|
|
|
|
stats, err := p.RunOnce(context.Background(), "w1", 10, 1)
|
|
if err != nil {
|
|
t.Fatalf("RunOnce: %v", err)
|
|
}
|
|
if stats.Succeeded != 1 {
|
|
t.Fatalf("stats = %+v (failed text %q), want success under symlinked root", stats, jobs.failedText)
|
|
}
|
|
if len(cacher.bytesReq) != 1 {
|
|
t.Fatal("expected the logical path to be read and cached")
|
|
}
|
|
}
|
|
|
|
func TestProcessLocalImageMissingFileIsStableFailure(t *testing.T) {
|
|
root := t.TempDir()
|
|
job := localArtworkJob("file://" + filepath.Join(root, "poster.jpg"))
|
|
p, jobs, _, _, _ := newLocalProcessorForTest(job, []string{root}, "")
|
|
|
|
stats, err := p.RunOnce(context.Background(), "w1", 10, 1)
|
|
if err != nil {
|
|
t.Fatalf("RunOnce: %v", err)
|
|
}
|
|
if stats.Failed != 1 {
|
|
t.Fatalf("stats = %+v, want 1 failed", stats)
|
|
}
|
|
if !isStableProviderImageFailure(jobs.failedText) {
|
|
t.Fatalf("ENOENT failure %q must be a stable failure", jobs.failedText)
|
|
}
|
|
}
|
|
|
|
func TestProcessLocalImageRejectsSymlinkedLeaf(t *testing.T) {
|
|
root := t.TempDir()
|
|
target := writeLocalPoster(t, root)
|
|
link := filepath.Join(root, "linked.jpg")
|
|
if err := os.Symlink(target, link); err != nil {
|
|
t.Skipf("symlinks unavailable: %v", err)
|
|
}
|
|
job := localArtworkJob("file://" + link)
|
|
p, _, cacher, _, _ := newLocalProcessorForTest(job, []string{root}, "")
|
|
|
|
stats, err := p.RunOnce(context.Background(), "w1", 10, 1)
|
|
if err != nil {
|
|
t.Fatalf("RunOnce: %v", err)
|
|
}
|
|
if stats.Failed != 1 {
|
|
t.Fatalf("stats = %+v, want 1 failed", stats)
|
|
}
|
|
if len(cacher.bytesReq) != 0 {
|
|
t.Fatal("must not cache a symlinked leaf")
|
|
}
|
|
}
|
|
|
|
func TestProcessLocalImageDeletesStalePrefixOnRecache(t *testing.T) {
|
|
root := t.TempDir()
|
|
posterPath := writeLocalPoster(t, root)
|
|
job := localArtworkJob("file://" + posterPath)
|
|
stale := "local/movies/movie-1/00000000/poster/original.webp"
|
|
p, _, cacher, _, deleter := newLocalProcessorForTest(job, []string{root}, stale)
|
|
|
|
stats, err := p.RunOnce(context.Background(), "w1", 10, 1)
|
|
if err != nil {
|
|
t.Fatalf("RunOnce: %v", err)
|
|
}
|
|
if stats.Succeeded != 1 {
|
|
t.Fatalf("stats = %+v, want success", stats)
|
|
}
|
|
if len(cacher.bytesReq) != 1 {
|
|
t.Fatal("expected one cache call")
|
|
}
|
|
if len(deleter.prefixes) != 1 || deleter.prefixes[0] != "local/movies/movie-1/00000000/poster/" {
|
|
t.Fatalf("stale prefixes deleted = %v", deleter.prefixes)
|
|
}
|
|
}
|
|
|
|
func TestProcessLocalImageSkipsPrefixDeleteWhenUnchanged(t *testing.T) {
|
|
root := t.TempDir()
|
|
posterPath := writeLocalPoster(t, root)
|
|
job := localArtworkJob("file://" + posterPath)
|
|
|
|
// First pass discovers the hash for these bytes.
|
|
probe, _, probeCacher, probeItems, _ := newLocalProcessorForTest(job, []string{root}, "")
|
|
if _, err := probe.RunOnce(context.Background(), "w1", 10, 1); err != nil {
|
|
t.Fatalf("probe RunOnce: %v", err)
|
|
}
|
|
if len(probeCacher.bytesReq) != 1 {
|
|
t.Fatal("probe cache call missing")
|
|
}
|
|
|
|
// Second pass with the same stored cached path must not delete anything.
|
|
p, _, _, _, deleter := newLocalProcessorForTest(job, []string{root}, probeItems.cachedPath)
|
|
stats, err := p.RunOnce(context.Background(), "w1", 10, 1)
|
|
if err != nil {
|
|
t.Fatalf("RunOnce: %v", err)
|
|
}
|
|
if stats.Succeeded != 1 {
|
|
t.Fatalf("stats = %+v, want success", stats)
|
|
}
|
|
if len(deleter.prefixes) != 0 {
|
|
t.Fatalf("unchanged art must not delete prefixes, got %v", deleter.prefixes)
|
|
}
|
|
}
|
|
|
|
func TestProcessLocalImageFailsWithoutRootResolver(t *testing.T) {
|
|
root := t.TempDir()
|
|
posterPath := writeLocalPoster(t, root)
|
|
job := localArtworkJob("file://" + posterPath)
|
|
jobs := &fakeLocalImageCacheJobs{
|
|
fakeImageCacheJobs: fakeImageCacheJobs{claimed: []*models.MetadataImageCacheJob{job}},
|
|
}
|
|
cacher := &fakeByteImageCacher{thumbhash: "th"}
|
|
p := NewImageCacheProcessorWithTargets(jobs, cacher, nil, ImageCacheProcessorTargets{Items: &fakeItemArtworkUpdater{updated: true}})
|
|
|
|
stats, err := p.RunOnce(context.Background(), "w1", 10, 1)
|
|
if err != nil {
|
|
t.Fatalf("RunOnce: %v", err)
|
|
}
|
|
if stats.Failed != 1 {
|
|
t.Fatalf("stats = %+v, want failure without a library root resolver", stats)
|
|
}
|
|
if len(cacher.bytesReq) != 0 {
|
|
t.Fatal("must not cache without confinement roots")
|
|
}
|
|
}
|