* feat(metadata): reconcile artwork cache after public S3 provider changes Changing the public S3 provider previously broke every cached image permanently: the DB keeps bucket-relative keys, the image cache pipeline treats a cached path as its durable dedup marker and never re-enqueues, and clients eat the 404s straight from S3 so the server never notices. Add a storage identity fingerprint (s3.public_storage_identity, seeded via SetIfAbsent at boot) and a reconcile_artwork_cache task whose startup trigger only fires when the identity changed; manual runs always sweep, doubling as bucket-data-loss recovery. The task probes a random sample of cached objects, then either bulk-resets (near-total miss) or per-row verifies. Missing provider-sourced artwork is reset to its *_source_path so the existing enqueue loop re-caches it; surfaces without a re-downloadable source (chapter thumbnails, collection artwork, library posters, branding refs, embedded book covers) are cleared so their owning pipelines refill them. Small upload-holding tables are always per-row verified so bulk mode cannot blind-clear an upload that survived migration, and transport errors never reset rows. Users never see broken images during the transition: reset rows serve the provider's original URL via the existing absolute-URL pass-through and thumbhashes are preserved. The storage settings page now warns that uploads cannot be re-downloaded when the identity fields are edited. Part of #348 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(metadata): harden artwork reconcile per code review Address the confirmed findings from the PR review: - Fingerprint the key prefix case-sensitively and slash-trimmed exactly as s3client applies it (new exported NormalizeKeyPrefix): a case-only prefix edit is a real storage move and must reconcile; a slash-only edit is not and must not. - Certify the storage fingerprint immediately after the artwork sweep succeeds and make the 4-object branding check non-fatal (reported in the task message), so a transient branding error cannot discard a completed catalog sweep and force it to repeat every boot. - Fail closed on conditional-task preflight errors in the task manager (previously fail-open ran the task), and retry transient settings reads in ShouldRun since the startup trigger fires once per process. - Track probe HEAD errors against a separate baseline so a flaky probe cannot consume the sweep's error budget. - Probe before counting: bulk mode skips the per-surface count(*) full scans entirely, and probe sampling drops ORDER BY random() (plain LIMIT answers "is the cache in this bucket" just as well). - Verify chapter thumbnails across a whole 500-file batch in one HEAD fan-out instead of per file, keeping the worker pool saturated. - Replace the 10 inline non-provider-scheme ARRAY literals in the enqueue query with the shared nonProviderImageSchemesSQL constant. Part of #348 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(metadata): guard bulk reset against degraded probes, certify only clean sweeps Address bot review feedback on the reconcile hardening: - A probe where more than half the HEAD requests error aborts the run: errored requests are excluded from the sample, so a partial outage could otherwise present a handful of surviving 404s as a ~100% miss rate and bulk-reset the catalog. Bulk mode additionally requires a minimum number of successful samples; thinned probes and tiny catalogs take the safe per-row verify path. - Track sweep errors separately from probe/branding errors (stats.sweep_errors) and certify the storage fingerprint only when the sweep completed with zero of them — skipped rows were never verified, so the next startup retries. Applied resets stay durable. - Give each ObjectExists attempt its own timeout so a stalled HEAD fails that attempt instead of pinning the retry loop to the run context. - Report branding assets checked (not just cleared) in stats.Checked. - Drop the dead settingsRepo/brandingSvc nil guards in cmd/silo and sync spec numbers with the implementation constants. Part of #348 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
182 lines
5.9 KiB
Go
182 lines
5.9 KiB
Go
package branding
|
|
|
|
import (
|
|
"context"
|
|
"crypto/sha256"
|
|
"encoding/hex"
|
|
"errors"
|
|
"log/slog"
|
|
"path"
|
|
|
|
"github.com/Silo-Server/silo-server/internal/s3client"
|
|
)
|
|
|
|
// SettingsStore is the subset of the server settings repository the branding
|
|
// service needs.
|
|
type SettingsStore interface {
|
|
Get(ctx context.Context, key string) (string, error)
|
|
Set(ctx context.Context, key, value string) error
|
|
}
|
|
|
|
// AssetStore is the subset of the S3 client used for branding asset bytes.
|
|
type AssetStore interface {
|
|
PutObject(ctx context.Context, bucket, key string, data []byte) error
|
|
GetObject(ctx context.Context, bucket, key string) ([]byte, error)
|
|
Bucket() string
|
|
}
|
|
|
|
// Service is the single source of truth for branding. It assembles a Snapshot
|
|
// from settings, processes/stores uploaded assets, and streams them back.
|
|
//
|
|
// The store is optional: when S3 is not configured it is nil and asset
|
|
// upload/serving returns ErrStorageUnavailable, while text branding (name,
|
|
// subtitle, accent, default theme) keeps working.
|
|
type Service struct {
|
|
settings SettingsStore
|
|
store AssetStore
|
|
}
|
|
|
|
// NewService constructs a branding Service. Pass a nil store when S3 is not
|
|
// configured: text branding keeps working while asset upload/serve returns
|
|
// ErrStorageUnavailable. Callers holding a concrete *s3client.Client must pass
|
|
// a nil AssetStore when that client is nil (rather than the typed-nil pointer)
|
|
// to avoid the typed-nil interface trap — see the construction in cmd/silo.
|
|
func NewService(settings SettingsStore, store AssetStore) *Service {
|
|
return &Service{settings: settings, store: store}
|
|
}
|
|
|
|
// HasStorage reports whether asset upload/serving is available.
|
|
func (s *Service) HasStorage() bool { return s != nil && s.store != nil }
|
|
|
|
// Load reads the current branding configuration. Per-key read errors are
|
|
// tolerated and fall back to defaults so the SPA always renders.
|
|
func (s *Service) Load(ctx context.Context) Snapshot {
|
|
get := func(key string) string {
|
|
v, _ := s.settings.Get(ctx, key)
|
|
return v
|
|
}
|
|
snap := Snapshot{
|
|
ServerName: firstNonEmpty(get(KeyServerName), DefaultServerName),
|
|
LoginSubtitle: firstNonEmpty(get(KeyLoginSubtitle), DefaultLoginSubtitle),
|
|
AccentColor: get(KeyAccentColor),
|
|
DefaultTheme: get(KeyDefaultTheme),
|
|
assets: make(map[AssetKind]string, len(assetSpecs)),
|
|
}
|
|
for kind, spec := range assetSpecs {
|
|
if ref := get(spec.settingKey); ref != "" {
|
|
snap.assets[kind] = ref
|
|
}
|
|
}
|
|
return snap
|
|
}
|
|
|
|
// UploadAsset validates, processes, and stores an uploaded branding image,
|
|
// recording its content ref in settings. It returns the new ref ("<hash><ext>").
|
|
func (s *Service) UploadAsset(ctx context.Context, kind AssetKind, data []byte, declaredType string) (string, error) {
|
|
spec, ok := assetSpecs[kind]
|
|
if !ok {
|
|
return "", ErrInvalidKind
|
|
}
|
|
if !s.HasStorage() {
|
|
return "", ErrStorageUnavailable
|
|
}
|
|
|
|
out, _, ext, err := spec.process(data, declaredType)
|
|
if err != nil {
|
|
return "", err
|
|
}
|
|
|
|
// Content-address by the hash of the *stored* bytes: the ref then truly
|
|
// identifies what is served, so the ?v=<ref> cache-buster changes exactly
|
|
// when the served content changes (and identical outputs dedupe).
|
|
sum := sha256.Sum256(out)
|
|
ref := hex.EncodeToString(sum[:])[:16] + ext
|
|
key := spec.s3Prefix + "/" + ref
|
|
|
|
if err := s.store.PutObject(ctx, s.store.Bucket(), key, out); err != nil {
|
|
return "", err
|
|
}
|
|
if err := s.settings.Set(ctx, spec.settingKey, ref); err != nil {
|
|
return "", err
|
|
}
|
|
return ref, nil
|
|
}
|
|
|
|
// DeleteAsset clears the custom asset of the given kind. The S3 object is left
|
|
// in place (orphaned objects are cheap and avoid concurrent-reader races); the
|
|
// empty settings value is what deactivates it.
|
|
func (s *Service) DeleteAsset(ctx context.Context, kind AssetKind) error {
|
|
spec, ok := assetSpecs[kind]
|
|
if !ok {
|
|
return ErrInvalidKind
|
|
}
|
|
return s.settings.Set(ctx, spec.settingKey, "")
|
|
}
|
|
|
|
// GetAsset fetches the bytes of the current custom asset of the given kind.
|
|
// Returns ErrAssetNotConfigured when none is set, ErrStorageUnavailable when S3
|
|
// is absent, and ErrAssetNotConfigured when the object is missing in S3.
|
|
func (s *Service) GetAsset(ctx context.Context, kind AssetKind) (data []byte, contentType, ref string, err error) {
|
|
spec, ok := assetSpecs[kind]
|
|
if !ok {
|
|
return nil, "", "", ErrInvalidKind
|
|
}
|
|
ref, _ = s.settings.Get(ctx, spec.settingKey)
|
|
if ref == "" {
|
|
return nil, "", "", ErrAssetNotConfigured
|
|
}
|
|
if !s.HasStorage() {
|
|
return nil, "", "", ErrStorageUnavailable
|
|
}
|
|
key := spec.s3Prefix + "/" + ref
|
|
data, err = s.store.GetObject(ctx, s.store.Bucket(), key)
|
|
if err != nil {
|
|
if errors.Is(err, s3client.ErrNotFound) {
|
|
return nil, "", "", ErrAssetNotConfigured
|
|
}
|
|
return nil, "", "", err
|
|
}
|
|
return data, contentTypeForExt(path.Ext(ref)), ref, nil
|
|
}
|
|
|
|
// ReconcileMissingAssets clears the ref of every configured branding asset
|
|
// whose stored object no longer exists (e.g. after the public S3 provider
|
|
// changed without migrating data), so the UI falls back to the built-in
|
|
// defaults instead of serving broken images. Returns how many configured
|
|
// assets were checked and how many of those were cleared.
|
|
func (s *Service) ReconcileMissingAssets(ctx context.Context) (checked, cleared int, err error) {
|
|
if s == nil || !s.HasStorage() {
|
|
return 0, 0, nil
|
|
}
|
|
for kind, spec := range assetSpecs {
|
|
ref, _ := s.settings.Get(ctx, spec.settingKey)
|
|
if ref == "" {
|
|
continue
|
|
}
|
|
checked++
|
|
key := spec.s3Prefix + "/" + ref
|
|
_, getErr := s.store.GetObject(ctx, s.store.Bucket(), key)
|
|
switch {
|
|
case getErr == nil:
|
|
case errors.Is(getErr, s3client.ErrNotFound):
|
|
if setErr := s.settings.Set(ctx, spec.settingKey, ""); setErr != nil {
|
|
return checked, cleared, setErr
|
|
}
|
|
cleared++
|
|
slog.Warn("branding: cleared asset whose stored object is missing", "kind", kind, "key", key)
|
|
default:
|
|
return checked, cleared, getErr
|
|
}
|
|
}
|
|
return checked, cleared, nil
|
|
}
|
|
|
|
func firstNonEmpty(values ...string) string {
|
|
for _, v := range values {
|
|
if v != "" {
|
|
return v
|
|
}
|
|
}
|
|
return ""
|
|
}
|