diff --git a/.env.example b/.env.example index e62ea990..1340227d 100644 --- a/.env.example +++ b/.env.example @@ -1,6 +1,12 @@ # Silo Docker deployment # For the default docker compose setup, MEDIA_ROOT is the main value you need to set. +# Optional: layer compose files through env. Leave unset for CPU-only defaults. +# Linux/macOS example for enabling NVIDIA override: +# COMPOSE_FILE=docker-compose.yml:docker-compose.nvidia.yml +# Windows example: +# COMPOSE_FILE=docker-compose.yml;docker-compose.nvidia.yml + # Required for docker compose: host path to your media library. # Example: MEDIA_ROOT=/srv/media MEDIA_ROOT=/srv/media @@ -20,6 +26,19 @@ SILO_IMAGE=ghcr.io/silo-server/silo-server:latest POSTGRES_USER=silo POSTGRES_PASSWORD=silo POSTGRES_DB=silo +POSTGRES_SHM_SIZE=8gb + +# PostgreSQL auto-tuning is enabled by default in docker-compose.yml. Silo +# applies pgtune-style OLTP recommendations through ALTER SYSTEM at startup. +# Set POSTGRES_TUNE=off if you manage PostgreSQL tuning yourself. +# POSTGRES_TUNE=off +# POSTGRES_TUNE_PROFILE=oltp +# POSTGRES_TUNE_MEMORY=auto +# POSTGRES_TUNE_MEMORY_BUDGET_PERCENT=75 +# POSTGRES_TUNE_CPUS=auto +# POSTGRES_TUNE_STORAGE=ssd +# POSTGRES_TUNE_DB_SIZE=auto +# POSTGRES_TUNE_CONNECTIONS=100 # Optional host port overrides for docker-compose.yml. # PORT=8090 @@ -27,6 +46,9 @@ POSTGRES_DB=silo # PROXY_PORT=8083 # TRANSCODE_PORT=8082 +# Optional NVIDIA override controls (used by docker-compose.nvidia.yml). +# NVIDIA_GPU_COUNT=1 + # Run from source / advanced overrides # Only DATABASE_URL is required when running Silo outside the default docker compose stack. # DATABASE_URL=postgres://silo:password@localhost:5432/silo diff --git a/AGENTS.md b/AGENTS.md index a72adb36..b69cfa06 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,7 +1,7 @@ # Repository Guidelines ## Project Structure & Module Organization -`cmd/silo` contains the main server entrypoint. Backend code lives in `internal/`, organized by domain (`api`, `catalog`, `metadata`, `playback`, `scanner`, `jellycompat`, etc.); keep new code in the package that owns the behavior instead of creating catch-all helpers. Database changes belong in `migrations/` as paired numbered `.up.sql` and `.down.sql` files. The React frontend lives in `web/src/`, with feature code split across `components/`, `pages/`, `hooks/`, `player/`, and `lib/`. Reference material belongs in `docs/architecture/` or `docs/superpowers/{specs,plans}/`; ad hoc SQL helpers live in `scripts/`. +`cmd/silo` contains the main server entrypoint. Backend code lives in `internal/`, organized by domain (`api`, `catalog`, `metadata`, `playback`, `scanner`, `jellycompat`, etc.); keep new code in the package that owns the behavior instead of creating catch-all helpers. Database changes belong in `migrations/sql/` as Goose SQL migrations. Legacy converted migrations intentionally keep their original numeric versions so existing `schema_versions` rows can bootstrap cleanly into Goose. New migrations must use timestamped filenames created with `make migrate-create NAME=add_thing`; do not run `goose fix` or create paired `.up.sql` / `.down.sql` files. The React frontend lives in `web/src/`, with feature code split across `components/`, `pages/`, `hooks/`, `player/`, and `lib/`. Reference material belongs in `docs/architecture/` or `docs/superpowers/{specs,plans}/`; ad hoc SQL helpers live in `scripts/`. When creating or editing `docs/superpowers/specs/` or `docs/superpowers/plans/`, never include local absolute filesystem paths or transient worktree IDs. Use repository-relative paths and wording like "Commands assume the repository root is the cwd." @@ -53,6 +53,7 @@ Use the checked-in `Makefile` for the common paths: - `make dev-frontend`: start the Vite dev server with HMR - `make dev-proxy` / `make dev-transcode`: run standalone worker modes - `make lint`: run `golangci-lint` and frontend ESLint +- `make migrate-status` / `make migrate-up`: inspect or apply Goose migrations through Silo's legacy-safe bootstrapping runner Run before opening a merge request: diff --git a/Dockerfile b/Dockerfile index ff82578e..acfac7ac 100644 --- a/Dockerfile +++ b/Dockerfile @@ -48,13 +48,13 @@ RUN apt-get update && \ echo "deb [signed-by=/usr/share/keyrings/jellyfin.gpg arch=${TARGETARCH}] https://repo.jellyfin.org/debian bookworm main" \ > /etc/apt/sources.list.d/jellyfin.list && \ apt-get update && \ - apt-get install -y --no-install-recommends jellyfin-ffmpeg7 libvips42 && \ + apt-get install -y --no-install-recommends jellyfin-ffmpeg7 libvips42 fonts-noto-core fonts-noto-cjk && \ apt-get purge -y gnupg && apt-get autoremove -y && \ rm -rf /var/lib/apt/lists/* RUN mkdir -p /tmp/silo-transcode COPY --from=build /silo /usr/local/bin/silo COPY third_party/jellyfin-web/ /srv/jellyfin-web/ -EXPOSE 8080 8096 +EXPOSE 8080 8096 13378 HEALTHCHECK --interval=15s --timeout=5s --start-period=10s --retries=3 \ CMD curl -f http://localhost:${PORT:-8080}/api/v1/health || exit 1 ENTRYPOINT ["silo"] diff --git a/Dockerfile.dev b/Dockerfile.dev index dd02ce3b..62848dee 100644 --- a/Dockerfile.dev +++ b/Dockerfile.dev @@ -53,7 +53,7 @@ RUN apt-get update && \ echo "deb [signed-by=/usr/share/keyrings/jellyfin.gpg arch=${TARGETARCH}] https://repo.jellyfin.org/debian bookworm main" \ > /etc/apt/sources.list.d/jellyfin.list && \ apt-get update && \ - apt-get install -y --no-install-recommends jellyfin-ffmpeg7 libvips42 && \ + apt-get install -y --no-install-recommends jellyfin-ffmpeg7 libvips42 fonts-noto-core fonts-noto-cjk && \ apt-get purge -y gnupg && apt-get autoremove -y && \ rm -rf /var/lib/apt/lists/* RUN mkdir -p /tmp/silo-transcode diff --git a/Makefile b/Makefile index 60a2eb4c..907b81d5 100644 --- a/Makefile +++ b/Makefile @@ -1,10 +1,13 @@ -.PHONY: frontend build dev-frontend dev-backend dev-proxy dev-transcode lint clean jellyfin-web-bundle migrate-continuum-check verify-local-paths install-hooks +.PHONY: frontend build dev-frontend dev-backend dev-proxy dev-transcode lint clean jellyfin-web-bundle migrate-continuum-check verify-local-paths install-hooks migrate-create migrate-validate migrate-status migrate-up GIT_COMMON_DIR := $(strip $(shell git rev-parse --git-common-dir 2>/dev/null)) MAIN_CHECKOUT_ROOT := $(if $(GIT_COMMON_DIR),$(abspath $(GIT_COMMON_DIR)/..)) SHARED_MAKEFILE_LOCAL := $(if $(GIT_COMMON_DIR),$(abspath $(GIT_COMMON_DIR)/../Makefile.local)) DEFAULT_PLUGIN_SDK_DIR := $(abspath ../silo-plugin-sdk) SHARED_PLUGIN_SDK_DIR := $(if $(MAIN_CHECKOUT_ROOT),$(abspath $(MAIN_CHECKOUT_ROOT)/../silo-plugin-sdk)) +GOOSE := go run github.com/pressly/goose/v3/cmd/goose@v3.27.1 +GOOSE_DIR := migrations/sql +ENV_FILE ?= .env ifneq ($(wildcard $(DEFAULT_PLUGIN_SDK_DIR)),) DEV_PLUGIN_SDK_DIR ?= $(DEFAULT_PLUGIN_SDK_DIR) @@ -54,6 +57,23 @@ lint: verify-local-paths: scripts/check-local-path-leaks.sh +# Create a timestamped Goose SQL migration. Usage: make migrate-create NAME=add_thing +migrate-create: + @if [ -z "$(NAME)" ]; then echo "usage: make migrate-create NAME=add_thing"; exit 1; fi + $(GOOSE) -dir $(GOOSE_DIR) create "$(NAME)" sql + +# Validate Goose migration annotations and SQL parsing without touching a database. +migrate-validate: + $(GOOSE) -dir $(GOOSE_DIR) validate + +# Show Goose migration status through Silo's bootstrapping runner. +migrate-status: + go run ./cmd/silo/ --env "$(ENV_FILE)" --migrate-status + +# Apply pending Goose migrations through Silo's bootstrapping runner. +migrate-up: + go run ./cmd/silo/ --env "$(ENV_FILE)" --migrate-only + # Install repo-local git hooks for this checkout/worktree. install-hooks: @existing="$$(git config --local core.hooksPath 2>/dev/null || true)"; \ diff --git a/README.md b/README.md index de86f351..fac1af14 100644 --- a/README.md +++ b/README.md @@ -2,6 +2,8 @@ A self-hosted media streaming server with a React frontend and Go backend. Supports direct play, remuxing, and hardware-accelerated transcoding. Compatible with Jellyfin/Emby clients (VidHub, Findroid) via a built-in compatibility layer. +Join the community on [Discord](https://discord.gg/4RxuUQAEnW). + ## Deploy with Docker (recommended) The easiest way to run Silo is with Docker Compose. The default stack assumes you do not already have PostgreSQL and Redis available, so it bundles PostgreSQL, Redis, FFmpeg, and the application for a one-command start. @@ -32,6 +34,29 @@ The easiest way to run Silo is with Docker Compose. The default stack assumes yo If you already have PostgreSQL and Redis available, omit those bundled service examples from compose and point Silo at your existing `DATABASE_URL` and `REDIS_URL` instead. + ### Optional NVIDIA/NVENC + + GPU support is kept out of the default compose file so hosts without NVIDIA drivers work unchanged. + + Install the NVIDIA Container Toolkit and use a Docker Compose version with GPU reservation support before enabling this override. + + Use the optional override file when you want NVENC: + + ```sh + docker compose -f docker-compose.yml -f docker-compose.nvidia.yml up -d + ``` + + If you want this controlled from `.env`, set `COMPOSE_FILE`: + + ```dotenv + COMPOSE_FILE=docker-compose.yml:docker-compose.nvidia.yml + NVIDIA_GPU_COUNT=1 + ``` + + Windows uses `;` instead of `:` between compose files. + + Then `docker compose up -d` will include the NVIDIA override automatically. + 4. **Configure through the admin UI** Add libraries, users, metadata providers, and playback settings from the web interface. @@ -176,8 +201,36 @@ Technical notes: | `make dev-backend` | Run Go backend (integrated mode) | | `make dev-proxy` | Run a standalone proxy node | | `make dev-transcode` | Run a standalone transcode node | +| `make migrate-create NAME=add_thing` | Create a timestamped Goose SQL migration | +| `make migrate-validate` | Validate Goose migration files without touching a database | +| `make migrate-status` | Show Goose migration status using Silo's bootstrapping runner | +| `make migrate-up` | Apply pending Goose migrations using Silo's bootstrapping runner | | `make clean` | Remove build artifacts | +### Database Migrations + +PostgreSQL schema migrations are managed by Goose. Migration SQL files live in +`migrations/sql/` and use Goose annotations. Converted legacy migrations keep +their original numeric versions so existing `schema_versions` rows can bootstrap +cleanly into Goose without replaying old SQL. New migrations should be created +with timestamped filenames: + +```sh +make migrate-create NAME=add_thing +make migrate-validate +``` + +Do not run `goose fix`; timestamped migrations are the repository policy because +they avoid version collisions across parallel PRs. The existing `001`-style +files are historical compatibility records, not the naming pattern for new work. +Runtime migrations are applied by the integrated/API server only. Proxy and +transcode modes never mutate schema. +For existing installs, use `make migrate-status` and `make migrate-up` rather +than invoking the Goose CLI directly; those targets copy legacy +`schema_versions` rows into `public.goose_db_version` under the migration lock +before reading or applying migrations. Set `ENV_FILE=path/to/.env` when the +database URL should be read from a non-default env file. + ### Running Tests ```sh @@ -216,93 +269,57 @@ Silo requires only a `DATABASE_URL` when running from source or against external | `proxy` | Stream proxy node that connects to the shared deployment database and Redis | | `transcode` | HLS transcode worker node that connects to the shared deployment database and Redis | -## PostgreSQL Performance Tuning +## PostgreSQL Auto-Tuning -The default PostgreSQL configuration is conservative and not optimized for any particular workload. For better performance, use [PGTune](https://pgtune.leopard.in.ua/) to generate settings tailored to your hardware. - -**Recommended PGTune settings:** - -| Setting | Value | -|---|---| -| DB Type | **Web application** | -| Total Memory | Your server's RAM | -| Number of CPUs | Your server's CPU count | -| Number of Connections | 20 (Silo's default pool size) | -| Data Storage | SSD or HDD depending on your setup | - -Silo's workload is predominantly read-heavy API queries (browse, search, metadata lookups) with the database small enough to fit in RAM for most libraries. The **Web application** profile optimizes for this: CPU-bound simple queries served from cached pages, high concurrent reads, and low write volume. - -If you have a very large library (50,000+ items) or heavy activity logging, consider switching to the **OLTP** profile instead. - -### Recommended Defaults for Modern Hardware - -Assuming you're using modern hardware with fast NVMe storage this is what I'm currently running. - -```ini -# Connections -max_connections = 100 - -# Memory (tuned to keep PG under ~8GB total) -shared_buffers = 4GB # holds the entire DB in cache (typical DB is 1-4GB) -effective_cache_size = 6GB # planner hint only, not an allocation -work_mem = 32MB # per-operation sort/hash memory -maintenance_work_mem = 512MB # VACUUM, CREATE INDEX, catalog imports - -# WAL / Write Performance -wal_buffers = 64MB -min_wal_size = 512MB -max_wal_size = 2GB -checkpoint_completion_target = 0.9 - -# Storage (NVMe/Optane) -random_page_cost = 1.1 # default 4.0; NVMe random ≈ sequential -effective_io_concurrency = 200 # NVMe handles many concurrent requests - -# Parallelism -max_worker_processes = 8 -max_parallel_workers_per_gather = 4 -max_parallel_workers = 8 -max_parallel_maintenance_workers = 4 - -# Huge Pages -huge_pages = try -``` - -These settings keep PostgreSQL under ~8GB of RAM while still caching the entire database. For HDD storage, change `random_page_cost` to `4.0` and `effective_io_concurrency` to `2`. - -### Applying Settings - -**Docker (recommended):** Save the settings above to a `postgresql.conf` file and mount it into your container: +The default Docker Compose stack does not require a checked-in `postgresql.conf`. +It enables Silo's [pgtune](https://github.com/le0pard/pgtune)-style OLTP tuning +by default: ```yaml -# docker-compose.yml -postgres: - image: pgvector/pgvector:pg18 - volumes: - - ./postgresql.conf:/etc/postgresql/postgresql.conf:ro - command: postgres -c config_file=/etc/postgresql/postgresql.conf +POSTGRES_TUNE: auto ``` -Then restart: `docker compose restart postgres` +When enabled, Silo connects with `DATABASE_URL` and applies recommendations with +`ALTER SYSTEM`, which writes to PostgreSQL's `postgresql.auto.conf` inside the +database data directory. Reloadable settings are applied immediately with +`pg_reload_conf()`. Settings that PostgreSQL marks as restart-only are written +too, and Silo logs the setting names so you can restart PostgreSQL once: -**Existing PostgreSQL:** You can apply settings without editing config files using `ALTER SYSTEM`, which writes to `postgresql.auto.conf` inside the data directory: - -```sql -ALTER SYSTEM SET work_mem = '32MB'; -ALTER SYSTEM SET effective_cache_size = '6GB'; -ALTER SYSTEM SET random_page_cost = 1.1; -ALTER SYSTEM SET effective_io_concurrency = 200; --- Apply without restart: -SELECT pg_reload_conf(); - --- These require a full restart: -ALTER SYSTEM SET shared_buffers = '4GB'; -ALTER SYSTEM SET max_connections = 100; -ALTER SYSTEM SET max_worker_processes = 8; -ALTER SYSTEM SET huge_pages = 'try'; +```sh +docker compose restart postgres ``` -Settings like `shared_buffers`, `max_connections`, `max_worker_processes`, and `huge_pages` require a PostgreSQL restart. Memory/planner settings (`work_mem`, `effective_cache_size`, `random_page_cost`) can be reloaded live. +The default Compose database user has the required PostgreSQL permissions. If +you use an external PostgreSQL server, make sure the configured `DATABASE_URL` +user can run `ALTER SYSTEM`, or set `POSTGRES_TUNE=off` and manage +PostgreSQL yourself. + +For `POSTGRES_TUNE_MEMORY=auto`, Silo uses the first trustworthy memory source: +a finite Docker cgroup limit, the read-only `/host/proc/meminfo` mount supplied +by the bundled Compose file, then `/proc/meminfo` with container safety guards. +Auto-detected memory is treated as a PostgreSQL budget, defaulting to 75% of +detected RAM so Silo, Redis, plugins, transcodes, and the OS retain headroom. +`POSTGRES_TUNE_DB_SIZE=auto` queries `pg_database_size(current_database())` and +classifies the workload by comparing the database size to that memory budget. + +Optional tuning overrides: + +| Variable | Default | Description | +|---|---:|---| +| `POSTGRES_TUNE_PROFILE` | `oltp` | Tuning profile. Only `oltp` is currently supported. | +| `POSTGRES_TUNE_MEMORY` | `auto` | Server/container RAM, such as `8GB` or `32GB`; explicit values are used as-is. | +| `POSTGRES_TUNE_MEMORY_BUDGET_PERCENT` | `75` | Percent of auto-detected RAM used for PostgreSQL recommendations. | +| `POSTGRES_TUNE_CPUS` | `auto` | CPU count used for worker recommendations. | +| `POSTGRES_TUNE_STORAGE` | `ssd` | One of `hdd`, `ssd`, `san`, or `nvme`. | +| `POSTGRES_TUNE_DB_SIZE` | `auto` | Use `less_ram` when the database comfortably fits in RAM, `mid_ram`, or `greater_ram` for very large databases. | +| `POSTGRES_TUNE_CONNECTIONS` | `100` | PostgreSQL `max_connections`; automatically raised if Silo's app pool is configured higher. | +| `POSTGRES_SHM_SIZE` | `8gb` | Docker `/dev/shm` size for the bundled PostgreSQL container. | + +Advanced operators can still supply their own PostgreSQL configuration or +override these env vars. Set `POSTGRES_TUNE=off` when you do not want Silo to +change PostgreSQL server settings. Settings already written with `ALTER SYSTEM` +remain in `postgresql.auto.conf`; reset those PostgreSQL parameters if you later +move fully to a custom `postgresql.conf`. ## Project Structure @@ -319,5 +336,5 @@ internal/ scanner/ Media file discovery and FFProbe worker/ Background jobs (scan, match, reconcile) web/ React + TypeScript frontend (Vite, Tailwind, shadcn/ui) -migrations/ PostgreSQL schema migrations +migrations/sql/ Goose-managed PostgreSQL schema migrations ``` diff --git a/cmd/silo/main.go b/cmd/silo/main.go index 907145fb..f4cdb480 100644 --- a/cmd/silo/main.go +++ b/cmd/silo/main.go @@ -4,8 +4,6 @@ import ( "context" "crypto/rand" "encoding/base64" - "encoding/json" - "errors" "flag" "fmt" "io/fs" @@ -16,24 +14,32 @@ import ( "os/signal" "path/filepath" "runtime/debug" + "sort" "strconv" "strings" + "sync/atomic" "syscall" "time" + "github.com/go-chi/chi/v5" + chimiddleware "github.com/go-chi/chi/v5/middleware" "github.com/google/uuid" "github.com/hashicorp/go-hclog" - "github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5/pgxpool" "github.com/prometheus/client_golang/prometheus/promhttp" pluginv1 "github.com/Silo-Server/silo-plugin-sdk/pkg/pluginproto/silo/plugin/v1" + sdkcapability "github.com/Silo-Server/silo-plugin-sdk/pkg/pluginsdk/capability" + "github.com/Silo-Server/silo-server/internal/access" "github.com/Silo-Server/silo-server/internal/activitylog" "github.com/Silo-Server/silo-server/internal/adminjob" "github.com/Silo-Server/silo-server/internal/api" "github.com/Silo-Server/silo-server/internal/api/handlers" + "github.com/Silo-Server/silo-server/internal/audiobooks" + "github.com/Silo-Server/silo-server/internal/audiobooks/podcastfeed" "github.com/Silo-Server/silo-server/internal/auth" + "github.com/Silo-Server/silo-server/internal/autoscan" "github.com/Silo-Server/silo-server/internal/cache" "github.com/Silo-Server/silo-server/internal/catalog" "github.com/Silo-Server/silo-server/internal/catalogseed" @@ -50,7 +56,6 @@ import ( "github.com/Silo-Server/silo-server/internal/logfilter" "github.com/Silo-Server/silo-server/internal/logstream" "github.com/Silo-Server/silo-server/internal/markers" - "github.com/Silo-Server/silo-server/internal/markers/introdb" "github.com/Silo-Server/silo-server/internal/mdblist" "github.com/Silo-Server/silo-server/internal/metadata" "github.com/Silo-Server/silo-server/internal/models" @@ -183,8 +188,74 @@ func configureOperationalLogging( return operationalWriter, opslog.NewRepo(pool), opsPM } +func maybeApplyPostgresTuning(ctx context.Context, pool *pgxpool.Pool, appMaxConnections int, mode string) { + switch strings.ToLower(strings.TrimSpace(mode)) { + case "", "integrated", "api": + default: + return + } + + opts, err := database.LoadPostgresTuneOptionsFromEnv(appMaxConnections) + if err != nil { + slog.Warn("postgres auto-tuning disabled", "error", err) + return + } + if !opts.Enabled { + return + } + + tuneCtx, cancel := context.WithTimeout(ctx, 30*time.Second) + defer cancel() + + result, err := database.ApplyPostgresTuning(tuneCtx, pool, opts) + for _, failure := range result.Failures { + slog.Warn("postgres auto-tuning setting failed", + "name", failure.Name, + "value", failure.Value, + "error", failure.Err, + ) + } + if err != nil { + slog.Warn("postgres auto-tuning failed", + "error", err, + "applied", result.Applied, + "failures", len(result.Failures), + ) + return + } + + slog.Info("postgres auto-tuning applied", + "profile", opts.Profile, + "postgres_major", result.PostgresMajorVersion, + "settings", result.Applied, + "resets", len(result.Reset), + "failures", len(result.Failures), + "memory_budget_bytes", opts.MemoryBudgetBytes, + "detected_memory_bytes", opts.DetectedMemoryBytes, + "memory_source", opts.MemorySource, + "memory_budget_percent", opts.MemoryBudgetPercent, + "cpus", opts.CPUs, + "connections", opts.Connections, + "storage", opts.Storage, + "db_size", result.DBSize, + "database_size_bytes", result.DatabaseSizeBytes, + ) + if len(result.RestartRequired) > 0 { + slog.Warn("postgres restart required to finish applying auto-tuned settings", + "settings", strings.Join(result.RestartRequired, ","), + ) + } + if len(result.Reset) > 0 { + slog.Info("postgres auto-tuning reset stale settings", + "settings", strings.Join(result.Reset, ","), + ) + } +} + func main() { envFile := flag.String("env", ".env", "path to .env bootstrap file") + migrateOnly := flag.Bool("migrate-only", false, "apply database migrations and exit") + migrateStatus := flag.Bool("migrate-status", false, "show database migration status and exit") flag.Parse() ctx := context.Background() @@ -204,12 +275,48 @@ func main() { defer pool.Close() slog.Info("connected to PostgreSQL") + if *migrateStatus { + migCtx, migCancel := context.WithTimeout(ctx, 5*time.Minute) + statuses, statusErr := database.MigrationStatuses(migCtx, pool, migrations.FS, "sql") + migCancel() + if statusErr != nil { + log.Fatalf("failed to read migration status: %v", statusErr) + } + + fmt.Printf("%-8s %8s %-25s %s\n", "STATE", "VERSION", "APPLIED_AT", "MIGRATION") + for _, status := range statuses { + appliedAt := "-" + if !status.AppliedAt.IsZero() { + appliedAt = status.AppliedAt.UTC().Format(time.RFC3339) + } + source := status.Source + if source != "" { + source = filepath.Base(source) + } else { + source = "-" + } + fmt.Printf("%-8s %8d %-25s %s\n", status.State, status.Version, appliedAt, source) + } + return + } + + if *migrateOnly { + migCtx, migCancel := context.WithTimeout(ctx, 5*time.Minute) + migErr := database.RunMigrations(migCtx, pool, migrations.FS, "sql") + migCancel() + if migErr != nil { + log.Fatalf("failed to run migrations: %v", migErr) + } + slog.Info("database migrations applied") + return + } + // Run migrations only for integrated/api modes. Proxy and transcode nodes // should never alter the schema — they may scale independently and would // race or apply migrations before the primary node is deliberately upgraded. if bc.Mode == "integrated" || bc.Mode == "api" || bc.Mode == "" { migCtx, migCancel := context.WithTimeout(ctx, 5*time.Minute) - if migErr := database.RunMigrations(migCtx, pool, migrations.FS, "."); migErr != nil { + if migErr := database.RunMigrations(migCtx, pool, migrations.FS, "sql"); migErr != nil { migCancel() log.Fatalf("failed to run migrations: %v", migErr) } @@ -313,10 +420,13 @@ func main() { slog.SetDefault(slog.New(logfilter.New(baseHandler, cfg.Server.LogQuiet))) mode := cfg.Server.Mode + maybeApplyPostgresTuning(ctx, pool, cfg.Database.MaxConnections, mode) slog.Info("silo starting", "mode", mode, "listen", cfg.Server.Listen, "log_level", cfg.Server.LogLevel, "node_id", nodeID) appCtx, appCancel := context.WithCancel(ctx) defer appCancel() + restartReqCh := make(chan struct{}, 1) + var restartRequested atomic.Bool eventBus := cache.NewEventBus(cfg.Redis.URL) logStreamHub := logstream.NewHub(nodeID, eventBus) @@ -417,7 +527,22 @@ func main() { OpsLogRepo: opsRepo, FFmpegLogSink: playback.NewSlogFFmpegLogSink(slog.Default(), nodeID), PublicURL: os.Getenv("SILO_PUBLIC_URL"), + RequestServerRestart: func(context.Context) error { + if !restartRequested.CompareAndSwap(false, true) { + return handlers.ErrServerRestartAlreadyRequested + } + restartReqCh <- struct{}{} + return nil + }, } + audiobooksService := audiobooks.New(&audiobooksSettingsAdapter{repo: settingsRepo}) + absCompatEnabled, err := audiobooksService.ABSCompatEnabled(appCtx) + if err != nil { + slog.Warn("Audiobookshelf compatibility disabled; failed to read setting", "err", err) + absCompatEnabled = false + } + adminJobCancelRegistry := adminjob.NewCancelRegistry() + deps.AdminJobCancelRegistry = adminJobCancelRegistry if needsWorkers && deps.DB != nil { deps.IntroRepository = intromarkers.NewRepository(deps.DB) deps.IntroAnalyzer = intromarkers.NewAnalyzer( @@ -428,36 +553,34 @@ func main() { } if deps.DB != nil { markerRegistry := markers.NewRegistry(slog.Default()) - introdbAPIKey, err := settingsRepo.Get(appCtx, "introdb.api_key") - if err != nil && !errors.Is(err, pgx.ErrNoRows) { - slog.Warn("load introdb.api_key from settings failed; provider will run with no key", + markerProviderConfig := markers.NewProviderConfigStore(deps.DB) + if err := markerProviderConfig.Reload(appCtx); err != nil { + slog.Warn("load marker provider config failed; falling back to registration-order fetch", "error", err) - introdbAPIKey = "" - } - introdbClient := introdb.NewClient(introdbAPIKey) - if err := markerRegistry.Register(introdb.NewProvider(introdbClient)); err != nil { - log.Fatalf("register introdb marker provider: %v", err) - } - deps.OnServerSettingUpdated = func(_ context.Context, key, value string) { - if key == "introdb.api_key" { - introdbClient.SetAPIKey(value) + } else { + markerRegistry.UseConfigStore(markerProviderConfig) + if deps.EventBus != nil { + if err := deps.EventBus.Subscribe(appCtx, cache.ChannelAdmin, func(event cache.Event) { + if event.Type != cache.EventMarkerProviderConfigChanged { + return + } + if err := markerProviderConfig.Reload(appCtx); err != nil { + slog.Warn("reload marker provider config failed", "provider", event.Payload, "error", err) + } + }); err != nil { + slog.Warn("subscribe marker provider config reload failed", "error", err) + } } } - if eventBus != nil { - _ = eventBus.Subscribe(appCtx, cache.ChannelAdmin, func(event cache.Event) { - if event.Type != cache.EventSettingsChanged || event.Payload != "introdb.api_key" { - return - } - value, loadErr := settingsRepo.Get(context.Background(), "introdb.api_key") - if loadErr != nil { - slog.Warn("introdb api key reload failed", "error", loadErr) - return - } - introdbClient.SetAPIKey(value) - }) - } + deps.MarkerProviderConfig = markerProviderConfig deps.MarkerRegistry = markerRegistry - deps.MarkerResolver = markers.NewDBExternalIDResolver(deps.DB) + markerResolver := markers.NewDBExternalIDResolver(deps.DB) + deps.MarkerResolver = markerResolver + markerContributionStore := markers.NewContributionStore(deps.DB) + deps.MarkerContributionStore = markerContributionStore + deps.MarkerContributionService = markers.NewContributionService( + markerRegistry, markerResolver, markerProviderConfig, markerContributionStore, slog.Default(), + ) } var watchProviderService *watchsync.Service if deps.DB != nil { @@ -669,6 +792,22 @@ func main() { installer, plugins.NewHostAdapter(pluginHost), ) + if deps.MarkerRegistry != nil && deps.MarkerProviderConfig != nil { + markerPluginResolver := markers.NewPluginResolverAdapter(pluginService) + pluginService.AddLifecycleHook(func(ctx context.Context) { + if err := reloadMarkerPluginProviders( + ctx, + deps.MarkerRegistry, + deps.MarkerProviderConfig, + installationStore, + runtimeConfigStore, + settingsRepo, + markerPluginResolver, + ); err != nil { + slog.Warn("reload marker plugin providers failed", "error", err) + } + }) + } if err := pluginService.PreloadEnabled(appCtx); err != nil { log.Fatalf("preload enabled plugins: %v", err) } @@ -748,6 +887,7 @@ func main() { var groupClaimRepo *catalog.GroupClaimRepository var seasonRepo *catalog.SeasonRepository var episodeRepo *catalog.EpisodeRepository + var audiobookEnricher *audiobooks.Enricher if needsWorkers && deps.DB != nil && deps.FileRepo != nil { chainRepo := metadata.NewChainRepository(deps.DB) skippedRootRepo = metadata.NewSkippedRootRepository(deps.DB) @@ -803,6 +943,8 @@ func main() { providerIDRepo := catalog.NewProviderIDRepository(deps.DB) movieQueueRepo = metadata.NewMovieMatchQueueRepository(deps.DB, deps.FileRepo) seriesQueueRepo = metadata.NewSeriesRootMatchQueueRepository(deps.DB) + deps.MovieMatchQueueRepo = movieQueueRepo + deps.SeriesRootMatchQueueRepo = seriesQueueRepo matchQueueCoordinator = metadata.NewMatchQueueCoordinator(movieQueueRepo, seriesQueueRepo) rootClaimRepo = catalog.NewRootClaimRepository(deps.DB) groupClaimRepo = catalog.NewGroupClaimRepository(deps.DB) @@ -816,6 +958,18 @@ func main() { personRefreshService = metadata.NewPersonRefreshService(deps.DB, pluginResolver, personRepo) personRefreshService.SetImageResolver(imageResolver) + // Wire the audiobook enricher. It uses the same plugin resolver and chain + // repo as the movie/TV pipeline, but resolves providers at + // content_level='audiobook' and sweeps items directly rather than via a queue. + audiobookEnricher = audiobooks.NewEnricher( + deps.DB, + chainRepo, + pluginResolver, + itemRepo, + personRepo, + providerIDRepo, + ) + // Always wire the image resolver so plugin-prefixed URLs (e.g. // metadb://) can be resolved to presigned HTTP URLs in API responses. metadataService.SetImageResolver(imageResolver) @@ -826,13 +980,21 @@ func main() { imageCacher := imagecache.New(deps.S3Public) metadataService.SetImageCacher(imageCacher) metadataService.SetAutoCacheImages(cfg.Metadata.CacheImages) + if deps.Scanner != nil { + deps.Scanner.SetImageCacher(imageCacher) + } if cfg.Metadata.CacheImages { personRefreshService.SetImageCacher(imageCacher) slog.Info("metadata image caching enabled") } + if audiobookEnricher != nil { + audiobookEnricher.SetImageCacher(imageCacher) + audiobookEnricher.SetFFmpegPath(scanner.FFmpegPathFromFFprobe(scanner.FFprobePathFromFFmpeg(cfg.Playback.FFmpegPath))) + } } matchWorker = metadata.NewMatchWorker(metadataService, deps.FileRepo, cfg.Matcher.Workers, cfg.Matcher.BatchSize, 30*time.Second) + matchWorker.SetRealtimeHub(deps.RealtimeHub) if movieQueueRepo != nil { matchWorker.SetMovieFileClaimer(movieQueueRepo) } @@ -1281,9 +1443,15 @@ func main() { if deps.FolderRepo != nil && deps.LibraryScanQueue != nil { taskMgr.Register(tasks.NewScanLibrariesTask(deps.FolderRepo, deps.LibraryScanQueue, deps.EventBus)) } + taskMgr.Register(tasks.NewCleanupOrphanedMediaItemsTask(catalog.NewOrphanedProvisionalCleaner(deps.DB))) if deps.IntroAnalyzer != nil { taskMgr.Register(tasks.NewDetectIntroMarkersTask(deps.IntroAnalyzer, settingsRepo)) } + if deps.MarkerContributionService != nil && deps.MarkerProviderConfig != nil && deps.MarkerContributionStore != nil && deps.FileRepo != nil { + taskMgr.Register(tasks.NewContributeMarkersTask( + deps.MarkerContributionService, deps.MarkerProviderConfig, deps.MarkerContributionStore, deps.FileRepo, + )) + } if chapterBackfiller, ok := deps.ChapterThumbnailQueuer.(*chapterthumbs.Service); ok { taskMgr.Register(tasks.NewChapterThumbnailBackfillTask(chapterBackfiller, 25)) } @@ -1320,13 +1488,52 @@ func main() { ) requestReconcileSvc.SetSecretResolver(settingsRepo) requestReconcileSvc.SetFulfillmentAdapters(radarr.NewClient(nil), sonarr.NewClient(nil)) + if userStoreProvider != nil { + reconcileResolver := access.NewResolver( + auth.NewUserRepository(deps.DB), + userStoreProvider, + access.NewProfileTokenService(cfg.Auth.JWTSecret, 0), + ) + requestReconcileSvc.SetEntitlementResolver(mediarequests.NewAccessEntitlements(reconcileResolver)) + } taskMgr.Register(tasks.NewReconcileRequestsTask(requestReconcileSvc, 100)) + if deps.FolderRepo != nil && deps.LibraryScanQueue != nil && pluginService != nil && pluginInstallationStore != nil { + autoscanRepo := autoscan.NewRepository(deps.DB) + if err := autoscanRepo.MarkInterruptedEvents(appCtx); err != nil { + slog.Warn("autoscan: failed to mark interrupted polls", "err", err) + } + autoscanSvc := api.BuildAutoscanService( + autoscanRepo, + pluginService, + pluginInstallationStore, + mediarequests.NewRepository(deps.DB), + settingsRepo, + deps.FolderRepo, + deps.LibraryScanQueue, + deps.RedisClient, + ) + // The poll task's default interval seeds the schedule from the stored + // settings (DefaultPollIntervalSeconds); per-cycle gating still runs + // off the live settings inside PollOnce. Seed in MILLISECONDS as + // seconds*1000 — the SAME computation HandleUpdateSettings uses to + // reschedule — so startup and reschedule agree for sub-minute and + // non-60-multiple intervals (the old seconds/60 minutes path diverged). + var intervalMs int64 = 10 * 60 * 1000 + if settings, serr := autoscanRepo.GetSettings(appCtx); serr == nil && settings.DefaultPollIntervalSeconds > 0 { + intervalMs = int64(settings.DefaultPollIntervalSeconds) * 1000 + } + taskMgr.Register(tasks.NewAutoscanPollTask(autoscanSvc, intervalMs)) + } reconcileProviderIDRepo := catalog.NewProviderIDRepository(deps.DB) reconcileEpisodeRepo := catalog.NewEpisodeRepository(deps.DB) historyResolver := watchstate.NewStableIdentityResolver(nil, reconcileEpisodeRepo, reconcileProviderIDRepo) historyReconciler := watchstate.NewHistoryReconciler(deps.DB, historyResolver) taskMgr.Register(tasks.NewRepairProviderIDIntegrityTask(metadata.NewProviderIDIntegrityRepairer(deps.DB), historyReconciler)) taskMgr.Register(tasks.NewReconcileWatchHistoryTask(historyReconciler)) + taskMgr.Register(tasks.NewSyncPodcastFeedsTask(podcastfeed.New(), podcastfeed.NewDBStore(deps.DB))) + if audiobookEnricher != nil { + taskMgr.Register(tasks.NewSyncAudiobookMetadataTask(audiobookEnricher)) + } if pluginInstallationStore != nil && pluginRuntimeConfigStore != nil && pluginService != nil { pluginTasks, err := plugins.NewTaskRegistryWithTypedResolver(pluginInstallationStore, pluginRuntimeConfigStore, pluginService).Tasks(appCtx) if err != nil { @@ -1343,6 +1550,57 @@ func main() { slog.Info("task manager started") } + // Build the ABS-compatible REST + Socket.io handler when a DB pool is + // available. Routes are mounted at the root level by NewRouter (not under + // /api/v1/) so ABS clients resolve /login, /api/*, /abs/api/*, and + // /abs/socket.io/* without path prefix hacks. + if absCompatEnabled && deps.DB != nil { + absUserRepo := auth.NewUserRepository(deps.DB) + absSessionRepo := auth.NewSessionRepository(deps.DB) + absJWTService := auth.NewJWTService( + cfg.Auth.JWTSecret, + cfg.Auth.AccessTokenExpiry, + cfg.Auth.RefreshTokenExpiry, + ) + absAuthSvc := auth.NewService( + auth.NewLocalProvider(absUserRepo, absSessionRepo), + absJWTService, + absSessionRepo, + absUserRepo, + nil, // invite codes: not needed for ABS compat + nil, // settings: not needed here + nil, // user store: not needed here + ) + absItemRepo := catalog.NewItemRepository(deps.DB) + absEpisodeRepo := catalog.NewEpisodeRepository(deps.DB) + absSeasonRepo := catalog.NewSeasonRepository(deps.DB) + absPersonRepo := catalog.NewPersonRepository(deps.DB) + var absFileFetcher catalog.FileVersionFetcher + if deps.FileRepo != nil { + absFileFetcher = deps.FileRepo + } + absDetailSvc := catalog.NewDetailService(absItemRepo, absEpisodeRepo, absSeasonRepo, absPersonRepo, absFileFetcher) + if deps.ImageResolver != nil { + absDetailSvc.SetImageResolver(deps.ImageResolver) + } + absHDeps := audiobooks.ABSHandlerDeps{ + Pool: deps.DB, + Items: absItemRepo, + Files: deps.FileRepo, + Settings: settingsRepo, + Auth: &audiobooks.SiloCredValidator{ + Auth: absAuthSvc, + Pool: deps.DB, + }, + AccessResolver: audiobooks.NewABSAccessResolver(absUserRepo, userStoreProvider), + Recs: recommendations.NewRepo(deps.DB), + Detail: absDetailSvc, + } + absH := audiobooksService.BuildABSHandler(absHDeps) + deps.ABSHandler = absH + } + _ = audiobooksService + if deps.DB != nil && pluginInstallationStore != nil && pluginRuntimeConfigStore != nil && deps.PluginService != nil { userRepo := auth.NewUserRepository(deps.DB) sessionRepo := auth.NewSessionRepository(deps.DB) @@ -1464,6 +1722,11 @@ func main() { metricsMux := http.NewServeMux() metricsMux.Handle("/metrics", promhttp.Handler()) metricsMux.Handle("/api/", router) + // ABS-compat is NOT mounted on the main listener — see the "ABS compat + // listener" block below. It binds its own port so the discovery probes + // (/ping, /healthcheck, /status, /init, /login, /socket.io) own the URL + // space without collision with silo's SPA fallback. Mirrors how the + // Jellyfin compat server is set up at :8096. metricsMux.Handle("/", server.FrontendHandler()) // Step 9: Start background workers (if needed). @@ -1530,6 +1793,7 @@ func main() { templateBundleApplyExecutor, deps.RealtimeHub, ) + adminJobRunner.SetCancelRegistry(adminJobCancelRegistry) adminJobRunner.Start() defer adminJobRunner.Stop() @@ -1632,47 +1896,17 @@ func main() { provider := auth.NewLocalProvider(userRepo, sessionRepo) compatDeps.AuthService = auth.NewService(provider, jwtService, sessionRepo, userRepo, nil, nil, nil) - // Access filter resolver for profile-scoped library access. - compatDeps.AccessFilterFn = func(ctx context.Context, userID int, profileID string) catalog.AccessFilter { - if userStoreProvider == nil { - return catalog.AccessFilter{} - } - store, err := userStoreProvider.ForUser(ctx, userID) - if err != nil { - return catalog.AccessFilter{} - } - profile, err := store.GetProfile(ctx, profileID) - if err != nil || profile == nil { - return catalog.AccessFilter{} - } - filter := catalog.AccessFilter{ - MaxContentRating: profile.MaxContentRating, - } - if profile.LibraryRestrictionsEnabled && len(profile.AllowedLibraryIDs) > 0 { - filter.AllowedLibraryIDs = profile.AllowedLibraryIDs - } - // Apply user-disabled library IDs. - if raw, err := store.GetSetting(ctx, "disabled_library_ids"); err == nil && raw != "" { - var disabled []int - if json.Unmarshal([]byte(raw), &disabled) == nil && len(disabled) > 0 { - if filter.AllowedLibraryIDs != nil { - filtered := make([]int, 0, len(filter.AllowedLibraryIDs)) - disSet := make(map[int]struct{}, len(disabled)) - for _, id := range disabled { - disSet[id] = struct{}{} - } - for _, id := range filter.AllowedLibraryIDs { - if _, ok := disSet[id]; !ok { - filtered = append(filtered, id) - } - } - filter.AllowedLibraryIDs = filtered - } else { - filter.DisabledLibraryIDs = disabled - } - } - } - return filter + // Access filter resolver for viewer-scoped library access. + // Backed by the shared access.Resolver so account-level library + // restrictions (users.library_ids), profile restrictions, + // user-disabled libraries, and rating/quality ceilings apply to + // the compat API exactly as they do to the native API. + if userStoreProvider != nil { + compatDeps.AccessFilterFn = jellycompat.NewScopeAccessFilter(access.NewResolver( + userRepo, + userStoreProvider, + nil, // profile tokens unused: compat login already verifies PINs + )) } } @@ -1685,6 +1919,28 @@ func main() { compatSrv.IdleTimeout = 120 * time.Second } + // ABS-compat listener — dedicated http.Server bound to its own port + // (default :13378) that hosts the Audiobookshelf-compatible API. + // Mirrors the Jellyfin compat layout above. The ABS handler mounts + // onto a fresh chi router here so /ping, /healthcheck, /status, /login, + // /socket.io, etc. own the URL space at the root — no SPA fallback, + // no collision with silo's /api/v1. + var absSrv *http.Server + if (mode == "integrated" || mode == "api") && deps.ABSHandler != nil && cfg.AudiobookshelfCompat.Listen != "" { + absRouter := chi.NewRouter() + absRouter.Use(chimiddleware.Recoverer) + absRouter.Use(chimiddleware.Compress(5)) + deps.ABSHandler.Mount(absRouter) + absSrv = &http.Server{ + Addr: cfg.AudiobookshelfCompat.Listen, + Handler: absRouter, + ReadHeaderTimeout: 10 * time.Second, + ReadTimeout: 60 * time.Second, + WriteTimeout: 0, + IdleTimeout: 120 * time.Second, + } + } + // Run non-critical startup work in the background so it doesn't delay the // HTTP listener from accepting connections. Steps run sequentially and stop // early if the app context is cancelled (shutdown). @@ -1709,7 +1965,7 @@ func main() { }() } - errCh := make(chan error, 2) + errCh := make(chan error, 3) go func() { slog.Info("HTTP server listening", "addr", cfg.Server.Listen) if listenErr := srv.ListenAndServe(); listenErr != nil && listenErr != http.ErrServerClosed { @@ -1724,6 +1980,14 @@ func main() { } }() } + if absSrv != nil { + go func() { + slog.Info("ABS compat server listening", "addr", absSrv.Addr) + if listenErr := absSrv.ListenAndServe(); listenErr != nil && listenErr != http.ErrServerClosed { + errCh <- fmt.Errorf("abs compat server error: %w", listenErr) + } + }() + } // Step 11: Wait for termination signal. sigCh := make(chan os.Signal, 1) @@ -1734,6 +1998,9 @@ func main() { case sig := <-sigCh: appCancel() slog.Info("received signal, shutting down", "signal", sig) + case <-restartReqCh: + appCancel() + slog.Info("server restart requested, shutting down") case serverErr := <-errCh: appCancel() slog.Error("server error, shutting down", "error", serverErr) @@ -1753,6 +2020,11 @@ func main() { slog.Error("jellyfin compat shutdown error", "error", shutdownErr) } } + if absSrv != nil { + if shutdownErr := absSrv.Shutdown(shutdownCtx); shutdownErr != nil { + slog.Error("abs compat shutdown error", "error", shutdownErr) + } + } // 2. Clean up stale sessions. if sessionCleaner != nil { @@ -1898,6 +2170,198 @@ func configureS3Clients(cfg *config.Config, deps *api.Dependencies) { } } +type markerPluginCapabilityStore interface { + ListEnabled(ctx context.Context) ([]*plugins.Installation, error) + ListCapabilities(ctx context.Context, installationID int) ([]*plugins.Capability, error) +} + +type markerPluginRuntimeConfigStore interface { + ListGlobalConfigs(ctx context.Context, installationID int) ([]*plugins.RuntimeConfig, error) + PutGlobalConfig(ctx context.Context, installationID int, key string, value map[string]any) error +} + +type markerLegacySettingsStore interface { + Get(ctx context.Context, key string) (string, error) +} + +func reloadMarkerPluginProviders( + ctx context.Context, + registry *markers.Registry, + configStore *markers.ProviderConfigStore, + store markerPluginCapabilityStore, + runtimeConfigs markerPluginRuntimeConfigStore, + legacySettings markerLegacySettingsStore, + resolver *markers.PluginResolverAdapter, +) error { + if registry == nil { + return nil + } + var providers []markers.Provider + if store == nil || resolver == nil { + return registry.SetProviders(providers) + } + + installations, err := store.ListEnabled(ctx) + if err != nil { + return fmt.Errorf("list enabled plugin installations: %w", err) + } + sort.Slice(installations, func(i, j int) bool { + if installations[i] == nil { + return false + } + if installations[j] == nil { + return true + } + return installations[i].ID < installations[j].ID + }) + + nextPriority := 1000 + for _, installation := range installations { + if installation == nil { + continue + } + capabilities, err := store.ListCapabilities(ctx, installation.ID) + if err != nil { + return fmt.Errorf("list marker provider capabilities for installation %d: %w", installation.ID, err) + } + sort.Slice(capabilities, func(i, j int) bool { + if capabilities[i] == nil { + return false + } + if capabilities[j] == nil { + return true + } + return capabilities[i].ID < capabilities[j].ID + }) + for _, capability := range capabilities { + if capability == nil || capability.Type != sdkcapability.MarkerProvider { + continue + } + descriptor, err := plugins.DecodeCapability(capability) + if err != nil { + return fmt.Errorf("decode marker provider capability %d/%s: %w", installation.ID, capability.ID, err) + } + metadataMap := markerCapabilityMetadata(descriptor) + provider, err := markers.NewPluginProvider(markers.PluginProviderOptions{ + InstallationID: installation.ID, + CapabilityID: capability.ID, + DisplayName: firstNonEmptyMarkerText(descriptor.GetDisplayName(), capability.ID), + PluginID: installation.PluginID, + RequiredExternalIDs: markers.PluginRequiredExternalIDsFromMetadata(metadataMap), + }, resolver) + if err != nil { + return err + } + providers = append(providers, provider) + + priority := nextPriority + nextPriority++ + if configuredPriority, ok := markers.PluginDefaultFetchPriorityFromMetadata(metadataMap); ok { + priority = configuredPriority + } + if configStore != nil { + defaultConfig := markers.ProviderConfig{ + Provider: provider.ID(), + FetchEnabled: true, + FetchPriority: priority, + ContributeEnabled: false, + ContributeAutoLocal: false, + ContributeMinConfidence: 0.95, + } + if legacy, ok := legacyIntroDBProviderConfig(configStore, installation, capability, provider.ID()); ok { + defaultConfig = legacy + } + if err := configStore.Ensure(ctx, defaultConfig); err != nil { + return err + } + } + if err := copyLegacyIntroDBPluginConfig(ctx, runtimeConfigs, legacySettings, installation, capability); err != nil { + return err + } + } + } + return registry.SetProviders(providers) +} + +func legacyIntroDBProviderConfig( + configStore *markers.ProviderConfigStore, + installation *plugins.Installation, + capability *plugins.Capability, + providerID string, +) (markers.ProviderConfig, bool) { + if configStore == nil || + installation == nil || + capability == nil || + installation.PluginID != "silo.theintrodb" || + capability.ID != "introdb" { + return markers.ProviderConfig{}, false + } + if _, exists := configStore.Get(providerID); exists { + return markers.ProviderConfig{}, false + } + legacy, ok := configStore.Get("introdb") + if !ok { + return markers.ProviderConfig{}, false + } + legacy.Provider = providerID + return legacy, true +} + +func copyLegacyIntroDBPluginConfig( + ctx context.Context, + runtimeConfigs markerPluginRuntimeConfigStore, + legacySettings markerLegacySettingsStore, + installation *plugins.Installation, + capability *plugins.Capability, +) error { + if runtimeConfigs == nil || + legacySettings == nil || + installation == nil || + capability == nil || + installation.PluginID != "silo.theintrodb" || + capability.ID != "introdb" { + return nil + } + configs, err := runtimeConfigs.ListGlobalConfigs(ctx, installation.ID) + if err != nil { + return fmt.Errorf("list TheIntroDB plugin config: %w", err) + } + for _, config := range configs { + if config != nil && config.Key == "account" { + return nil + } + } + apiKey, err := legacySettings.Get(ctx, "introdb.api_key") + if err != nil { + return fmt.Errorf("load legacy introdb.api_key: %w", err) + } + if strings.TrimSpace(apiKey) == "" { + return nil + } + if err := runtimeConfigs.PutGlobalConfig(ctx, installation.ID, "account", map[string]any{ + "api_key": strings.TrimSpace(apiKey), + }); err != nil { + return fmt.Errorf("copy legacy introdb.api_key to plugin config: %w", err) + } + return nil +} + +func markerCapabilityMetadata(descriptor *pluginv1.CapabilityDescriptor) map[string]any { + if descriptor == nil || descriptor.GetMetadata() == nil { + return nil + } + return descriptor.GetMetadata().AsMap() +} + +func firstNonEmptyMarkerText(values ...string) string { + for _, value := range values { + if strings.TrimSpace(value) != "" { + return strings.TrimSpace(value) + } + } + return "" +} + // mapFolderTypeToMediaType maps silo's MediaFolder.Type values // ("movies", "series", "mixed") to the SDK's MediaType values // ("movie", "tv", "mixed"). Unknown values map to "mixed". @@ -1911,3 +2375,14 @@ func mapFolderTypeToMediaType(t string) string { return "mixed" } } + +// audiobooksSettingsAdapter bridges catalog.ServerSettingsRepo (which +// exposes Get) to the audiobooks.SettingsReader interface (which +// requires GetString). The two signatures are identical modulo name. +type audiobooksSettingsAdapter struct { + repo *catalog.ServerSettingsRepo +} + +func (a *audiobooksSettingsAdapter) GetString(ctx context.Context, key string) (string, error) { + return a.repo.Get(ctx, key) +} diff --git a/cmd/silo/main_test.go b/cmd/silo/main_test.go index d5419357..b5a9cf04 100644 --- a/cmd/silo/main_test.go +++ b/cmd/silo/main_test.go @@ -132,6 +132,9 @@ func TestBuildLiveSessionSync_UsesTransportPlayMethod(t *testing.T) { RequestedMediaFileID: 41, PlayMethod: playback.PlayTranscode, BasePlayMethod: playback.PlayRemux, + TranscodeHWAccel: "qsv", + Position: 125.5, + IsPaused: true, }, want: "transcode", }, @@ -164,6 +167,18 @@ func TestBuildLiveSessionSync_UsesTransportPlayMethod(t *testing.T) { if got.SessionID != tc.session.ID { t.Fatalf("SessionID = %q, want %q", got.SessionID, tc.session.ID) } + if got.ProfileID != tc.session.ProfileID { + t.Fatalf("ProfileID = %q, want %q", got.ProfileID, tc.session.ProfileID) + } + if got.PositionSeconds != tc.session.Position { + t.Fatalf("PositionSeconds = %v, want %v", got.PositionSeconds, tc.session.Position) + } + if got.IsPaused != tc.session.IsPaused { + t.Fatalf("IsPaused = %v, want %v", got.IsPaused, tc.session.IsPaused) + } + if got.TranscodeHWAccel != tc.session.TranscodeHWAccel { + t.Fatalf("TranscodeHWAccel = %q, want %q", got.TranscodeHWAccel, tc.session.TranscodeHWAccel) + } }) } } diff --git a/cmd/silo/marker_plugin_migration_test.go b/cmd/silo/marker_plugin_migration_test.go new file mode 100644 index 00000000..aeaa0f87 --- /dev/null +++ b/cmd/silo/marker_plugin_migration_test.go @@ -0,0 +1,85 @@ +package main + +import ( + "context" + "testing" + + "github.com/Silo-Server/silo-server/internal/plugins" +) + +type fakeMarkerRuntimeConfigs struct { + configs map[int][]*plugins.RuntimeConfig + puts []plugins.RuntimeConfig +} + +func (f *fakeMarkerRuntimeConfigs) ListGlobalConfigs(_ context.Context, installationID int) ([]*plugins.RuntimeConfig, error) { + return append([]*plugins.RuntimeConfig(nil), f.configs[installationID]...), nil +} + +func (f *fakeMarkerRuntimeConfigs) PutGlobalConfig(_ context.Context, installationID int, key string, value map[string]any) error { + f.puts = append(f.puts, plugins.RuntimeConfig{InstallationID: installationID, Key: key, Value: value}) + f.configs[installationID] = append(f.configs[installationID], &plugins.RuntimeConfig{ + InstallationID: installationID, + Key: key, + Value: value, + }) + return nil +} + +type fakeMarkerLegacySettings map[string]string + +func (f fakeMarkerLegacySettings) Get(_ context.Context, key string) (string, error) { + return f[key], nil +} + +func TestCopyLegacyIntroDBPluginConfigCopiesAPIKeyOnce(t *testing.T) { + runtimeConfigs := &fakeMarkerRuntimeConfigs{configs: map[int][]*plugins.RuntimeConfig{}} + installation := &plugins.Installation{ID: 42, PluginID: "silo.theintrodb"} + capability := &plugins.Capability{ID: "introdb"} + + if err := copyLegacyIntroDBPluginConfig( + context.Background(), + runtimeConfigs, + fakeMarkerLegacySettings{"introdb.api_key": " legacy-key "}, + installation, + capability, + ); err != nil { + t.Fatalf("copyLegacyIntroDBPluginConfig: %v", err) + } + if len(runtimeConfigs.puts) != 1 { + t.Fatalf("puts = %d, want 1", len(runtimeConfigs.puts)) + } + got := runtimeConfigs.puts[0] + if got.InstallationID != 42 || got.Key != "account" || got.Value["api_key"] != "legacy-key" { + t.Fatalf("put = %+v, want account api_key copy", got) + } + + if err := copyLegacyIntroDBPluginConfig( + context.Background(), + runtimeConfigs, + fakeMarkerLegacySettings{"introdb.api_key": "new-key"}, + installation, + capability, + ); err != nil { + t.Fatalf("second copyLegacyIntroDBPluginConfig: %v", err) + } + if len(runtimeConfigs.puts) != 1 { + t.Fatalf("second copy overwrote config; puts = %d, want 1", len(runtimeConfigs.puts)) + } +} + +func TestCopyLegacyIntroDBPluginConfigIgnoresOtherPlugins(t *testing.T) { + runtimeConfigs := &fakeMarkerRuntimeConfigs{configs: map[int][]*plugins.RuntimeConfig{}} + if err := copyLegacyIntroDBPluginConfig( + context.Background(), + runtimeConfigs, + fakeMarkerLegacySettings{"introdb.api_key": "legacy-key"}, + &plugins.Installation{ID: 7, PluginID: "silo.other"}, + &plugins.Capability{ID: "introdb"}, + ); err != nil { + t.Fatalf("copyLegacyIntroDBPluginConfig: %v", err) + } + if len(runtimeConfigs.puts) != 0 { + t.Fatalf("puts = %d, want 0 for non-TheIntroDB plugin", len(runtimeConfigs.puts)) + } +} diff --git a/cmd/silo/session_sync.go b/cmd/silo/session_sync.go index 08d699d8..267ba0bb 100644 --- a/cmd/silo/session_sync.go +++ b/cmd/silo/session_sync.go @@ -31,8 +31,10 @@ func buildLiveSessionSync(s *playback.Session, reportingNode string) worker.Sess TargetVideoCodec: s.TargetVideoCodec, TargetAudioCodec: s.TargetAudioCodec, TargetBitrateKbps: s.TargetBitrateKbps, + TranscodeHWAccel: s.TranscodeHWAccel, StartedAt: s.StartedAt, UpdatedAt: s.UpdatedAt, + PositionSeconds: s.Position, IsPaused: s.IsPaused, HasWebSocket: s.HasWebSocket, } diff --git a/docker-compose.nvidia.yml b/docker-compose.nvidia.yml new file mode 100644 index 00000000..b63de25a --- /dev/null +++ b/docker-compose.nvidia.yml @@ -0,0 +1,20 @@ +services: + silo: + deploy: + resources: + reservations: + devices: + - driver: nvidia + count: ${NVIDIA_GPU_COUNT:-1} + capabilities: [gpu] + + # Optional distributed-mode example. Most single-host installs should + # just use the integrated service above. + # silo-transcode: + # deploy: + # resources: + # reservations: + # devices: + # - driver: nvidia + # count: ${NVIDIA_GPU_COUNT:-1} + # capabilities: [gpu] diff --git a/docker-compose.yml b/docker-compose.yml index f995e672..9b4c83c6 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -2,7 +2,7 @@ services: postgres: image: pgvector/pgvector:pg18 restart: unless-stopped - shm_size: 1gb + shm_size: ${POSTGRES_SHM_SIZE:-8gb} environment: POSTGRES_USER: ${POSTGRES_USER:-silo} POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-silo} @@ -11,8 +11,7 @@ services: - "5432:5432" volumes: - ${SILO_DATA_ROOT:-/opt/silo}/postgres:/var/lib/postgresql - - ./postgres/postgresql.conf:/etc/postgresql/postgresql.conf:ro - command: postgres -c config_file=/etc/postgresql/postgresql.conf + command: ["postgres", "-c", "listen_addresses=*"] healthcheck: test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-silo}"] interval: 5s @@ -40,14 +39,19 @@ services: DATABASE_URL: postgres://${POSTGRES_USER:-silo}:${POSTGRES_PASSWORD:-silo}@postgres:5432/${POSTGRES_DB:-silo}?sslmode=disable REDIS_URL: redis://redis:6379 SILO_PLUGIN_CACHE_DIR: /var/lib/silo/plugins + POSTGRES_TUNE: ${POSTGRES_TUNE:-auto} ports: - "${PORT:-8090}:8080" - "${JF_PORT:-8096}:8096" + - "${ABS_PORT:-13378}:13378" volumes: - ${MEDIA_ROOT:?Set MEDIA_ROOT in .env to the host media path}:${MEDIA_CONTAINER_ROOT:-/mnt/media}:ro + - ${MEDIA_BOOKS_ROOT:-${MEDIA_ROOT}}:${MEDIA_BOOKS_CONTAINER_ROOT:-${MEDIA_CONTAINER_ROOT:-/mnt/media}/books}:ro - ${SILO_DATA_ROOT:-/opt/silo}/plugins:/var/lib/silo/plugins - ${SILO_DATA_ROOT:-/opt/silo}/transcode:/tmp/silo-transcode - ${SILO_DATA_ROOT:-/opt/silo}/catalog-seeds:/catalog-seeds:ro + - ${SILO_DATA_ROOT:-/opt/silo}/audiobook-covers:/var/lib/silo/audiobook-covers + - /proc/meminfo:/host/proc/meminfo:ro depends_on: postgres: condition: service_healthy diff --git a/docs/superpowers/notes/2026-05-27-abs-wire-shape-verification.md b/docs/superpowers/notes/2026-05-27-abs-wire-shape-verification.md new file mode 100644 index 00000000..173e3a0a --- /dev/null +++ b/docs/superpowers/notes/2026-05-27-abs-wire-shape-verification.md @@ -0,0 +1,138 @@ +# ABS Wire-Shape Verification (post collections-unify cutover) + +Breadcrumbs for the next person debugging an ABS endpoint wire-shape issue +after the canonical-tables cutover (migration 156 + commits `0dc830e`, +`8c7fe1b`, `b64ce17`). + +The Go in-memory structs (`abs.Collection`, `abs.Playlist`, +`abs.SmartCollection`, `abs.CollectionItem`, `abs.PlaylistItem`) have **no +`json:"..."` struct tags**. The JSON wire contract is defined entirely by +the `*ToABS()` map-builder helpers in `internal/audiobooks/abs/`. As long as +the store layer populates the struct fields with the same values, the wire +shape is preserved. The rewrites in `0dc830e`, `8c7fe1b`, `b64ce17` did NOT +modify the emitters — only the SQL-backed store implementations. + +## Envelope tests — which one guards which endpoint + +Run with `go test ./internal/audiobooks/abs/ -run Envelope -v -count=1`. + +| Test file | Functions | Guards | +|---|---|---| +| `collections_envelope_test.go` | `TestCollectionEnvelope_HasRequiredKeys`, `TestCollectionListShape_OmitsBooks` | `collectionToABS` keys; list-shape (no `books`) vs detail-shape (with `books`) for `GET /api/collections`, `GET /api/collections/{id}`, `GET /api/libraries/{id}/collections`, and all POST/PATCH/DELETE collection endpoints | +| `playlists_envelope_test.go` | `TestPlaylistEnvelope_HasRequiredKeys`, `TestPlaylistEnvelope_OmitsCoverPathWhenEmpty`, `TestPlaylistListShape_OmitsItems` | `playlistToABS` keys; list-shape (no `items`) vs detail-shape (with `items`); `coverPath` is omitted when `CoverItem == ""` — covers `GET /api/playlists`, `GET /api/playlists/{id}`, `GET /api/libraries/{id}/playlists`, batch and item-add/remove endpoints | +| `smart_collections_envelope_test.go` | `TestSmartCollectionEnvelope_HasRequiredKeys`, `TestSmartCollectionEnvelope_EmptyQueryDef` | `smartCollectionToABS` keys; `queryDef` decoded from raw JSONB bytes into nested object, empty bytes → `{}` — covers `GET /api/me/smart-collections`, `GET /api/me/smart-collections/{id}`, POST/PATCH equivalents | +| `bookmarks_envelope_test.go` | `TestBookmarkEnvelope_HasRequiredKeys` | Bookmarks emitter (separate from this cutover, not affected by migration 156) | +| `login_envelope_test.go` | `TestLoginEnvelope_HasRequiredKeys` and three xReturnTokens / displayName variants | Login envelope (not affected by migration 156) | + +In addition, handler-level round-trip tests live in +`playlists_handler_test.go` and `bookmarks_handler_test.go`. There is NO +snapshot/goldenfile harness in the repo today — these envelope tests are +the primary regression guard. + +## Manual live-DB diff procedure + +For a pre/post-deploy wire-shape verification against a live silo, see the +plan's Task 5 "manual verification" section at +`docs/superpowers/plans/2026-05-27-collections-unify-3-abs-adapters.md` +(§ `Task 5: Wire-shape regression test`). Summary: + +1. Pre-cutover, seed one of each (collection, playlist with item, + smart collection) via the old `abs_*` tables, then capture each list + endpoint's response to `/tmp/wire_before_.json` using a curl + against the running silo with a valid ABS bearer token (HS256 JWT — + minted by the login flow, NOT the raw `abs_sessions.token` value). +2. Apply migration 156. Seed equivalent rows in `user_personal_collections` + with the same IDs and content. Capture again to + `/tmp/wire_after_.json`. +3. `diff /tmp/wire_before_.json /tmp/wire_after_.json` for each + `kind in {collections,playlists,smart_collections}`. Expected: empty + diff. + +This is an MR-description-level manual step, not a committed test. + +## Intentionally-zero fields after the rewrite + +These wire keys are still emitted, but the store always populates the +in-memory field with the zero value because the canonical +`user_personal_collections` schema has no analog column (per spec §6 of the +collections-unify plan). They are NOT bugs — do not "fix" them by reaching +for some other column. + +| In-memory field | Wire key | Zero value | Spec ref | Disposition | +|---|---|---|---|---| +| `abs.Playlist.CoverItem` | `coverPath` | `""` (key omitted entirely when empty — see `playlistToABS`) | spec §6.1 | Dropped. PATCH `coverPath` body field is silently ignored by the store. Cover regeneration from first-item poster is the chosen long-term path. | +| `abs.SmartCollection.Color` | `color` | `""` (key always emitted as empty string) | spec §6.3 | Deferred. No column on `user_personal_collections`. Wire key stays present for client compatibility. | +| `abs.SmartCollection.IsPinned` | `isPinned` | `false` (key always emitted) | spec §6.2 | Deferred. Same rationale. | + +If you're adding a "Pin this smart collection" feature later, the column +needs to land in a new migration on `user_personal_collections` first; +don't try to thread it through some adjacent column. + +## Canonical mapping — struct field → source column + +The full pre-cutover wire contract was captured in a working note that does +not persist (`/tmp/abs_wire_contract.md`). The essentials are reproduced +here so the next maintainer doesn't have to re-derive them. + +All three struct families now read from `user_personal_collections` +(and `user_personal_collection_items` for collections + playlists), +discriminated by `collection_type IN ('manual','playlist','smart')`. + +### `abs.Collection` (`collection_type = 'manual'`) + +| Field | Source column | +|---|---| +| ID | `user_personal_collections.id` | +| UserID | `user_personal_collections.user_id::text` (column is `integer`) | +| ProfileID | `user_personal_collections.profile_id` | +| Name | `user_personal_collections.name` | +| Description | `user_personal_collections.description` | +| IsPublic | `user_personal_collections.is_shared` | +| CreatedAt | `user_personal_collections.created_at` | +| UpdatedAt | `user_personal_collections.updated_at` | + +`abs.CollectionItem` reads `user_personal_collection_items` with +`sub_item_id = ''` filter (the manual-collection sentinel established in +migration 156 step 1). LibraryItemID ← `media_item_id`. ORDER BY +`added_at ASC`. + +### `abs.Playlist` (`collection_type = 'playlist'`) + +Same column mapping as `abs.Collection` (modulo `collection_type` filter) +EXCEPT `CoverItem` which is always `""` — see "Intentionally-zero fields" +above. + +`abs.PlaylistItem` reads `user_personal_collection_items` with NO +`sub_item_id` filter (playlists can carry episode entries). Mapping: +LibraryItemID ← `media_item_id`, EpisodeID ← `sub_item_id`, +Position ← `position`. ORDER BY `position ASC, added_at ASC`. + +### `abs.SmartCollection` (`collection_type = 'smart'`) + +Same column mapping as `abs.Collection` EXCEPT: + +- `Color`, `IsPinned` → always zero (see above). +- `QueryDef` ← `user_personal_collections.query_definition` (JSONB → `[]byte` + round-trip; column is `NOT NULL DEFAULT '{}'::jsonb` per migration 016). + +No items table — smart-collection membership is evaluated at request time +via the `smartcoll` package. + +### Wire-shape quirks worth remembering + +- Collection/Playlist emit `lastUpdate` (NOT `updatedAt`). SmartCollection + emits `updatedAt`. Cross-struct inconsistency, carry forward verbatim. +- All timestamps are `UnixMilli()` int64, NOT RFC3339 strings. +- `ProfileID` is carried in memory but NEVER emitted on the wire — it's + scope/auth only. +- The list vs detail shape distinction is implicit: list responses pass + `nil` for the items/books slice; the emitter then omits the key + entirely. Clients differentiate on key presence. + +## Verification status (2026-05-27) + +- All 13 envelope tests pass (run: `go test ./internal/audiobooks/abs/ + -run Envelope -v -count=1`). +- Full audiobooks test suite passes (`go test ./internal/audiobooks/... + -short -count=1 -timeout 120s`). +- Live-DB diff was NOT executed in CI — see manual procedure above. diff --git a/docs/superpowers/plans/2026-05-24-audiobook-ui-redesign.md b/docs/superpowers/plans/2026-05-24-audiobook-ui-redesign.md new file mode 100644 index 00000000..c00d6b81 --- /dev/null +++ b/docs/superpowers/plans/2026-05-24-audiobook-ui-redesign.md @@ -0,0 +1,3072 @@ +# Audiobook UI Redesign Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Bring the audiobook detail page and player to visual + interaction parity with Silo's video player, translated for audio, per the [design spec](../specs/2026-05-24-audiobook-ui-redesign-design.md). + +**Architecture:** Extract `CircleButton` and add new menu primitives (`SpeedMenu`, `SleepTimerMenu`) into `web/src/player/components/` so both players consume the same source of truth. Split today's monolithic `AudiobookPlayer.tsx` into a state hook (`useAudiobookPlayback`) plus two chrome components (`MiniBar`, `NowListening`) under `web/src/pages/audiobooks/player/`. The same `