Compare commits
44
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5b70dc7357 | ||
|
|
ade3a3c5d4 | ||
|
|
ce06dc8ca5 | ||
|
|
604bbf1a0f | ||
|
|
631d95530a | ||
|
|
d590eda22a | ||
|
|
c75c519e3d | ||
|
|
012006e7a7 | ||
|
|
f983c54e87 | ||
|
|
a4402d4621 | ||
|
|
866392fecd | ||
|
|
287ff3658f | ||
|
|
b250dbb59b | ||
|
|
26089950af | ||
|
|
d1fd7c236f | ||
|
|
18e9856227 | ||
|
|
57fbc1e0b4 | ||
|
|
9cae868a27 | ||
|
|
1e8f79fe18 | ||
|
|
9c6252dace | ||
|
|
8b3ebc78b6 | ||
|
|
2486345679 | ||
|
|
eb35141ecc | ||
|
|
cf0db385f3 | ||
|
|
eafa22a3c3 | ||
|
|
652b5d1e9f | ||
|
|
4fb8f6a711 | ||
|
|
1dbcf2cd9a | ||
|
|
75cf5e3b11 | ||
|
|
8afe783277 | ||
|
|
9cea2bb4f0 | ||
|
|
a0fbe2cdda | ||
|
|
4d2a990de1 | ||
|
|
96a279819e | ||
|
|
443445f0fe | ||
|
|
02e62767a1 | ||
|
|
b4a66b29e2 | ||
|
|
6854eac80e | ||
|
|
6414061022 | ||
|
|
a3aa93534d | ||
|
|
d83f7fadef | ||
|
|
185a90b048 | ||
|
|
78d4f9acbf | ||
|
|
7f67bb80ad |
@@ -0,0 +1,264 @@
|
||||
---
|
||||
name: issue-to-pr
|
||||
disable-model-invocation: true
|
||||
description: >-
|
||||
End-to-end implement a GitHub issue and open a ready pull request. Invoke
|
||||
explicitly as `/issue-to-pr <number|#ref|url>` (this is a side-effecting
|
||||
workflow — it creates branches and opens PRs — so it does not auto-trigger).
|
||||
It creates an isolated git worktree off the latest main, reproduces the bug
|
||||
(or scopes the feature), finds the root cause, implements the smallest correct
|
||||
change following repo conventions, verifies it (build/lint/tests), runs a Codex
|
||||
adversarial review and addresses the material findings, then commits and opens
|
||||
a ready PR. Not for pure triage (use triage-issue), pure review of an existing
|
||||
PR (use review-pr-or-mr / code-review), or analysis without code changes.
|
||||
---
|
||||
|
||||
# Issue → PR
|
||||
|
||||
Turn a single GitHub issue into a verified, reviewed, ready-to-merge pull
|
||||
request — autonomously — while staying honest about uncertainty and stopping
|
||||
when a human decision is genuinely required.
|
||||
|
||||
The goal is not "produce a diff." It is: **confirm the real problem exists,
|
||||
make the smallest change that correctly solves it, prove it works by execution,
|
||||
have an adversary try to break it, and only then ship a PR.** Speed comes from
|
||||
doing these in order, not from skipping them.
|
||||
|
||||
## Inputs
|
||||
|
||||
- The issue reference is passed as `$ARGUMENTS` — a number (`123`), a `#123`
|
||||
reference, or a full issue URL. If none is given, ask; do not guess.
|
||||
- The repo is the current git repository; PRs target `origin`
|
||||
(`Silo-Server/silo-server`) and base `main`.
|
||||
|
||||
## Operating principles
|
||||
|
||||
- **Default to "maybe nothing needs fixing."** Coding agents "fix" already-correct
|
||||
or stale code in a large fraction of cases because they don't stop to confirm the
|
||||
problem is real. Confirm the issue reproduces on current `main` before changing
|
||||
anything — it may already be fixed.
|
||||
- **Root cause over symptom.** A fix you can't explain is a fix you can't trust.
|
||||
- **Verify by execution, not by reading.** Build/lint/tests are the primary
|
||||
correctness signal. The adversarial review is a *second* layer on top — never a
|
||||
substitute for green checks. If you can't verify it, don't ship it.
|
||||
- **Smallest correct change.** Match surrounding code's idioms, naming, and comment
|
||||
density. Put code in the package that owns the behavior (CLAUDE.md); don't create
|
||||
catch-all helpers or duplicate logic — search for an existing helper before
|
||||
writing a new one, and extract shared logic instead of copying.
|
||||
- **Don't trust recall for APIs.** Verify every symbol, function, and signature you
|
||||
call actually exists by grepping the codebase — agents hallucinate plausible-looking
|
||||
APIs.
|
||||
- **Don't gold-plate.** A reviewer told to find gaps will always find some. Address
|
||||
what affects correctness or the issue's stated requirements; treat the rest as
|
||||
optional. Extra abstraction "just in case" is a defect, not diligence.
|
||||
- **Know when to stop.** See [Stop and ask](#stop-and-ask). Autonomy is for
|
||||
execution, not for inventing product decisions or shipping unverifiable guesses.
|
||||
|
||||
Track the phases below with a task list (TaskCreate/TaskUpdate) so progress is
|
||||
visible and nothing is skipped on long runs.
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Understand the issue
|
||||
|
||||
Read everything before writing anything.
|
||||
|
||||
```bash
|
||||
gh issue view <N> --json number,title,body,labels,state,author,url,comments
|
||||
```
|
||||
|
||||
Determine: **bug vs. feature** (labels + body → branch prefix `fix/` vs `feat/`);
|
||||
**acceptance criteria** (what does "done" mean?); **linked scope** (does it serve
|
||||
an epic / sub-issue to reference as `Part of #NNN`?); and **client surface** (does
|
||||
it touch API/auth/playback/session/library/metadata behavior consumed by
|
||||
`silo-android` / `silo-apple`? — if so, coordinated client follow-up may be needed).
|
||||
|
||||
If the issue is vague, underspecified, or needs a product call, stop and ask now —
|
||||
before spending a worktree on the wrong problem.
|
||||
|
||||
### 2. Create a worktree off the latest main
|
||||
|
||||
Work in isolation so the user's checkout is untouched. Create it at the **main
|
||||
repository root**, not inside the current worktree.
|
||||
|
||||
```bash
|
||||
# Resolve the primary repo root (works from inside any worktree)
|
||||
MAIN_ROOT="$(cd "$(dirname "$(git rev-parse --git-common-dir)")" && pwd)"
|
||||
git -C "$MAIN_ROOT" fetch origin # always branch off fresh main
|
||||
|
||||
SLUG=<short-kebab-slug-from-issue-title>
|
||||
BRANCH=fix/issue-<N>-$SLUG # use feat/ for features
|
||||
WT="$MAIN_ROOT/.claude/worktrees/issue-<N>"
|
||||
|
||||
git -C "$MAIN_ROOT" worktree add -b "$BRANCH" "$WT" origin/main
|
||||
cd "$WT"
|
||||
```
|
||||
|
||||
If the branch/worktree already exists from a prior run, reuse it (cd in) rather
|
||||
than failing — but confirm it's based on current `origin/main`; if it's stale, tell
|
||||
the user before continuing. From here, **all work happens in `$WT`** (the Bash tool
|
||||
keeps the working directory between calls); use absolute paths when in doubt.
|
||||
|
||||
### 3. Reproduce / confirm — a hard gate
|
||||
|
||||
This is the highest-leverage step. Do not skip it.
|
||||
|
||||
- **Bug:** reproduce it concretely — ideally a **test that fails on current `main`**
|
||||
(red), or a precise script/trace through the faulty path. Pin down the exact code
|
||||
path and the invariant it violates.
|
||||
- **If you cannot reproduce it, STOP and report** — do not patch. The issue may
|
||||
already be fixed, be environment-specific, or be a misunderstanding. Shipping a
|
||||
speculative fix for a non-reproducing bug is the most common way agents make
|
||||
things worse.
|
||||
- **Feature:** confirm the capability is actually missing, then map where it fits —
|
||||
the owning package, patterns to follow, the data flow, the smallest set of files.
|
||||
|
||||
**Plan proportionally.** If you could describe the diff in one sentence (a typo, a
|
||||
log line, a one-line guard), skip formal planning and make the change. Plan first
|
||||
when the work is multi-file, unfamiliar, or architecturally uncertain — name the
|
||||
files/interfaces involved, state what's out of scope, and end the plan with the
|
||||
verification step. For anything large, confirm direction with the user before a big
|
||||
build-out.
|
||||
|
||||
### 4. Implement
|
||||
|
||||
Make the change, scoped to this one concern (one concern per PR). Respect the hard
|
||||
rules:
|
||||
|
||||
- **API:** additive-only within `/api/v1` — never rename/remove a response field,
|
||||
change a field's type, or repurpose a status code. New behavior = new
|
||||
fields/endpoints; expose capability endpoints for feature detection.
|
||||
- **Migrations:** new DB changes are Goose SQL migrations via
|
||||
`make migrate-create NAME=...` (timestamped). Never `goose fix`, never paired
|
||||
`.up.sql`/`.down.sql`.
|
||||
- **Maintainability:** extract shared logic instead of duplicating; change existing
|
||||
code rather than bolting on local workarounds.
|
||||
|
||||
For a bug, the failing test from step 3 should now pass.
|
||||
|
||||
### 5. Verify — the primary correctness gate
|
||||
|
||||
Run the checks and make them genuinely pass. The exact commands and the worktree
|
||||
quirks (`GOWORK=off`, stubbing `web/dist`, the v2 golangci invocation, the flaky
|
||||
GPU tests) are in **[references/verification.md](references/verification.md)** —
|
||||
read it first, because plain `go build ./...` / `make lint` fail inside
|
||||
`.claude/worktrees`.
|
||||
|
||||
At minimum: Go build + the relevant package tests pass; if `web/` changed,
|
||||
`pnpm run lint` + `pnpm run format:check` + build pass; `make verify-local-paths` is
|
||||
clean. Run the **full relevant suite**, not just your new test — catch regressions
|
||||
in code you didn't touch. Show the command output as evidence; don't assert success.
|
||||
|
||||
**Never reach green by weakening the checks.** Deleting/skipping/`xfail`-ing tests,
|
||||
loosening assertions, lowering coverage thresholds, or relaxing lint/CI to make a
|
||||
red check pass is a blocker, full stop — if a test genuinely must change, that needs
|
||||
explicit human sign-off, not a quiet edit. If you can't make the checks pass on the
|
||||
merits, stop and report.
|
||||
|
||||
### 6. Adversarial review (Codex)
|
||||
|
||||
Commit your work first (so the review sees exactly what will ship), then run a Codex
|
||||
adversarial review of the branch against `main`. Full invocation, output handling,
|
||||
the iteration cap, and the fallback are in
|
||||
**[references/codex-review.md](references/codex-review.md)** — follow it exactly.
|
||||
|
||||
The loop, in short:
|
||||
1. Commit the implementation on the branch.
|
||||
2. Run the adversarial review (`--base main`, foreground/`--wait`, extended timeout).
|
||||
3. For each finding: **first confirm it's real and grounded in the actual code**
|
||||
(reviewers over-flag — don't fix phantom nits). Then **fix** material findings
|
||||
that affect correctness or the stated requirements (and re-run the review), or
|
||||
**rebut** them with a defensible, written reason. Note non-material findings as
|
||||
optional.
|
||||
4. **Cap the loop at 2 iterations (3 absolute max).** Debugging effectiveness decays
|
||||
sharply after ~2–3 attempts, so thrashing won't help — if material findings still
|
||||
stand after the cap, stop and escalate to the user instead of looping.
|
||||
|
||||
Do not open the PR while a material, un-rebutted adversarial finding stands.
|
||||
|
||||
### 7. Commit, push, open the PR
|
||||
|
||||
Conventional Commit subjects scoped to the domain, e.g.
|
||||
`fix(playback): guard against nil session on reconnect`. End commit messages with:
|
||||
|
||||
```
|
||||
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||||
```
|
||||
|
||||
Push and open a **ready** (non-draft) PR against `main`:
|
||||
|
||||
```bash
|
||||
git push -u origin "$BRANCH"
|
||||
gh pr create --base main --head "$BRANCH" --title "<conventional subject>" --body "$(cat <<'EOF'
|
||||
## Problem
|
||||
<what's broken / missing, grounded in the issue — and how you reproduced it>
|
||||
|
||||
## Approach
|
||||
<what you changed and *why this approach* — the tradeoff, not a file list>
|
||||
|
||||
## Verification
|
||||
<commands run and their result; the failing→passing test; manual checks>
|
||||
|
||||
## Adversarial review
|
||||
<clean, or the material findings and how each was fixed/rebutted>
|
||||
|
||||
## Risks & follow-up
|
||||
<rollout/migration/compatibility risks; any silo-android / silo-apple
|
||||
follow-up; anything deferred>
|
||||
|
||||
Closes #<N>
|
||||
<add "Part of #<epic>" if the issue serves a capability epic/sub-issue>
|
||||
|
||||
## AI-use disclosure
|
||||
Implemented by Claude Code (issue-to-pr skill) with human oversight.
|
||||
EOF
|
||||
)"
|
||||
```
|
||||
|
||||
End the PR body with:
|
||||
|
||||
```
|
||||
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||||
```
|
||||
|
||||
For **UI changes**, attach before/after screenshots or a recording (the repo
|
||||
requires this); if you can't capture them, explicitly flag in the PR that they're
|
||||
still needed.
|
||||
|
||||
### 8. Report back
|
||||
|
||||
Give the user the PR URL, a 2–3 line summary, the verification result, the
|
||||
adversarial-review outcome, any flagged risks or required client follow-up, and the
|
||||
worktree path so they can inspect it.
|
||||
|
||||
## Long or multi-iteration runs
|
||||
|
||||
Performance degrades on long sessions, and context compaction can silently drop
|
||||
constraints. So: keep a short progress note and a "failed approaches" list in the
|
||||
scratchpad so you don't repeat dead ends; and after any compaction, re-read CLAUDE.md
|
||||
and re-assert the hard guardrails (additive-only API, migration rules, no weakening
|
||||
of tests) before continuing.
|
||||
|
||||
## Stop and ask
|
||||
|
||||
Pause and ask the user (don't push a PR) when:
|
||||
|
||||
- The issue is ambiguous, underspecified, or needs a product/UX decision.
|
||||
- **The bug doesn't reproduce** on current `main`.
|
||||
- The fix would require a non-additive API change, or a risky/irreversible migration.
|
||||
- The change balloons into something large or architectural — propose a plan first.
|
||||
- Checks can't be made to pass on the merits, or material adversarial findings remain
|
||||
after the iteration cap.
|
||||
- A test/CI change would be needed to "pass" — get sign-off, don't do it silently.
|
||||
- The right fix clearly belongs in a sibling repo (`silo-android`, `silo-apple`, a
|
||||
plugin/SDK repo) — see CLAUDE.md's multi-repo guidance.
|
||||
|
||||
In these cases, summarize what you found and what you'd do, and let the user decide.
|
||||
|
||||
## Notes
|
||||
|
||||
- Authorized to open ready PRs autonomously once checks pass and the adversarial
|
||||
review is clean — that's the durable instruction from setup. Not authorized to
|
||||
merge, force-push shared branches, or close issues by hand.
|
||||
- Never invent a fix you can't verify or explain. An honest "here's what's uncertain"
|
||||
beats a confident wrong PR.
|
||||
@@ -0,0 +1,88 @@
|
||||
# Codex adversarial review
|
||||
|
||||
The `/codex:adversarial-review` command is marked `disable-model-invocation: true`,
|
||||
so you cannot trigger it as a tool/skill. Call the same underlying runtime
|
||||
directly via Bash: the `codex-companion.mjs` helper, subcommand
|
||||
`adversarial-review`. It challenges the *approach and design*, not just defects —
|
||||
it tries to find the strongest reasons the change should not ship.
|
||||
|
||||
## 1. Locate the companion script
|
||||
|
||||
```bash
|
||||
CODEX_COMPANION="${CLAUDE_PLUGIN_ROOT:+$CLAUDE_PLUGIN_ROOT/scripts/codex-companion.mjs}"
|
||||
if [ -z "$CODEX_COMPANION" ] || [ ! -f "$CODEX_COMPANION" ]; then
|
||||
CODEX_COMPANION="$(find "$HOME/.claude/plugins/marketplaces" "$HOME/.claude/plugins/cache" \
|
||||
-name codex-companion.mjs 2>/dev/null | head -n1)"
|
||||
fi
|
||||
echo "$CODEX_COMPANION"
|
||||
```
|
||||
|
||||
If nothing is found, the Codex plugin isn't installed/set up. Don't silently skip
|
||||
the review — tell the user, point them at `/codex:setup`, and ask whether to
|
||||
proceed without it (or fall back to a `feature-dev:code-reviewer` agent pass).
|
||||
The user asked specifically for the Codex adversarial review, so treat its absence
|
||||
as a real blocker, not a detail.
|
||||
|
||||
## 2. Commit first, then review the branch against main
|
||||
|
||||
Review what will actually ship. Commit the implementation on the branch, then
|
||||
diff against `main`:
|
||||
|
||||
```bash
|
||||
git add -A && git commit -m "<conventional subject>"
|
||||
```
|
||||
|
||||
Run it in the **foreground** so results come back this turn, and set an extended
|
||||
Bash timeout (reviews can take several minutes) — pass `timeout: 600000` on the
|
||||
Bash call:
|
||||
|
||||
```bash
|
||||
node "$CODEX_COMPANION" adversarial-review --wait --base main "optional focus text"
|
||||
```
|
||||
|
||||
- `--wait` runs foreground and prints the result to stdout.
|
||||
- `--base main` reviews the whole branch diff (`main...HEAD`).
|
||||
- Append focus text to steer it (e.g. the specific failure mode, the risky
|
||||
concurrency path, the migration). Keep the adversarial framing — don't soften it.
|
||||
- Do **not** use `--background` here; you need the findings in-hand to act on them.
|
||||
|
||||
## 3. Interpret the output
|
||||
|
||||
The review returns a verdict plus findings:
|
||||
|
||||
- **`approve` / no material findings** → green light for the PR.
|
||||
- **`needs-attention`** → there is at least one material risk. Each finding names
|
||||
a file, a line range, an impact, and a concrete recommendation.
|
||||
|
||||
LLM reviewers systematically **over-flag** — they report gaps even when the work is
|
||||
sound. So for every finding, in order:
|
||||
|
||||
1. **Confirm it's real.** Open the cited file/lines and verify the finding is
|
||||
grounded in the actual code and actually reachable. Drop findings that reference
|
||||
code that doesn't exist, lines outside the diff, or behavior the code doesn't have
|
||||
— don't fix phantom nits.
|
||||
2. For a confirmed finding, do one of two things — never silently ignore it:
|
||||
- **Fix it** if it affects correctness or the issue's stated requirements, then
|
||||
re-run the review (go back to step 2 after committing the fix); or
|
||||
- **Rebut it** with a defensible, written reason it doesn't apply, and capture
|
||||
that reasoning in the PR's "Adversarial review" section.
|
||||
3. Treat style/naming/speculative/low-value findings as **optional** — don't
|
||||
gold-plate. The adversarial prompt is told not to emit these, but if any slip
|
||||
through, note and move on; don't add abstraction or defensive code for cases that
|
||||
can't happen.
|
||||
|
||||
## 4. Loop — but cap it
|
||||
|
||||
Repeat implement → verify → commit → review until the review is `approve` or every
|
||||
material finding is fixed or defensibly rebutted.
|
||||
|
||||
**Cap the loop at 2 iterations (3 absolute max).** Debugging effectiveness decays
|
||||
sharply after ~2–3 attempts — if material findings still stand after the cap, stop
|
||||
and escalate to the user with a summary rather than thrashing or shipping over a
|
||||
standing objection. Only open the PR once the review is clean within the cap.
|
||||
|
||||
## 5. Record it in the PR
|
||||
|
||||
Summarize the outcome in the PR body's **Adversarial review** section: either
|
||||
"clean — no material findings" or a terse list of what was raised and how each was
|
||||
fixed or rebutted. This gives reviewers the adversary's perspective for free.
|
||||
@@ -0,0 +1,77 @@
|
||||
# Verification inside a worktree
|
||||
|
||||
Commands assume the repository root is the cwd (the new worktree, `$WT`).
|
||||
|
||||
## Why the obvious commands fail here
|
||||
|
||||
A `.claude/worktrees/*` checkout lives under the parent Silo workspace, whose
|
||||
`go.work` only lists the primary `./silo-server`, not worktree checkouts. So:
|
||||
|
||||
- `go env GOWORK` resolves to the parent workspace from anywhere under `SiloServer/`.
|
||||
- Plain `go build ./...` / `go test ./...` fail with
|
||||
*"directory prefix . does not contain modules listed in go.work"*.
|
||||
- `go build ./...` also needs `web/dist` to exist (the embed directive).
|
||||
- `golangci-lint` is not installed locally.
|
||||
|
||||
## Go
|
||||
|
||||
Prefix every Go command with `GOWORK=off`.
|
||||
|
||||
```bash
|
||||
# Build the parts that matter without needing web/dist:
|
||||
GOWORK=off go build ./internal/... ./cmd/...
|
||||
|
||||
# Run the tests for the package(s) you touched (fast, targeted):
|
||||
GOWORK=off go test ./internal/<pkg>/...
|
||||
|
||||
# If you need a full ./... build, stub the embed target first:
|
||||
mkdir -p web/dist && touch web/dist/.gitkeep
|
||||
GOWORK=off go build ./...
|
||||
```
|
||||
|
||||
Lint Go without installing golangci-lint — must be the **v2** module path (the
|
||||
repo's `.golangci` config rejects v1):
|
||||
|
||||
```bash
|
||||
GOWORK=off go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest run ./internal/...
|
||||
```
|
||||
|
||||
**Known flake, not a real failure:** `internal/playback` GPU-probe tests
|
||||
(`TestResolveHWAccel*`, the NVENC smoke test) can fail with `signal: killed`
|
||||
under parallel load on macOS, but pass in isolation. If they're the only
|
||||
failures, re-run them alone before treating them as real:
|
||||
|
||||
```bash
|
||||
GOWORK=off go test ./internal/playback/ -run TestResolveHWAccel -count=1
|
||||
```
|
||||
|
||||
## Frontend (only if `web/` changed)
|
||||
|
||||
```bash
|
||||
cd web
|
||||
pnpm install # if deps changed or node_modules is absent
|
||||
pnpm run lint
|
||||
pnpm run format:check
|
||||
pnpm run build # catches type/build errors before the PR
|
||||
cd ..
|
||||
```
|
||||
|
||||
The repo requires `pnpm run lint` and `pnpm run format:check` to be clean before
|
||||
a merge request.
|
||||
|
||||
## Repo-wide
|
||||
|
||||
```bash
|
||||
make verify-local-paths # must be clean; CLAUDE.md forbids absolute paths in docs/specs
|
||||
```
|
||||
|
||||
## Migrations
|
||||
|
||||
If the change needs schema work, create a timestamped Goose migration — never
|
||||
hand-number, never `goose fix`, never paired up/down files:
|
||||
|
||||
```bash
|
||||
make migrate-create NAME=add_thing
|
||||
make migrate-validate
|
||||
make migrate-status
|
||||
```
|
||||
@@ -0,0 +1 @@
|
||||
github: quick104
|
||||
+8
-1
@@ -50,7 +50,11 @@ docs/inspiration
|
||||
|
||||
|
||||
.agents/
|
||||
/.claude/
|
||||
# Ignore everything in .claude/ except committed, shared skills.
|
||||
/.claude/*
|
||||
!/.claude/skills/
|
||||
/.claude/skills/*
|
||||
!/.claude/skills/issue-to-pr/
|
||||
.cursor/
|
||||
.superpowers/
|
||||
.codex/
|
||||
@@ -60,10 +64,13 @@ logs/
|
||||
.playwright-mcp/
|
||||
demo/
|
||||
docker-compose.dev.yml
|
||||
docker-compose.local.yml
|
||||
# Local-only deployment override (unpublishes bundled redis/postgres host
|
||||
# ports via `ports: !override []`). Kept out of git so a rebase from main
|
||||
# never disturbs it and it never lands in a PR; its absence once exposed
|
||||
# Redis to the internet, so it must persist on the box.
|
||||
docker-compose.override.yml
|
||||
# Local compose customization layered onto docker-compose.yml by make dev-deploy.
|
||||
docker-compose.local.yml
|
||||
.playwright-cli/
|
||||
output/
|
||||
|
||||
+112
@@ -0,0 +1,112 @@
|
||||
# Developing Silo
|
||||
|
||||
This document covers building, running, and contributing to the Silo server. If you just want to run Silo, see the [README](README.md).
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution expectations, merge request guidance, and the policy for AI-assisted submissions.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Go** 1.24+
|
||||
- **Bun** 1.0+
|
||||
- **PostgreSQL** 18+
|
||||
- **FFmpeg** (for transcoding support)
|
||||
|
||||
## Local Development
|
||||
|
||||
Local development remains intentionally separate from the deploy-oriented compose setup. Use [docker-compose.yml](docker-compose.yml) for local services and the source-build workflow below.
|
||||
|
||||
```sh
|
||||
# Start local PostgreSQL and Redis
|
||||
docker compose up -d postgres redis
|
||||
|
||||
# Run the frontend dev server (hot reload, proxies API to :8090)
|
||||
make dev-frontend
|
||||
|
||||
# Run the Go backend
|
||||
make dev-backend
|
||||
```
|
||||
|
||||
The main compose file expects `MEDIA_ROOT` to be set even if you only want the bundled PostgreSQL and Redis services, so set that in `.env` first (`cp .env.example .env`).
|
||||
|
||||
If you are developing `Silo` and `silo-plugin-sdk` together, keep using the local [`go.work`](go.work) workspace. That workspace is a developer convenience only. CI and release builds run with `GOWORK=off`, so any new SDK helper used here must be pushed and tagged in `silo-plugin-sdk` before this repo can merge or release the change.
|
||||
|
||||
Plugin authors should start with [docs/architecture/plugin-development.md](docs/architecture/plugin-development.md), which covers the RPC plugin package format, generated proto workflow, SDK import paths, route and asset exposure, and auth or user-config integration points.
|
||||
|
||||
## Make Targets
|
||||
|
||||
| Target | Description |
|
||||
|---|---|
|
||||
| `make build` | Build frontend + Go binary |
|
||||
| `make frontend` | Build frontend only |
|
||||
| `make dev-frontend` | Vite dev server with HMR |
|
||||
| `make dev-backend` | Run Go backend (integrated mode) |
|
||||
| `make dev-proxy` | Run a standalone proxy node |
|
||||
| `make dev-transcode` | Run a standalone transcode node |
|
||||
| `make migrate-create NAME=add_thing` | Create a timestamped Goose SQL migration |
|
||||
| `make migrate-validate` | Validate Goose migration files without touching a database |
|
||||
| `make migrate-status` | Show Goose migration status using Silo's bootstrapping runner |
|
||||
| `make migrate-up` | Apply pending Goose migrations using Silo's bootstrapping runner |
|
||||
| `make clean` | Remove build artifacts |
|
||||
|
||||
## Database Migrations
|
||||
|
||||
PostgreSQL schema migrations are managed by Goose. Migration SQL files live in
|
||||
`migrations/sql/` and use Goose annotations. Converted legacy migrations keep
|
||||
their original numeric versions so existing `schema_versions` rows can bootstrap
|
||||
cleanly into Goose without replaying old SQL. New migrations should be created
|
||||
with timestamped filenames:
|
||||
|
||||
```sh
|
||||
make migrate-create NAME=add_thing
|
||||
make migrate-validate
|
||||
```
|
||||
|
||||
Do not run `goose fix`; timestamped migrations are the repository policy because
|
||||
they avoid version collisions across parallel PRs. The existing `001`-style
|
||||
files are historical compatibility records, not the naming pattern for new work.
|
||||
Runtime migrations are applied by the integrated/API server only. Proxy and
|
||||
transcode modes never mutate schema.
|
||||
For existing installs, use `make migrate-status` and `make migrate-up` rather
|
||||
than invoking the Goose CLI directly; those targets copy legacy
|
||||
`schema_versions` rows into `public.goose_db_version` under the migration lock
|
||||
before reading or applying migrations. Set `ENV_FILE=path/to/.env` when the
|
||||
database URL should be read from a non-default env file.
|
||||
|
||||
## Running Tests
|
||||
|
||||
```sh
|
||||
# Go tests (uses testcontainers — Docker must be running)
|
||||
go test ./...
|
||||
|
||||
# Frontend tests
|
||||
cd web && bun test
|
||||
```
|
||||
|
||||
## Linting
|
||||
|
||||
```sh
|
||||
# Go
|
||||
golangci-lint run
|
||||
|
||||
# Frontend
|
||||
cd web && bun run lint
|
||||
cd web && bun run format:check
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
cmd/silo/ Entry point
|
||||
internal/
|
||||
api/ HTTP router, handlers, middleware
|
||||
auth/ JWT authentication and sessions
|
||||
catalog/ Media item, episode, season repositories
|
||||
config/ YAML + env var configuration
|
||||
jellycompat/ Jellyfin/Emby protocol compatibility
|
||||
metadata/ Plugin-driven metadata matching and enrichment
|
||||
playback/ Direct play, remux, transcode session management
|
||||
scanner/ Media file discovery and FFProbe
|
||||
worker/ Background jobs (scan, match, reconcile)
|
||||
web/ React + TypeScript frontend (Vite, Tailwind, shadcn/ui)
|
||||
migrations/sql/ Goose-managed PostgreSQL schema migrations
|
||||
```
|
||||
@@ -1,8 +1,17 @@
|
||||
# Silo
|
||||
|
||||
A self-hosted media streaming server with a React frontend and Go backend. Supports direct play, remuxing, and hardware-accelerated transcoding. Includes optional Jellyfin/Emby-compatible app support for clients such as VidHub and Findroid.
|
||||
Silo is a self-hosted media streaming server for your movies, shows, music, and books. Point it at your media folders and stream to your devices — at home or away — with direct play, remuxing, and hardware-accelerated transcoding handled automatically.
|
||||
|
||||
Join the community on [Discord](https://discord.gg/4RxuUQAEnW).
|
||||
Join the community on [Discord](https://discord.gg/4RxuUQAEnW). If Silo is useful to you, consider [sponsoring the project](https://github.com/sponsors/quick104) — see [Supporting Silo](#supporting-silo).
|
||||
|
||||
## Highlights
|
||||
|
||||
- **Plays your media, your way** — direct play when the device supports it, remux or hardware-accelerated transcode (including NVENC) when it doesn't.
|
||||
- **Web app included** — a full-featured web client and admin interface ship with the server.
|
||||
- **Works with apps you already use** — optional Jellyfin/Emby-compatible API supports clients such as VidHub, Findroid, and Infuse.
|
||||
- **Household profiles** — multiple profiles per account, with per-profile watch state and parental controls.
|
||||
- **Plugin-driven metadata** — match and enrich your libraries with providers like TMDB and TVDB, installed as plugins.
|
||||
- **Fast setup** — one `docker compose up -d` brings up the whole stack; everything else is configured in the admin UI.
|
||||
|
||||
## Deploy with Docker (recommended)
|
||||
|
||||
@@ -105,165 +114,6 @@ Silo is externally stateful by default rather than fully stateless. Durable appl
|
||||
Migrating an existing Continuum Docker install should be done with the preflight
|
||||
helper and cutover guide in [docs/continuum-to-silo-docker-migration.md](docs/continuum-to-silo-docker-migration.md).
|
||||
|
||||
## Build from Source
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- **Go** 1.24+
|
||||
- **Bun** 1.0+
|
||||
- **PostgreSQL** 18+
|
||||
- **FFmpeg** (for transcoding support)
|
||||
|
||||
### Quick Start
|
||||
|
||||
1. **Start PostgreSQL** (skip if you already have one running)
|
||||
|
||||
```sh
|
||||
docker compose up -d postgres redis
|
||||
```
|
||||
|
||||
The main compose file still expects `MEDIA_ROOT` to be set even if you only want the bundled PostgreSQL and Redis services, so set that in `.env` first.
|
||||
|
||||
2. **Configure the database connection**
|
||||
|
||||
```sh
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Edit `.env` and set `DATABASE_URL` to point to your PostgreSQL instance.
|
||||
|
||||
3. **Build and run**
|
||||
|
||||
```sh
|
||||
make build
|
||||
./silo
|
||||
```
|
||||
|
||||
The server starts at `http://localhost:8080` by default. All other settings are configured through the admin UI.
|
||||
|
||||
## Development
|
||||
|
||||
Local development remains intentionally separate from the deploy-oriented compose setup. Use [docker-compose.yml](docker-compose.yml) and the existing source-build workflow for local development.
|
||||
|
||||
```sh
|
||||
# Run the frontend dev server (hot reload, proxies API to :8090)
|
||||
make dev-frontend
|
||||
|
||||
# Run the Go backend
|
||||
make dev-backend
|
||||
```
|
||||
|
||||
If you are developing `Silo` and `silo-plugin-sdk` together, keep using the local [`go.work`](go.work) workspace. That workspace is a developer convenience only. CI and release builds run with `GOWORK=off`, so any new SDK helper used here must be pushed and tagged in `silo-plugin-sdk` before this repo can merge or release the change.
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution expectations, merge request guidance, and the policy for AI-assisted submissions.
|
||||
|
||||
Plugin authors should start with [docs/architecture/plugin-development.md](docs/architecture/plugin-development.md), which covers the RPC plugin package format, generated proto workflow, SDK import paths, route and asset exposure, and auth or user-config integration points.
|
||||
|
||||
## Reporting Issues
|
||||
|
||||
If you are reporting a bug, install problem, or performance issue, start with the admin workflow and reproduction steps, not Claude/Codex analysis.
|
||||
|
||||
Please include:
|
||||
|
||||
- What you were trying to do
|
||||
- Exact steps you took
|
||||
- What you expected to happen
|
||||
- What actually happened
|
||||
- What exact action is slow or broken (`save`, `scan`, `browse`, `import`, `playback`, etc.)
|
||||
- Whether it happens every time or only sometimes
|
||||
- The library, media type, filter, setting, or value involved
|
||||
- Version, branch, commit, and deployment details if you know them
|
||||
- Screenshots, recordings, or log snippets if relevant
|
||||
|
||||
If you used Claude/Codex for debugging, put that under `Technical notes` at the end. Suspected files, SQL output, stack traces, and root-cause theories can be helpful, but only after the workflow and repro steps are clear.
|
||||
|
||||
Use this template:
|
||||
|
||||
```text
|
||||
Goal:
|
||||
Steps:
|
||||
Expected:
|
||||
Actual:
|
||||
What is slow/broken:
|
||||
Scope:
|
||||
Version/branch:
|
||||
Deployment:
|
||||
Technical notes:
|
||||
```
|
||||
|
||||
### Make Targets
|
||||
|
||||
| Target | Description |
|
||||
|---|---|
|
||||
| `make build` | Build frontend + Go binary |
|
||||
| `make frontend` | Build frontend only |
|
||||
| `make dev-frontend` | Vite dev server with HMR |
|
||||
| `make dev-backend` | Run Go backend (integrated mode) |
|
||||
| `make dev-proxy` | Run a standalone proxy node |
|
||||
| `make dev-transcode` | Run a standalone transcode node |
|
||||
| `make migrate-create NAME=add_thing` | Create a timestamped Goose SQL migration |
|
||||
| `make migrate-validate` | Validate Goose migration files without touching a database |
|
||||
| `make migrate-status` | Show Goose migration status using Silo's bootstrapping runner |
|
||||
| `make migrate-up` | Apply pending Goose migrations using Silo's bootstrapping runner |
|
||||
| `make clean` | Remove build artifacts |
|
||||
|
||||
### Database Migrations
|
||||
|
||||
PostgreSQL schema migrations are managed by Goose. Migration SQL files live in
|
||||
`migrations/sql/` and use Goose annotations. Converted legacy migrations keep
|
||||
their original numeric versions so existing `schema_versions` rows can bootstrap
|
||||
cleanly into Goose without replaying old SQL. New migrations should be created
|
||||
with timestamped filenames:
|
||||
|
||||
```sh
|
||||
make migrate-create NAME=add_thing
|
||||
make migrate-validate
|
||||
```
|
||||
|
||||
Do not run `goose fix`; timestamped migrations are the repository policy because
|
||||
they avoid version collisions across parallel PRs. The existing `001`-style
|
||||
files are historical compatibility records, not the naming pattern for new work.
|
||||
Runtime migrations are applied by the integrated/API server only. Proxy and
|
||||
transcode modes never mutate schema.
|
||||
For existing installs, use `make migrate-status` and `make migrate-up` rather
|
||||
than invoking the Goose CLI directly; those targets copy legacy
|
||||
`schema_versions` rows into `public.goose_db_version` under the migration lock
|
||||
before reading or applying migrations. Set `ENV_FILE=path/to/.env` when the
|
||||
database URL should be read from a non-default env file.
|
||||
|
||||
### Running Tests
|
||||
|
||||
```sh
|
||||
# Go tests (uses testcontainers — Docker must be running)
|
||||
go test ./...
|
||||
|
||||
# Frontend tests
|
||||
cd web && bun test
|
||||
```
|
||||
|
||||
### Linting
|
||||
|
||||
```sh
|
||||
# Go
|
||||
golangci-lint run
|
||||
|
||||
# Frontend
|
||||
cd web && bun run lint
|
||||
cd web && bun run format:check
|
||||
```
|
||||
|
||||
## License & Trademarks
|
||||
|
||||
Silo's source code is licensed under the **GNU Affero General Public License
|
||||
v3.0 or later** (`AGPL-3.0-or-later`) — see [LICENSE](LICENSE).
|
||||
|
||||
The **Silo name, logo, and wordmark are trademarks of Silo Media L.L.C.** and
|
||||
are **not** covered by the AGPL. You're free to fork and redistribute the code,
|
||||
but forks and redistributions must not use the Silo brand as their identity and
|
||||
must remove or replace the brand assets. Publishing a Silo-branded app to an app
|
||||
store requires written permission. See [TRADEMARK.md](TRADEMARK.md) for what's
|
||||
permitted — including referential use like "compatible with Silo."
|
||||
|
||||
## Configuration
|
||||
|
||||
Silo requires only a `DATABASE_URL` when running from source or against external infrastructure. In the default Docker Compose path, the stack wires the database and Redis URLs for you. All other settings — libraries, metadata providers, transcoding, users — are managed through the admin UI after first launch.
|
||||
@@ -277,7 +127,7 @@ Silo requires only a `DATABASE_URL` when running from source or against external
|
||||
| `proxy` | Stream proxy node that connects to the shared deployment database and Redis |
|
||||
| `transcode` | HLS transcode worker node that connects to the shared deployment database and Redis |
|
||||
|
||||
## PostgreSQL Auto-Tuning
|
||||
### PostgreSQL Auto-Tuning
|
||||
|
||||
The default Docker Compose stack does not require a checked-in `postgresql.conf`.
|
||||
It enables Silo's [pgtune](https://github.com/le0pard/pgtune)-style OLTP tuning
|
||||
@@ -329,20 +179,93 @@ change PostgreSQL server settings. Settings already written with `ALTER SYSTEM`
|
||||
remain in `postgresql.auto.conf`; reset those PostgreSQL parameters if you later
|
||||
move fully to a custom `postgresql.conf`.
|
||||
|
||||
## Project Structure
|
||||
## Build from Source
|
||||
|
||||
If you prefer running Silo without Docker:
|
||||
|
||||
1. **Install prerequisites**: Go 1.24+, Bun 1.0+, PostgreSQL 18+, and FFmpeg.
|
||||
|
||||
2. **Start PostgreSQL and Redis** (skip if you already have them running)
|
||||
|
||||
```sh
|
||||
docker compose up -d postgres redis
|
||||
```
|
||||
|
||||
The main compose file still expects `MEDIA_ROOT` to be set even if you only want the bundled PostgreSQL and Redis services, so set that in `.env` first.
|
||||
|
||||
3. **Configure the database connection**
|
||||
|
||||
```sh
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Edit `.env` and set `DATABASE_URL` to point to your PostgreSQL instance.
|
||||
|
||||
4. **Build and run**
|
||||
|
||||
```sh
|
||||
make build
|
||||
./silo
|
||||
```
|
||||
|
||||
The server starts at `http://localhost:8080` by default. All other settings are configured through the admin UI.
|
||||
|
||||
## Reporting Issues
|
||||
|
||||
If you are reporting a bug, install problem, or performance issue, start with the admin workflow and reproduction steps, not Claude/Codex analysis.
|
||||
|
||||
Please include:
|
||||
|
||||
- What you were trying to do
|
||||
- Exact steps you took
|
||||
- What you expected to happen
|
||||
- What actually happened
|
||||
- What exact action is slow or broken (`save`, `scan`, `browse`, `import`, `playback`, etc.)
|
||||
- Whether it happens every time or only sometimes
|
||||
- The library, media type, filter, setting, or value involved
|
||||
- Version, branch, commit, and deployment details if you know them
|
||||
- Screenshots, recordings, or log snippets if relevant
|
||||
|
||||
If you used Claude/Codex for debugging, put that under `Technical notes` at the end. Suspected files, SQL output, stack traces, and root-cause theories can be helpful, but only after the workflow and repro steps are clear.
|
||||
|
||||
Use this template:
|
||||
|
||||
```text
|
||||
Goal:
|
||||
Steps:
|
||||
Expected:
|
||||
Actual:
|
||||
What is slow/broken:
|
||||
Scope:
|
||||
Version/branch:
|
||||
Deployment:
|
||||
Technical notes:
|
||||
```
|
||||
cmd/silo/ Entry point
|
||||
internal/
|
||||
api/ HTTP router, handlers, middleware
|
||||
auth/ JWT authentication and sessions
|
||||
catalog/ Media item, episode, season repositories
|
||||
config/ YAML + env var configuration
|
||||
jellycompat/ Jellyfin/Emby protocol compatibility
|
||||
metadata/ Plugin-driven metadata matching and enrichment
|
||||
playback/ Direct play, remux, transcode session management
|
||||
scanner/ Media file discovery and FFProbe
|
||||
worker/ Background jobs (scan, match, reconcile)
|
||||
web/ React + TypeScript frontend (Vite, Tailwind, shadcn/ui)
|
||||
migrations/sql/ Goose-managed PostgreSQL schema migrations
|
||||
```
|
||||
|
||||
## Contributing & Development
|
||||
|
||||
Silo is open source and contributions are welcome. See [DEVELOPMENT.md](DEVELOPMENT.md) for building from source in a dev workflow, running tests, database migrations, and project layout, and [CONTRIBUTING.md](CONTRIBUTING.md) for contribution expectations, merge request guidance, and the policy for AI-assisted submissions.
|
||||
|
||||
## Supporting Silo
|
||||
|
||||
Silo is an open-source hobby project, developed in spare time and funded out of pocket. If you'd like to support development, you can sponsor via [GitHub Sponsors](https://github.com/sponsors/quick104).
|
||||
|
||||
Donations go directly toward the costs of building and running the project:
|
||||
|
||||
- AI development tooling subscriptions (Claude, Codex) used to build and maintain Silo
|
||||
- Push notification relay infrastructure
|
||||
- Future development costs
|
||||
|
||||
Sponsoring is entirely optional — Silo is and will remain free and open source. Bug reports, contributions, and feedback are just as valuable.
|
||||
|
||||
## License & Trademarks
|
||||
|
||||
Silo's source code is licensed under the **GNU Affero General Public License
|
||||
v3.0 or later** (`AGPL-3.0-or-later`) — see [LICENSE](LICENSE).
|
||||
|
||||
The **Silo name, logo, and wordmark are trademarks of Silo Media L.L.C.** and
|
||||
are **not** covered by the AGPL. You're free to fork and redistribute the code,
|
||||
but forks and redistributions must not use the Silo brand as their identity and
|
||||
must remove or replace the brand assets. Publishing a Silo-branded app to an app
|
||||
store requires written permission. See [TRADEMARK.md](TRADEMARK.md) for what's
|
||||
permitted — including referential use like "compatible with Silo."
|
||||
|
||||
@@ -50,6 +50,7 @@ import (
|
||||
"github.com/Silo-Server/silo-server/internal/clientip"
|
||||
"github.com/Silo-Server/silo-server/internal/config"
|
||||
"github.com/Silo-Server/silo-server/internal/database"
|
||||
"github.com/Silo-Server/silo-server/internal/downloads"
|
||||
"github.com/Silo-Server/silo-server/internal/ebooks"
|
||||
evt "github.com/Silo-Server/silo-server/internal/events"
|
||||
"github.com/Silo-Server/silo-server/internal/historyimport"
|
||||
@@ -68,6 +69,7 @@ import (
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
"github.com/Silo-Server/silo-server/internal/nodeconfig"
|
||||
"github.com/Silo-Server/silo-server/internal/nodepool"
|
||||
"github.com/Silo-Server/silo-server/internal/noderecipe"
|
||||
"github.com/Silo-Server/silo-server/internal/nodesessions"
|
||||
"github.com/Silo-Server/silo-server/internal/notifications"
|
||||
"github.com/Silo-Server/silo-server/internal/opslog"
|
||||
@@ -95,6 +97,7 @@ import (
|
||||
"github.com/Silo-Server/silo-server/internal/userdb"
|
||||
"github.com/Silo-Server/silo-server/internal/userstore"
|
||||
"github.com/Silo-Server/silo-server/internal/userstore/pgstore"
|
||||
"github.com/Silo-Server/silo-server/internal/watchlist"
|
||||
"github.com/Silo-Server/silo-server/internal/watchstate"
|
||||
"github.com/Silo-Server/silo-server/internal/watchsync"
|
||||
watchmdblist "github.com/Silo-Server/silo-server/internal/watchsync/providers/mdblist"
|
||||
@@ -614,6 +617,10 @@ func main() {
|
||||
} else {
|
||||
srv := transcodenode.NewServer(watcher, tracker)
|
||||
srv.SetFFmpegLogSink(playback.NewSlogFFmpegLogSink(slog.Default(), nodeID))
|
||||
// Read jellycompat reconstruction recipes central wrote at transcode
|
||||
// start, so this node can rebuild a Jellyfin transcode after its own
|
||||
// restart (the node hop token is recipe-less). Shares the offload Redis.
|
||||
srv.SetRecipeStore(noderecipe.NewStore(redisClient, 0))
|
||||
handler = srv.Handler()
|
||||
}
|
||||
|
||||
@@ -1397,6 +1404,7 @@ func main() {
|
||||
)
|
||||
userStoreProvider = notifications.WrapUserStoreProvider(userStoreProvider, notificationSystem)
|
||||
deps.Notifications = notificationSystem
|
||||
|
||||
if libraryIngestExecutor != nil {
|
||||
libraryIngestExecutor.SetAvailabilityDetector(notificationSystem.Detector)
|
||||
}
|
||||
@@ -1449,6 +1457,15 @@ func main() {
|
||||
}
|
||||
})
|
||||
}
|
||||
// Auto-remove fully-watched items from the watchlist (standalone behavior,
|
||||
// default-on per profile), propagating removals to connected providers.
|
||||
if itemRepo != nil && episodeRepo != nil && userStoreProvider != nil {
|
||||
maintainer := watchlist.NewMaintainer(userStoreProvider, itemRepo, episodeRepo)
|
||||
if watchProviderService != nil {
|
||||
maintainer.WithListEventDispatcher(watchProviderService)
|
||||
}
|
||||
deps.WatchCompletionObserver = maintainer
|
||||
}
|
||||
deps.SessionMgr = sessionMgr
|
||||
deps.PlaybackRealtimeHub = playback.NewRealtimeHub()
|
||||
if chapterThumbService != nil && deps.S3Public != nil {
|
||||
@@ -1724,6 +1741,41 @@ func main() {
|
||||
}
|
||||
taskMgr.Register(tasks.NewActivityLogCleanupTask(deps.DB, settingsRepo, activityPM))
|
||||
taskMgr.Register(tasks.NewOperationalLogCleanupTask(deps.DB, settingsRepo, opsPM))
|
||||
if deps.FileRepo != nil {
|
||||
// Download prepare-to-file pipeline (Phase 3): a durable, leased encode
|
||||
// queue hosted on the task manager. Built here (before Start) and shared
|
||||
// with the API via deps so the download service can enqueue jobs.
|
||||
artifactMgr := downloads.NewArtifactManager(
|
||||
downloads.NewArtifactRepository(deps.DB),
|
||||
downloads.NewRepository(deps.DB),
|
||||
deps.FileRepo,
|
||||
downloads.NewPlaybackPreparer(),
|
||||
deps.NodeID,
|
||||
func() *config.Config {
|
||||
if deps.LiveConfig != nil {
|
||||
if c := deps.LiveConfig(); c != nil {
|
||||
return c
|
||||
}
|
||||
}
|
||||
return deps.Config
|
||||
},
|
||||
func(ctx context.Context, d *downloads.Download) {
|
||||
if deps.EventsHub == nil {
|
||||
return
|
||||
}
|
||||
_ = deps.EventsHub.PublishJSON(ctx, evt.ChannelUserState, "download", map[string]any{
|
||||
"download_id": d.ID,
|
||||
"status": d.Status,
|
||||
"media_item_id": d.ContentID,
|
||||
"format": d.Format,
|
||||
}, evt.PublishOptions{UserID: d.UserID, ProfileID: d.ProfileID})
|
||||
},
|
||||
)
|
||||
encodeTask := tasks.NewEncodeDownloadArtifactsTask(artifactMgr)
|
||||
artifactMgr.SetKick(func() { _ = taskMgr.RunTask(appCtx, encodeTask.Key()) })
|
||||
taskMgr.Register(encodeTask)
|
||||
deps.ArtifactManager = artifactMgr
|
||||
}
|
||||
if notificationSystem != nil {
|
||||
taskMgr.Register(tasks.NewSeedContentAvailabilityTask(notificationSystem))
|
||||
taskMgr.Register(tasks.NewRebuildReleaseInterestTask(notificationSystem))
|
||||
@@ -1881,6 +1933,8 @@ func main() {
|
||||
AccessResolver: audiobooks.NewABSAccessResolver(absUserRepo, userStoreProvider),
|
||||
Recs: recommendations.NewRepo(deps.DB),
|
||||
Detail: absDetailSvc,
|
||||
SessionMgr: sessionMgr,
|
||||
SessionSyncer: deps.SessionSyncer,
|
||||
}
|
||||
absH := audiobooksService.BuildABSHandler(absHDeps)
|
||||
deps.ABSHandler = absH
|
||||
@@ -2135,6 +2189,10 @@ func main() {
|
||||
JWTSecret: cfg.Auth.JWTSecret,
|
||||
RecWorker: recWorker,
|
||||
FrontendFS: deps.FrontendFS,
|
||||
// Hand remote-transcode recipes to the shared recipe store so a dedicated
|
||||
// transcode node that restarts can rebuild a jellycompat session.
|
||||
RecipeNodeStore: noderecipe.NewStore(apiRedisClient, 0),
|
||||
SessionSyncer: deps.SessionSyncer,
|
||||
}
|
||||
|
||||
// Wire direct dependencies when DB is available.
|
||||
@@ -2170,8 +2228,20 @@ func main() {
|
||||
compatDeps.FolderRepo = folderRepo
|
||||
compatDeps.SessionMgr = sessionMgr
|
||||
compatDeps.UserStoreProvider = userStoreProvider
|
||||
compatDeps.WatchCompletionObserver = deps.WatchCompletionObserver
|
||||
compatDeps.SettingsRepo = settingsRepo
|
||||
compatDeps.PersonRepo = personRepo
|
||||
compatSearchService := catalog.NewCatalogSearchService(
|
||||
appCtx,
|
||||
settingsRepo,
|
||||
itemRepo,
|
||||
catalog.NewSearchIndexEventRepository(deps.DB),
|
||||
deps.CatalogSearchVectorizer,
|
||||
)
|
||||
if compatSearchService != nil {
|
||||
compatSearchService.StartCoverageRefresh(appCtx)
|
||||
compatDeps.CatalogSearchProvider = compatSearchService.Provider()
|
||||
}
|
||||
|
||||
if deps.S3Public != nil {
|
||||
compatDeps.PosterPresigner = deps.S3Public
|
||||
|
||||
@@ -0,0 +1,787 @@
|
||||
# Restart-resilient playback (TR-lease)
|
||||
|
||||
How Silo keeps playback alive across a server restart, reconnect, or partial
|
||||
stream — and how the design arrived at **TR-lease** (token-carried
|
||||
reconstruction + a server-side session-deny marker for revocation).
|
||||
|
||||
> **Status: the revocation half is deferred to a future PR.** Only the
|
||||
> token-carried **reconstruction** core shipped in this PR. The **session-deny /
|
||||
> stream-revocation** mechanism described throughout (the `silo:streamauth:<sid>`
|
||||
> deny marker, the proxy's `Allowed()` enforcement, and the admin
|
||||
> Stop/Terminate deny write) is **not present in the current implementation**.
|
||||
> Admin Terminate and user Stop tear down the live in-memory session and the
|
||||
> ffmpeg producer, but they do **not** prevent a still-valid stream token from
|
||||
> reconstructing the session until its 24h TTL expires. Sections describing that
|
||||
> mechanism are kept for design continuity and are individually flagged as
|
||||
> deferred below.
|
||||
|
||||
This document is the consolidated design record. It folds together four earlier
|
||||
working notes so a future engineer can see the full evolution, the options
|
||||
weighed, the issues raised across review rounds, the variables traded off, and
|
||||
why the final shape was chosen. The superseded notes were:
|
||||
|
||||
- *unified-playback-reconstruct* — the reconstruct core (PR #174), "RC".
|
||||
- *token-carried-playback-reconstruct* — the storage evolution, "TR".
|
||||
- *async-recipe-card-playback* — a monitoring-preserving middle path, "RC-async".
|
||||
- *playback-reconstruct-options-comparison* — the goals/options matrix.
|
||||
|
||||
Paths are repository-relative; assume the repository root is the cwd.
|
||||
|
||||
---
|
||||
|
||||
## 1. The problem
|
||||
|
||||
Before this work, a missing in-memory playback session was a `404`. A server
|
||||
restart (deploy, crash, OOM) wiped the in-memory session map, so every in-flight
|
||||
stream died: the client's next manifest/segment/byte-range request hit a session
|
||||
that no longer existed and playback stopped hard. The same gap appeared on any
|
||||
front-end that had never held the session (horizontal scale) and on a dedicated
|
||||
transcode node reboot.
|
||||
|
||||
Core priorities for the fix (from the repo guidelines): **performance first,
|
||||
reliability first, predictable behavior under load and during failures**. The
|
||||
hot path (per-segment serve) must stay off central/Postgres; bytes must stay
|
||||
offloaded to nodes; correctness and robustness win over convenience.
|
||||
|
||||
---
|
||||
|
||||
## 2. The core insight: reconstruct, don't rehydrate
|
||||
|
||||
A missing session is a **reconstruct trigger, not a 404**. The client always
|
||||
re-requests the next chunk after an outage, and its request already states where
|
||||
it is (an HTTP `Range`, a `?seek=`, or a `seg_NNNNN` number). So the server
|
||||
rebuilds the session just-in-time from two things:
|
||||
|
||||
1. a small **durable descriptor** (identity, ownership, and the byte-affecting
|
||||
encode parameters — the "recipe"), and
|
||||
2. the **position the client supplies** on the triggering request.
|
||||
|
||||
Nothing live is serialized (no ffmpeg process, context, channels, or log sink).
|
||||
This is the contribution of PR #174 ("RC"), which also unified the native and
|
||||
Jellyfin-compat paths behind one `playback.TranscodeManager` so the
|
||||
card-lifetime rules, the reconstruct concurrency cap, and the node-affinity rule
|
||||
live in exactly one place. The reconstruct skeleton —
|
||||
`LoadOrReconstructSession`, `ReconstructSession` / `ReconstructTranscode`, the
|
||||
single-flight + `reconstructSem` cap that paces the post-restart thundering
|
||||
herd, `RegisterReconstructed`, the `CloseProcess`/`Close` split — is the
|
||||
foundation every later option reused. **It was kept; only the descriptor storage
|
||||
and the revocation model evolved.**
|
||||
|
||||
The six routes that must survive a restart:
|
||||
|
||||
| # | Route | Tier | How it is rebuilt |
|
||||
|---|-------|:----:|-------------------|
|
||||
| 1 | native direct | 1 | client `Range` re-serve; descriptor = identity |
|
||||
| 2 | native remux | 1 | re-spawn ffmpeg at `?seek`; descriptor = id + audio |
|
||||
| 3 | native transcode | 2 | rebuild `Session` + ffmpeg seeked to the requested segment; descriptor = full encode opts |
|
||||
| 4 | jellycompat direct | 1 | compat store + `Range` re-serve |
|
||||
| 5 | jellycompat remux | 1 | compat store + `?seek` |
|
||||
| 6 | jellycompat transcode | 2 | compat store + shared-manager reconstruct |
|
||||
|
||||
Tier 1 = stateless re-serve (client re-supplies position). Tier 2 = rebuild
|
||||
ffmpeg. Routes 4–6 always retain a durable central compat store
|
||||
(`jellycompat_playback_sessions`): third-party Jellyfin clients cannot
|
||||
round-trip a native descriptor, so the compat layer keeps the full
|
||||
`PlaybackSession` (route + media source resolved before native reconstruct
|
||||
runs).
|
||||
|
||||
---
|
||||
|
||||
## 3. The two axes the options differ on
|
||||
|
||||
Every option is a point on a 2-D grid: **where the descriptor lives** (vertical)
|
||||
× **how revocation is bounded** (horizontal).
|
||||
|
||||
```text
|
||||
AUTHORITY / REVOCATION MODEL ----------------------->
|
||||
re-resolve at long token short TTL +
|
||||
reconstruct (24h, expire) refresh checkpoint
|
||||
D central +----------------+----------------+----------------+
|
||||
E Postgres row | RC (#174) | (n/a) | RC-async |
|
||||
S +----------------+----------------+----------------+
|
||||
C client/node- | (n/a) | TR-24h | TR + re-mint |
|
||||
R carried token | | | |
|
||||
I +----------------+----------------+----------------+
|
||||
P
|
||||
T top row = descriptor stays central (node serves bytes but cannot
|
||||
| self-reconstruct → node-restart gap)
|
||||
| bottom = recipe travels to the node (node self-sufficient, no central record)
|
||||
```
|
||||
|
||||
**TR-lease is a fourth point off this grid:** bottom row (token-carried, 24h),
|
||||
but the revocation column is replaced by a *server-side deny marker* — on an
|
||||
admin kill, central would write a per-session deny to Redis that the node
|
||||
enforces by withholding bytes. Revocation is decoupled from the token TTL and
|
||||
needs no client refresh. (An earlier iteration polled and re-resolved every
|
||||
stream on a timer; that was cut to the event-driven session-deny — see §7 and
|
||||
the §8 design note.)
|
||||
|
||||
> **Status: deferred to a future PR.** The deny-marker column described above is
|
||||
> the design target, not the current shipped behavior. As shipped, the
|
||||
> token-carried row offers no revocation tighter than the 24h token TTL on the
|
||||
> node path; admin Terminate and user Stop only tear down the live session and
|
||||
> ffmpeg producer.
|
||||
|
||||
---
|
||||
|
||||
## 4. Goals weighed
|
||||
|
||||
| ID | Goal |
|
||||
|----|------|
|
||||
| G1 | Hot path (per-segment serve) stays off central/Postgres |
|
||||
| G2 | Bytes offloaded to nodes (served from disk, signature-verified, never central) |
|
||||
| G3 | Restart resiliency across all 6 routes |
|
||||
| G4 | Two-factor ownership (auth caller + ownership rebind; refuse `userID==0`/mismatch) |
|
||||
| G5 | Revocation latency (how fast a banned user / pulled access stops playing) |
|
||||
| G6 | Single-box: zero external store required for reconstruction |
|
||||
| G7 | Restart runway (descriptor stays valid longer than the outage) |
|
||||
| G8 | Multi-front-end / verify-anywhere (no integrated-transcode split-brain) |
|
||||
| G9 | Observability — durable, identity-rich active-stream record |
|
||||
| G10 | Low operational surface (tables, migrations, stores, endpoints) |
|
||||
| G11 | Node-restart recovery |
|
||||
| G12 | No new client coordination required |
|
||||
| G13 | Small signing-key blast radius |
|
||||
| G14 | Cleanup correctness (never wipe a live session's segment dir) |
|
||||
|
||||
A note on G9 (observability): the *live* "who is streaming now" view does **not**
|
||||
depend on the descriptor strategy. Every offload node calls the Redis-backed
|
||||
node-session tracker (`internal/nodesessions/tracker.go`) on every serve, tied to
|
||||
actual byte-serving (so a non-cooperative client or replayed token still shows
|
||||
up) and surviving a central restart. G9 scores only the *durable, identity-rich*
|
||||
(who + what + recipe) slice on top of that.
|
||||
|
||||
---
|
||||
|
||||
## 5. The options, and the comparison
|
||||
|
||||
Legend: ✓ meets · ✓\* minor asterisk · ~ partial · ✗ fails · ✓✓ best-in-class
|
||||
|
||||
| Goal | `main` | RC #174 | TR-24h | TR + re-mint | RC-async | **TR-lease** |
|
||||
|------|:------:|:-------:|:------:|:------------:|:--------:|:------------:|
|
||||
| G1 hot path free of central DB | ✓ | ✓\* | ✓ | ✓ | ✓ | ✓ |
|
||||
| G2 bytes offloaded | ✓ | ✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ |
|
||||
| G3 restart resiliency (6/6) | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| G4 two-factor ownership | ✓ | ✓ | ~ | ~ | ~ | ✓~ |
|
||||
| G5 revocation latency | ~ | ~ | ✗ (≤24h) | ✓ (≤TTL) | ✓✓ | ✓✓ |
|
||||
| G6 single-box, no external store | ✓ | ✗ | ✓✓ | ~ | ~ | ~ |
|
||||
| G7 restart runway | n/a | ✓ (30m) | ✓✓ (24h) | ~ | ~ | ✓✓ (24h) |
|
||||
| G8 multi-front-end | ✗ | ~ | ✓~ | ✓~ | ✓~ | ✓~ |
|
||||
| G9 durable identity-rich record | ✗ | ✓✓ | ✗ | ✗ | ✓ | ~ |
|
||||
| G10 low operational surface | ✓✓ | ~ | ✓ | ~ | ✗ | ~ |
|
||||
| G11 node-restart recovery | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ |
|
||||
| G12 no new client coordination | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ |
|
||||
| G13 small signing-key blast radius | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
|
||||
| G14 cleanup correctness | ✓ | ✓✓ | ~ | ~ | ~ | ~ |
|
||||
|
||||
The TTL/revocation strategies head-to-head. **Note: the session-deny column is
|
||||
the deferred design target — see §7; as shipped, the node path has no revocation
|
||||
tighter than the 24h token TTL.**
|
||||
|
||||
| Strategy | Revocation latency | Restart runway | Client refresh? | Central on refresh? |
|
||||
|----------|--------------------|----------------|-----------------|---------------------|
|
||||
| Re-resolve at reconstruct (RC) | on reconstruct; node hop ≤24h | 30 min | no | only on reconstruct |
|
||||
| Leave 24h, expire only (TR-24h) | up to 24h | 24h | no | never |
|
||||
| Short TTL + refresh + denylist (RC-async / TR item B) | ≤~5 min | ~90 s grace | **yes** | every ~3.5 min/session |
|
||||
| 24h token + session-deny marker (TR-lease — **deferred**) | *design target:* admin kill cuts node bytes on next serve; passive ban = next-play / ≤24h. *As shipped:* node path bounded by ≤24h token only | 24h | **no** | nothing on the serve path; one write per admin-kill event |
|
||||
|
||||
---
|
||||
|
||||
## 6. Why each non-final option was set aside
|
||||
|
||||
- **RC (PR #174, the recipe card in Postgres).** Correct and the basis of
|
||||
everything here, but it couples reconstruction to a shared per-session Postgres
|
||||
store: a write on every start, per audio/quality change, and a ≤1/min refresh,
|
||||
with resilience tied to the DB being reachable. It does not give the single-box
|
||||
zero-dependency property, and the node hop's 24h token stays unrevocable
|
||||
anyway. RC's reconstruct *core* was kept; its *storage* was the thing to evolve.
|
||||
|
||||
- **TR-24h (token-carried, leave the token to expire).** Wins the single-box case
|
||||
(zero external dependency) and removes the per-start DB write, but revocation is
|
||||
bounded by *nothing tighter than the 24h token TTL* on the node path — a banned
|
||||
user keeps node-served direct/remux for up to 24h. This is actually the
|
||||
*status quo* for the node path even pre-branch (the proxy verifies by signature
|
||||
alone, no DB), so TR-24h only adds identity claims to the token without
|
||||
improving revocation. Not acceptable as the endpoint.
|
||||
|
||||
- **TR + re-mint (short TTL + authenticated playlist refresh + denylist), and
|
||||
RC-async (the same revocation idea but keeping an async-written Postgres mirror
|
||||
for observability).** Both bound revocation to ~5 min by having the client
|
||||
refetch a stable `RequireAuth` playlist that re-checks access and re-mints
|
||||
short-lived segment URLs. The cost: a **hard client-coordination requirement**
|
||||
(silo-android / silo-apple / Jellyfin clients must implement the refresh flow),
|
||||
a shrunken restart runway (~90 s grace instead of 30 min — if central is down
|
||||
past the grace, playback stalls), and a new authenticated re-mint transport
|
||||
that does not exist on the supported proxy/node topology. RC-async additionally
|
||||
keeps a write pipeline + denylist + the central row (the highest operational
|
||||
surface of all). They solved revocation by pushing work onto the client and
|
||||
shortening the runway — the opposite of the reliability priorities.
|
||||
|
||||
The open item across TR and RC-async was always **revocation enforcement** ("item
|
||||
B"): bounding revocation needs *either* a per-segment token check on the hot path
|
||||
*or* an authenticated re-mint checkpoint the client drives. TR-lease removes that
|
||||
dilemma.
|
||||
|
||||
---
|
||||
|
||||
## 7. TR-lease: the chosen design
|
||||
|
||||
TR-lease keeps TR's 24h token (so the restart runway is untouched and no client
|
||||
refresh flow is needed) and adds a server-side **session-deny marker** for the
|
||||
one revocation case the node path cannot otherwise cover — no client
|
||||
participation, no steady-state cost.
|
||||
|
||||
**Reconstruction (token-carried).** The signed stream token *is* the durable
|
||||
descriptor. Its claims carry the full byte-affecting recipe plus `uid/pid/mfid`
|
||||
ownership lookup keys. A front-end that lost its in-memory session rebuilds
|
||||
ffmpeg from the token the client re-presents — no shared per-session store, zero
|
||||
external dependency for reconstruction on a single box. `HWAccel`/`HWDevice` are
|
||||
deliberately *not* carried; they are re-resolved from live config so an operator
|
||||
config change applies to reconstructed sessions too. Ownership re-binds to the
|
||||
live authenticated caller and refuses `userID==0` / mismatch — the token's `uid`
|
||||
is never trusted alone (a leaked URL is useless without the owner's auth session
|
||||
on the native path).
|
||||
|
||||
**Revocation (session-deny marker).**
|
||||
|
||||
> **Status: deferred to a future PR.** Everything in this subsection (the
|
||||
> `silo:streamauth:<sid>` deny marker, the admin Stop/Terminate deny write, and
|
||||
> the node-side `Allowed()` enforcement) describes the design target and is **not
|
||||
> present in the current implementation**. As shipped, admin Terminate and user
|
||||
> Stop tear down the live in-memory session and the ffmpeg producer, but a
|
||||
> still-valid stream token can reconstruct the session until its 24h TTL expires;
|
||||
> there is no node-side byte-withholding and no guaranteed/instant revocation on
|
||||
> the node path. The design below is retained for when revocation lands.
|
||||
|
||||
The revocation surface was deliberately
|
||||
kept to the *one* case the offloaded topology cannot otherwise cover, rather than
|
||||
a periodic re-check of every stream. An admin Stop/Terminate would write
|
||||
`silo:streamauth:<sid> = deny` (TTL ≥ the 24h token lifetime); the offload node
|
||||
would read it (a node-local Redis GET, sub-ms) before serving any bytes and
|
||||
return `403` with no bytes on a hit. Enforcement is **byte-withholding, not
|
||||
client cooperation** — a revoked client that ignores the 403 keeps hitting a
|
||||
wall.
|
||||
|
||||
Why this would be sufficient, and what it deliberately does *not* do:
|
||||
|
||||
- **New playback is already blocked at central** — `/playback/start` is
|
||||
`RequireAuth` + access check, so a ban/scope change always takes effect on the
|
||||
next play, everywhere, with no marker needed.
|
||||
- **Cooperative clients already stop** — central holds the realtime WebSocket and
|
||||
can drop it + send a stop on a ban.
|
||||
- **The only thing needing a hard node-side cut** is the *explicit, session-scoped
|
||||
admin kill* of a *non-cooperative* client holding a valid token on a
|
||||
node-served direct/remux stream (no producer to kill). That is exactly what the
|
||||
(deferred) session-deny marker would cover; until it lands this case is bounded
|
||||
by the ≤24h token like the passive bans below.
|
||||
- **Not covered (accepted limitation):** a *passive* ban or *partial* access
|
||||
change (lost one library, lowered rating) does NOT hard-revoke an
|
||||
already-running, non-cooperative node stream. It is enforced at next play and
|
||||
otherwise bounded by the ≤24h token. Closing this would require either a
|
||||
periodic re-resolver (a poll over every active stream — rejected as constant
|
||||
DB load for a rare case) or per-content deny enumeration; both were judged not
|
||||
worth the machinery for a self-hosted server. A user-scoped deny was rejected
|
||||
too: it over-blocks content the viewer still legitimately has access to.
|
||||
|
||||
**Fail-open on absence (deferred).** In the design, the normal case is *no*
|
||||
marker, which serves; a Redis error also serves; only a present deny withholds
|
||||
bytes. This is a deliberate reliability-first choice: availability over
|
||||
revocation latency.
|
||||
|
||||
**Why this shape.** It keeps token-carried reconstruction's single-box and
|
||||
verify-anywhere properties and would add revocation with no client coordination
|
||||
(G12) and essentially no steady-state cost — the marker is written only on the
|
||||
rare admin-kill event, never on a timer. It is, in effect, **TR-24 plus a single
|
||||
session-scoped kill switch**. The kill switch is **deferred**; what shipped is
|
||||
TR-24 (token-carried reconstruction, ≤24h token, no node-side revocation).
|
||||
|
||||
---
|
||||
|
||||
## 8. What was implemented, and where
|
||||
|
||||
This PR ships **Commit 1 only** (token-carried reconstruction). **Commit 2 (the
|
||||
node session-deny marker) is deferred to a future PR** — the `internal/streamauth`
|
||||
package, the proxy `Allowed()` guard, and the admin deny write described below
|
||||
are **not present in the current implementation**.
|
||||
|
||||
**Commit 1 — token-carried reconstruction; retire `transcode_recipes`. (shipped)**
|
||||
- `streamtoken.Claims` gains the recipe + `uid/pid/mfid`
|
||||
(`internal/streamtoken/token.go`).
|
||||
- `playback.RecipeCard` projects to/from claims (`ToClaims` /
|
||||
`RecipeCardFromClaims`); the `Reconstruct*` front door takes the decoded recipe
|
||||
instead of reading Postgres (`internal/playback/recipecard.go`,
|
||||
`transcode_manager.go`).
|
||||
- Native serve URLs carry the token as `?st=`; the manifest rewriter already
|
||||
appends `RawQuery` to segment URIs, so segments inherit it with no route change
|
||||
(`internal/api/handlers/playback.go`, `stream.go`). The integrated server is
|
||||
hit directly, so the query-stripping-proxy concern that motivates a path
|
||||
segment does not apply there; the proxy/node path keeps the token in the URL
|
||||
path as before.
|
||||
- Jellycompat folds the recipe into its durable compat store
|
||||
(`PlaybackSession.Recipe`) since Jellyfin clients cannot round-trip a native
|
||||
token (`internal/jellycompat/...`).
|
||||
- Segment-dir cleanup switches from the card index to in-memory liveness (live
|
||||
map + in-flight reconstruct set + age past the max token TTL); a dedicated node
|
||||
keeps its boot-time full wipe (`internal/playback/transcode_cleanup.go`,
|
||||
`transcode_manager.go`).
|
||||
- `PostgresRecipeStore`, the `RecipeStore` interface, and the unmerged
|
||||
`transcode_recipes` migration are deleted.
|
||||
|
||||
**Commit 2 — node session-deny marker. (DEFERRED — not in this PR)**
|
||||
|
||||
> The items below describe the planned revocation commit. They were removed from
|
||||
> this PR and deferred to a future one; none of these symbols exist in the
|
||||
> current implementation. The admin Stop/Terminate path still tears down the live
|
||||
> session and ffmpeg producer, but writes no deny marker, so a valid token can
|
||||
> reconstruct until its 24h TTL expires.
|
||||
|
||||
- `internal/streamauth`: a deny-only Redis `Store` — `Deny(sid)` (TTL ≥ token
|
||||
lifetime) and a fail-open `Allowed(sid)` reader. No poller, no allow-leases, no
|
||||
access re-resolver.
|
||||
- The node guard in `internal/proxy/server.go` (`verifyToken` consults
|
||||
`Allowed`).
|
||||
- An admin Stop/Terminate writes the deny marker
|
||||
(`internal/api/handlers/admin_playback_control.go` → `PlaybackHandler.LeaseDenier`).
|
||||
- `nodesessions.SessionInfo` carries the numeric ownership keys
|
||||
(`auth_user_id`/`profile_id`/`media_file_id`), populated by the node from the
|
||||
verified token, to enrich the live admin "active streams" view (who/what, not
|
||||
just session id) — see §10 monitoring.
|
||||
- A single integrated box reads no marker and needs none — the producer-kill +
|
||||
session removal on Terminate is the stop there.
|
||||
|
||||
> Design note (about the deferred commit 2): an earlier iteration added a central
|
||||
> *revalidator* that re-resolved access for every active stream every ~2 min and
|
||||
> wrote allow/deny leases. It was dropped in favour of the event-driven
|
||||
> session-deny above: the poll re-derived, at a constant all-streams DB cost,
|
||||
> signals the
|
||||
> system already emits (new playback is gated at `/playback/start`; admin kill is
|
||||
> an explicit event), and its only unique coverage — sub-24h *passive* revocation
|
||||
> of an in-flight node stream — was judged not worth the machinery (see §7).
|
||||
|
||||
---
|
||||
|
||||
## 9. Issues raised across review, and their disposition
|
||||
|
||||
From two adversarial review rounds on the reconstruct core and the storage
|
||||
evolution:
|
||||
|
||||
- **Integrated-transcode split-brain across front-ends (P-1 / A).** The
|
||||
`SessionManager` + reconstruct single-flight are per-process and the descriptor
|
||||
carries no owning-node id, so without sticky affinity two front-ends can run
|
||||
divergent ffmpeg. **Disposition: deferred (owner-identity claim).** Supported
|
||||
topology is dedicated `--mode=transcode` nodes (routed via `tnode`, no
|
||||
split-brain) + single-box / LB-session-affinity integrated transcode. The
|
||||
caveat is documented, not silently assumed.
|
||||
- **24h node-path token is unrevocable (P-2).** The (deferred, see §7–§8)
|
||||
session-deny marker would let the explicit admin kill cut node bytes on the next
|
||||
serve. **Until that ships, this remains open:** the node-path token is
|
||||
unrevocable, so admin kill and passive bans / partial access changes are all
|
||||
bounded by next-play + the ≤24h token.
|
||||
- **Dedicated transcode-node restart (P-3 / tr-10). Fixed for both paths.**
|
||||
*Native:* the proxy forwards the verified stream token to the transcode node
|
||||
(`X-Silo-Stream-Token`); on a manifest/segment miss the node re-verifies the
|
||||
token, decodes the recipe, and self-reconstructs ffmpeg seeked to the requested
|
||||
segment — single-flighted and concurrency-capped exactly like the integrated
|
||||
reconstruct. *Jellycompat:* the node-hop token is server-minted and *could* carry the
|
||||
recipe like native, but it deliberately does not — the recipe is mutated in
|
||||
place under a stable session id (a Jellyfin audio/subtitle switch can arrive as
|
||||
a `/Sessions/Playing/Progress` report that restarts ffmpeg without re-minting
|
||||
the client's token), and a third-party Jellyfin client cannot be driven to
|
||||
refresh a stale token, so a token *snapshot* could reconstruct a stale
|
||||
rendition. Central instead writes the recipe to a shared Redis recipe store
|
||||
(`internal/noderecipe`) keyed by upstream session id and overwrites it on every
|
||||
switch, and the node (which cannot reach Postgres) reads the *current* recipe on
|
||||
the reconstruct miss — over the same Redis the session tracker already uses (and
|
||||
the deferred deny-lease would use), so no central URL is needed. The boot-time
|
||||
full segment-dir wipe
|
||||
stands; reconstruct re-transcodes from the requested segment. See §10 for the
|
||||
full rationale (and why this is *not* a token-carryable case).
|
||||
- **`userID==0` tolerance / "auth optional" (P-5).** Reconstruct hard-rejects
|
||||
`userID==0` and a card whose session id does not match the URL.
|
||||
- **Mutable recipe under a stable id (D).** A parameter change re-mints a new
|
||||
manifest + token and the client reloads, so the common case self-heals. The
|
||||
only residual is a restart *during* the brief switch transition with a prior
|
||||
token in flight: it reconstructs the prior *rendition* (access is re-resolved,
|
||||
only the rendition can be stale). TTL-bounded, self-healing on reload; minor,
|
||||
accepted.
|
||||
- **Cleanup liveness (F / tr-9).** Each process owns its `TranscodeDir`, so the
|
||||
liveness signal is the in-process live map + the in-flight reconstruct set +
|
||||
age-past-max-TTL; a dir is reaped only when absent from both and older than any
|
||||
surviving token. No cross-process enumeration-failure mode for the supported
|
||||
topology.
|
||||
- **Signing-key blast radius (E / tr-7, G13).** A single `JWTSecret` signs native
|
||||
auth, the stream token, and the node bearer, so a node-secret leak can forge
|
||||
native logins. **Accepted risk** on the trusted-node precondition. **Deferred
|
||||
hardening:** split the stream-token secret from the native-auth secret, then
|
||||
asymmetric stream-token signing before any untrusted/edge node.
|
||||
|
||||
---
|
||||
|
||||
## 10. Residuals and follow-ups
|
||||
|
||||
- **Revocation of an in-flight node stream (all cases, until deny-lease ships).**
|
||||
Because the session-deny marker is deferred (§7–§8), *no* ban — admin kill,
|
||||
passive ban, or partial access change — hard-cuts an already-running,
|
||||
non-cooperative node-served stream; all are enforced at next play, bounded by
|
||||
the ≤24h token. The future deny-lease closes the *admin-kill* case; passive
|
||||
bans would still need a per-content deny enumeration on the access-change event,
|
||||
or accepting it. See §7.
|
||||
- **Monitoring.** Two distinct questions:
|
||||
- *Live "who is watching what right now"* — covered. Every node serve writes a
|
||||
record to the Redis tracker (`silo:sessions:*`), now enriched with
|
||||
`auth_user_id` / `media_file_id` from the verified token, so the admin
|
||||
active-streams view answers who/what/where, tied to real byte-serving and
|
||||
surviving a central restart. Integrated (no-node) sessions show in central's
|
||||
in-memory session list.
|
||||
- *Durable historical/audit ("who watched what last week")* — NOT provided by
|
||||
this subsystem (there is no longer a durable per-session recipe record, by
|
||||
design — G9 was the deliberate trade for token-carried reconstruction). That
|
||||
history lives in the separate watch-progress / scrobble system.
|
||||
- **Multiple viewers on one stream token.** The token is per playback session, so
|
||||
a shared/leaked stream URL lets several clients pull bytes under one session id.
|
||||
They **collapse to a single entry** in the tracker (one `sid`) — monitoring
|
||||
cannot distinguish or count them — and the (deferred) session-deny would cut
|
||||
them as a group, not individually. Detecting/limiting this needs a per-account
|
||||
concurrency cap or
|
||||
per-device session ids (device binding); deferred. Do not bind to client IP
|
||||
(mobile / CGNAT).
|
||||
- **Token in logs (tr-1).** Scrub the `?st=` token / path token from request
|
||||
loggers (the jellycompat logger logs `RawQuery`; the native logger omits it).
|
||||
Short TTL + signature bound the exposure.
|
||||
- **Owner-identity (A).** Required before multi-front-end *integrated* transcode
|
||||
without sticky affinity; defines owner election, self-URL discovery, a
|
||||
peer-forward endpoint, and failover.
|
||||
- **Node-side reconstruct (P-3). Done** for native (token-forwarded) and
|
||||
jellycompat (Redis recipe handoff via `internal/noderecipe`: central writes the
|
||||
recipe at remote-transcode start, the node reads it on a reconstruct miss).
|
||||
Precondition: central and the nodes share the same Redis (already required for
|
||||
the session tracker, and for the deferred deny-lease). Residual: the recipe
|
||||
store is consulted
|
||||
only for a jellycompat token, so a node restart still depends on the recipe key
|
||||
surviving (24h TTL, ≥ token lifetime).
|
||||
- *Why a server-side store and not the token here (the real rationale).* It is
|
||||
tempting to delete `internal/noderecipe` and let the jellycompat node-hop
|
||||
token carry the recipe like native — the token is server-minted and opaque to
|
||||
the Jellyfin client (it follows it only as a redirect target), so it *could*.
|
||||
That does **not** work, and the reason is **not** the often-stated "a Jellyfin
|
||||
client can't round-trip a token." The reason is that the recipe is **mutated
|
||||
in place under a stable session id**: a Jellyfin audio/subtitle switch can
|
||||
arrive as a `/Sessions/Playing/Progress` report
|
||||
(`restartCompatTranscodeForAudioSelection`) that restarts ffmpeg in place
|
||||
**without re-minting the client's token**. A third-party Jellyfin client
|
||||
cannot be driven to refresh the now-stale token — its VOD/`ENDLIST` manifest
|
||||
is fetched once, and the compat WebSocket (`HandleSocket`) is a receive-only
|
||||
keep-alive stub with no server→client command to force a manifest re-fetch. A
|
||||
token is a snapshot at mint time, so after such a switch it would reconstruct
|
||||
the **stale** rendition (old audio/subtitle) until the client happens to
|
||||
re-fetch the manifest (e.g. on a seek), which for a VOD stream may be never.
|
||||
Because the node also cannot reach Postgres (where the authoritative compat
|
||||
recipe lives), the only way the node rebuilds the *current* recipe is a
|
||||
node-reachable, server-authoritative store that central overwrites on every
|
||||
switch — i.e. `internal/noderecipe`. Native has no equivalent path: every
|
||||
native audio switch re-mints the manifest + token, so the native client's
|
||||
token is never stale, which is exactly why native needs no such store. The
|
||||
store is **load-bearing, not redundant**.
|
||||
- **Key split / asymmetric signing (E).** Closes the account-takeover path; a
|
||||
precondition for untrusted edge nodes.
|
||||
|
||||
---
|
||||
|
||||
## 11. Preconditions
|
||||
|
||||
- **Persistent `TranscodeDir` (recommended).** With persistence, reconstruct
|
||||
serves surviving segments; with an ephemeral dir it re-transcodes from the
|
||||
descriptor (which survives a wiped dir). A node *restart* still loses the
|
||||
session regardless (P-3).
|
||||
- **Re-transcode of one segment beats the buffer runway** on worst-case hardware
|
||||
(software decode, subtitle burn-in). Measure.
|
||||
- **Client retry/backoff exists.** Clients retry segment/manifest GETs on
|
||||
5xx/connection error and reconnect the WebSocket. The token rides the URL, so no
|
||||
extra client logic is needed to carry it; a client that does not retry sees the
|
||||
failed request as fatal.
|
||||
|
||||
---
|
||||
|
||||
## 12. Diagrams
|
||||
|
||||
The `mermaid` blocks render in GitHub / VS Code / most Markdown viewers; the
|
||||
ASCII blocks render anywhere.
|
||||
|
||||
### 12.1 Topology — where state lives
|
||||
|
||||
> **Deferred:** the `silo:streamauth:<sid> = deny` marker, the "Admin
|
||||
> Stop/Terminate → write session-deny marker" arrow, and the node's "deny GET"
|
||||
> are the planned revocation path and are **not** in the current implementation.
|
||||
> The tracker (`silo:sessions:*`) and the recipe handoff are shipped.
|
||||
|
||||
```text
|
||||
┌─────────────────────────── CENTRAL (mode=server) ───────────────────────────┐
|
||||
Client │ RequireAuth /playback/start /playback/transcode/start (mint token) │
|
||||
(web / android / │ native serve: /playback/transcode/{sid}/master.m3u8?st= /stream/{sid}?st= │
|
||||
apple / jellyfin) │ Admin Stop/Terminate → write session-deny marker (event, not a timer) │
|
||||
└───────────┬──────────────────────────┬───────────────────────┬──────────────┘
|
||||
│ Postgres │ Redis │
|
||||
▼ ▼ │ (multi-node only)
|
||||
┌───────────────────────┐ ┌──────────────────────────┐ ▼
|
||||
│ catalog / access │ │ silo:sessions:* (tracker)│ ┌────────────────────────┐
|
||||
│ jellycompat_playback_ │ │ silo:streamauth:<sid> │ │ Proxy node (mode=proxy) │
|
||||
│ sessions (compat │ │ = deny (session kill) │◀─│ verifyToken + deny GET │
|
||||
│ store, carries Recipe)│ └──────────────────────────┘ │ serves direct/remux │
|
||||
└───────────────────────┘ ▲ │ proxies transcode ──────┼──▶ Transcode node
|
||||
✗ NO transcode_recipes table (deleted this branch) │ node-local GET └────────────────────────┘ (mode=transcode)
|
||||
└── sub-ms, off Postgres ffmpeg /transcode/{sid}/…
|
||||
```
|
||||
|
||||
Descriptor location by route (the core change). The "lease (node)" revocation
|
||||
column is the **deferred** design; as shipped, the node path has no revocation
|
||||
tighter than the ≤24h token:
|
||||
|
||||
| route family | descriptor (how it reconstructs) | revocation |
|
||||
|--------------|----------------------------------|------------|
|
||||
| native (1–3) | the signed token the client re-presents | *deferred:* lease (node) / shipped: producer-kill (integrated), ≤24h token (node) |
|
||||
| jellycompat (4–6) | `jellycompat_playback_sessions.data.Recipe` (token can't round-trip) | *deferred:* lease (node) / shipped: producer-kill, ≤24h token (node) |
|
||||
| ~~all~~ | ~~`transcode_recipes` Postgres row~~ — removed | — |
|
||||
|
||||
### 12.2 Where the token rides
|
||||
|
||||
```text
|
||||
NATIVE (integrated server is hit directly → query param; segments inherit it)
|
||||
manifest : GET /playback/transcode/{sid}/master.m3u8?st=<JWT>
|
||||
segment : GET /playback/transcode/{sid}/segment/seg_00042.m4s?st=<JWT> ← RawQuery propagated by the manifest rewriter
|
||||
direct : GET /stream/{sid}?st=<JWT>
|
||||
|
||||
PROXY/NODE (capability URL → path segment; survives a query-stripping hop)
|
||||
GET {proxy}/stream/transcode/{JWT}/master.m3u8 → relative segment URIs inherit {JWT}
|
||||
GET {proxy}/stream/direct/{JWT} GET {proxy}/stream/remux/{JWT}
|
||||
|
||||
JELLYCOMPAT → recipe is NOT in the token; it lives in the compat store.
|
||||
the path token is used only for the node hop.
|
||||
```
|
||||
|
||||
### 12.3 Normal playback — native, integrated box (no nodes)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as Client
|
||||
participant S as Central server (integrated)
|
||||
participant PG as Postgres (access)
|
||||
Note over C,S: transcode start
|
||||
C->>S: POST /playback/transcode/start (RequireAuth)
|
||||
S->>PG: resolve file + access
|
||||
S->>S: StartTranscode (ffmpeg) + RegisterTranscodeSession
|
||||
S->>S: card = NewRecipeCard(opts); token = Sign(card.ToClaims(), 24h)
|
||||
S-->>C: 202 manifestURL = /playback/transcode/{sid}/master.m3u8?st=TOKEN
|
||||
Note over C,S: hot path — NO Postgres, NO lease (live session in memory)
|
||||
C->>S: GET .../master.m3u8?st=TOKEN
|
||||
S->>S: LoadOrReconstructSession → live hit → BuildPlaybackManifest (segments carry ?st)
|
||||
S-->>C: manifest
|
||||
loop every segment
|
||||
C->>S: GET .../segment/seg_N.m4s?st=TOKEN
|
||||
S->>S: live session → GetSegment → ServeFile
|
||||
S-->>C: bytes
|
||||
end
|
||||
```
|
||||
|
||||
The hot path touches neither Postgres nor the lease — it is an in-memory map
|
||||
hit. `?st` is verified only to decode a card if reconstruct is needed; a live
|
||||
session ignores it.
|
||||
|
||||
### 12.4 Normal playback — native, multi-node (offloaded)
|
||||
|
||||
> **Deferred:** the `GET silo:streamauth:{sid}` lease guard and its `deny → 403`
|
||||
> branch are the planned revocation path, **not** in the current implementation.
|
||||
> As shipped the node verifies the token, tracks the session, and serves — there
|
||||
> is no deny check.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as Client
|
||||
participant S as Central server
|
||||
participant N as Proxy node
|
||||
participant T as Transcode node
|
||||
participant R as Redis
|
||||
C->>S: POST /playback/transcode/start (RequireAuth)
|
||||
S->>S: card(full recipe, tnode=T); token = Sign(card.ToClaims())
|
||||
S-->>C: manifestURL = {N}/stream/transcode/TOKEN/master.m3u8
|
||||
loop manifest + segments
|
||||
C->>N: GET /stream/transcode/TOKEN/...
|
||||
N->>N: verifyToken(TOKEN)
|
||||
N->>R: GET silo:streamauth:{sid} (lease guard)
|
||||
alt deny
|
||||
N-->>C: 403 (no bytes)
|
||||
else allow / absent
|
||||
N->>R: Track silo:sessions:{node}:{sid} (uid, mfid from claims)
|
||||
N->>T: proxy /transcode/{sid}/...
|
||||
T-->>N: bytes
|
||||
N-->>C: bytes
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
The node copies `uid`/`mfid` from the verified token into the Redis tracker —
|
||||
that is what enriches the live admin "active streams" view (who/what per node).
|
||||
|
||||
### 12.5 Restart → reconstruct (native transcode)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as Client
|
||||
participant S as Central server (just restarted — session map empty)
|
||||
Note over S: in-memory session GONE; NO transcode_recipes table to read
|
||||
C->>S: GET /playback/transcode/{sid}/segment/seg_42.m4s?st=TOKEN
|
||||
S->>S: card = streamCardFromQuery(?st) → Verify + decode recipe
|
||||
S->>S: LoadOrReconstructSession → GetSession = NotFound, card != nil
|
||||
S->>S: ReconstructSession (re-bind: refuse userID==0 / != card.uid; card.sid == {sid})
|
||||
S->>S: ReconstructTranscode(seg=42) → single-flight, mark in-flight, respawn ffmpeg SEEKED to seg 42
|
||||
S-->>C: bytes (resumes near where the client was)
|
||||
```
|
||||
|
||||
- direct/remux restart is the same but Tier-1: reconstruct the `Session` from the
|
||||
identity token, then `Range` / `?seek` re-serve — no ffmpeg rebuild.
|
||||
- multi-node restart: the integrated front-end reconstructs only the `Session`
|
||||
(to learn `tnode`) and re-proxies; the transcode node keeps serving. If the
|
||||
transcode *node* itself reboots, the proxy forwards the stream token
|
||||
(`X-Silo-Stream-Token`) and the node self-reconstructs ffmpeg seeked to the
|
||||
requested segment (native path, P-3). A jellycompat session on a rebooted node
|
||||
reconstructs too: the node fetches the recipe central wrote to the shared Redis
|
||||
recipe store (`internal/noderecipe`) at transcode start, since the Jellyfin
|
||||
client's token cannot carry it.
|
||||
- cleanup race: a segment dir is reaped only if absent from the live map AND the
|
||||
in-flight set AND older than 24h (max token TTL), so a token that can still
|
||||
reconstruct never gets its dir wiped.
|
||||
|
||||
### 12.6 Auth deny — the session-deny marker
|
||||
|
||||
> **Status: deferred to a future PR.** The entire "Admin Stop/Terminate → write
|
||||
> deny marker" and "Offload node → GET deny → 403" flow below is the planned
|
||||
> revocation design and is **not** in the current implementation. As shipped,
|
||||
> admin Stop/Terminate does the "realtime WS stop + producer teardown" half only
|
||||
> (the right branch); it writes no marker, and the node performs no deny check, so
|
||||
> a valid token reconstructs until its ≤24h TTL. New playback is still gated at
|
||||
> `/playback/start` (the top subgraph is accurate).
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph New["New playback (any topology)"]
|
||||
P["/playback/start — RequireAuth + access check"] --> PB{allowed?}
|
||||
PB -- no --> PR[refused — ban takes effect at next play]
|
||||
PB -- yes --> PM[mint token]
|
||||
end
|
||||
subgraph Admin["Admin Stop/Terminate (event, not a timer)"]
|
||||
K[denyStreamLease] --> WD[SET silo:streamauth:sid = deny<br/>TTL = 24h]
|
||||
K --> WS[realtime WS stop + producer teardown]
|
||||
end
|
||||
subgraph Node["Offload node — every serve"]
|
||||
G[verifyToken OK] --> L{GET silo:streamauth:sid}
|
||||
L -- present deny --> F[403, no bytes]
|
||||
L -- absent / redis-err --> SV[serve bytes fail-open]
|
||||
end
|
||||
```
|
||||
|
||||
Invariants:
|
||||
- No steady-state writes: the marker is set only on an admin kill, never on a
|
||||
timer. The normal serve path finds *no* key and serves.
|
||||
- Fail-open: absent key or Redis error → serve. The only hard stop is a present
|
||||
deny.
|
||||
- The integrated box reads no marker — the producer-kill + session removal on
|
||||
Terminate is the stop there; the marker only matters where nodes serve.
|
||||
- Not covered here (by design): a *passive* ban / partial access change of an
|
||||
in-flight non-cooperative node stream — enforced at next play, bounded by the
|
||||
≤24h token (§7).
|
||||
|
||||
### 12.7 Jellycompat — what differs
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant J as Jellyfin client
|
||||
participant S as Central (jellycompat)
|
||||
participant PG as Postgres compat store
|
||||
J->>S: negotiate → PlaybackSession{UpstreamSessionID, MediaSources}
|
||||
S->>PG: Put (jellycompat_playback_sessions)
|
||||
Note over S: transcode start
|
||||
S->>S: StartTranscode; recipe = NewRecipeCard(opts)
|
||||
S->>PG: Update(ps): TranscodeStarted=true, Recipe=recipe (same write)
|
||||
Note over J,S: restart → reconstruct from the COMPAT STORE, not a token
|
||||
J->>S: GET HLS segment
|
||||
S->>PG: Get(playSessionID) → ps.Recipe
|
||||
S->>S: LoadOrReconstructSession(UpstreamSessionID, card = ps.Recipe) → ReconstructTranscode(*ps.Recipe)
|
||||
S-->>J: bytes
|
||||
```
|
||||
|
||||
Jellyfin clients cannot carry a native `?st` token, so the recipe is folded into
|
||||
the durable compat row (`PlaybackSession.Recipe`, persisted in the `data` JSONB)
|
||||
and written in the same `Update` that flips `TranscodeStarted`. The node hop
|
||||
(multi-node jellycompat) still uses a path token exactly like native (the lease
|
||||
guard is deferred along with the rest of the revocation path).
|
||||
|
||||
### 12.8 Hot-path cost per request
|
||||
|
||||
The "lease" / "SET deny" Redis costs below are **deferred** (the revocation path
|
||||
is not shipped); as shipped the node does only the tracker write, and admin
|
||||
Stop/Terminate writes no Redis key:
|
||||
|
||||
| request | Postgres | Redis | central CPU | reconstruct |
|
||||
|---------|----------|-------|-------------|-------------|
|
||||
| native segment, live, integrated | — | — | map hit | — |
|
||||
| native segment, live, node | — | 1 track (+ deferred: GET lease) | — | — |
|
||||
| native segment, after restart, integrated | — | — | verify JWT + ffmpeg respawn (once, single-flight) | yes |
|
||||
| jellycompat segment, live | — | track (+ deferred: lease) | map hit | — |
|
||||
| jellycompat segment, after restart | 1 Get (compat row) | track (+ deferred: lease) | respawn | yes |
|
||||
| admin Stop/Terminate (event, not per-request) | — | deferred: 1 SET deny | one write | — |
|
||||
|
||||
---
|
||||
|
||||
## 13. Restart-resiliency behavior matrix (role × path)
|
||||
|
||||
The observable reconstruction behavior of every playback path under a restart of
|
||||
each server role. **✅** = the path reconstructs and resumes on the client's next
|
||||
request; **—** = the path never executes on that role (nothing to reconstruct);
|
||||
**⚠️** = reconstructs only while a stated precondition holds.
|
||||
|
||||
| path | integrated restart | proxy-node restart | transcode-node restart |
|
||||
|------|--------------------|--------------------|------------------------|
|
||||
| native · direct | ✅ rebuild `Session` from `?st` token → `Range` re-serve | ✅ stateless re-serve from `claims.MediaPath` | — not served on a transcode node |
|
||||
| native · remux | ✅ token + `?seek` → re-spawn pipe | ✅ stateless per-request pipe re-spawn | — |
|
||||
| native · transcode | ✅ token (full recipe) → respawn ffmpeg seeked to segment | ✅ transparent re-route + token forward | ✅ self-contained — token is recipe-complete |
|
||||
| jellycompat · direct | ✅ re-resolve from durable compat row → `Range` re-serve | ✅ stateless re-serve from token `MediaPath` | — |
|
||||
| jellycompat · remux | ✅ compat row + `?seek` | ✅ stateless per-request pipe re-spawn | — |
|
||||
| jellycompat · transcode | ✅ `ReconstructTranscode(*ps.Recipe)` from compat row | ✅ transparent re-route + token forward | ⚠️ reconstructs **iff** the shared Redis recipe (`internal/noderecipe`, ≤24h TTL) still exists; fails closed (404) otherwise |
|
||||
|
||||
**What carries the recovery state, per role**
|
||||
|
||||
| role | native carrier | jellycompat carrier |
|
||||
|------|----------------|---------------------|
|
||||
| integrated | `?st` stream token — identity for direct/remux, full recipe for transcode | durable compat row `PlaybackSession.Recipe` |
|
||||
| proxy node | self-describing path token, re-derived per request — the proxy holds no per-session state | same path token (identity); the recipe is not needed at the proxy |
|
||||
| transcode node | forwarded `X-Silo-Stream-Token`, recipe-complete | identity token + Redis `noderecipe` recipe (the Jellyfin token cannot carry it) |
|
||||
|
||||
**Reading the matrix**
|
||||
|
||||
- **Integrated and proxy-node restarts are unconditional** for every applicable
|
||||
path. The proxy holds no authoritative per-session state — direct/remux are
|
||||
re-served from the token's `MediaPath`, transcode is reverse-proxied — so a
|
||||
proxy bounce, or a reroute to any in-group proxy (`pickProxy` is round-robin
|
||||
with no session affinity), is transparent.
|
||||
- **The four `—` cells are structural, not gaps.** Direct and remux never run on
|
||||
a transcode node — the proxy (or the integrated box) serves them from source
|
||||
media — so a transcode-node restart has nothing to reconstruct on those paths.
|
||||
- **The one conditional (⚠️)** is jellycompat transcode after a *transcode-node*
|
||||
restart. The node-hop token is identity-only — not because a Jellyfin client
|
||||
cannot round-trip it (the token is server-minted and opaque to the client), but
|
||||
because the recipe is mutated in place under a stable session id and a
|
||||
third-party Jellyfin client cannot be driven to refresh a stale token, so a
|
||||
token snapshot could reconstruct a stale rendition (see §10). The node therefore
|
||||
rebuilds from the Redis `noderecipe` entry central overwrites on every switch
|
||||
rather than from the token. Native transcode has no such dependency — every
|
||||
native audio switch re-mints the token, so it is always recipe-complete and
|
||||
current. This is the only native-vs-jellycompat asymmetry in reconstruction
|
||||
(see §10).
|
||||
|
||||
**Out of scope of this matrix** (accepted behaviors, not restart-reconstruction
|
||||
holes — see §7, §10):
|
||||
|
||||
- Passive ban / partial access change of an in-flight node stream — enforced at
|
||||
next play, bounded by the ≤24h token, not by reconstruction.
|
||||
- Integrated transcode behind a non-sticky load balancer can split-brain into
|
||||
divergent segment dirs (P-1); safe single-front-end or with LB affinity. Remote
|
||||
(node) sessions are immune — node affinity routes every front-end to the same
|
||||
transcode node.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,711 @@
|
||||
# Offline sync for mobile (downloads v2) — design
|
||||
|
||||
**Goal:** Let mobile clients (Android, Apple) download media for **fully offline** playback and keep
|
||||
watch state consistent across reconnects. Three pillars: (1) a **device-scoped download registry** —
|
||||
the server tracks what each device has downloaded; (2) an **offline playback manifest** — one bundle
|
||||
giving the client everything it needs to play with no network (metadata, subtitle files, artwork,
|
||||
chapter + intro/credits markers); (3) **offline progress reconciliation** — the client queues watch
|
||||
progress while offline and the server merges it on reconnect (last-write-wins). Downloads are requested
|
||||
through an admin-gated **quality** ladder (`original`, then bitrate presets), with server-side
|
||||
**transcode-to-a-single-file** for bitrate requests and compatibility fallback for devices that cannot
|
||||
play the source directly.
|
||||
|
||||
**Status:** Approved — implementing. This capability cleared the Silo v1 scope gate on
|
||||
2026-06-18 (via the **v1 capability proposal** triage; template
|
||||
`.github/ISSUE_TEMPLATE/v1-capability-proposal.yml`), and the offline-sync/downloads-v2 work is now
|
||||
being implemented phase by phase against this design. Per the decision below, the
|
||||
`/api/v1/downloads` contract is **reshaped, not just extended** — acceptable because the change lands
|
||||
**before** scope lock (`docs/architecture/v1-scope.md`) and **before** any mobile client ships against
|
||||
it. This document began as the design input to that proposal and now serves as the implementation
|
||||
reference; it is kept in sync as the implementation lands.
|
||||
|
||||
Decisions locked with the requester:
|
||||
- **Replace** the existing `internal/download` package with a new unified package (`internal/downloads`)
|
||||
that absorbs today's download behavior and adds the three pillars.
|
||||
- **The existing `/api/v1/downloads` contract may be reshaped, not just extended.** The only current
|
||||
consumer is the first-party web app, which is updated in lockstep. So downloads v2 is a single
|
||||
coherent surface: one device-aware `downloads` table and one `/downloads/*` endpoint family — *not* a
|
||||
frozen legacy table plus a parallel registry. (This intentionally diverges from the v1 additive-only
|
||||
API rule, which is acceptable here because the change lands **before** scope lock and **before** any
|
||||
mobile client ships against it — exactly the right time to reshape. The proposal must call this out so
|
||||
reviewers/clients know the download contract changed pre-lock.)
|
||||
- **Quality is user-facing; delivery format is internal.** Clients request `quality` from the ordered
|
||||
ladder `original > 20mbps > 10mbps > 5mbps > 2mbps > 1mbps`. `remux` is not a preset; it is a
|
||||
server-selected `delivery_format` when `original` needs container/audio compatibility. Bitrate
|
||||
presets are gated by a server setting **and** the per-user `download_transcode_allowed` flag.
|
||||
- **Transcode-to-single-file is in v1.** (Today's transcode pipeline only emits ephemeral HLS segments.)
|
||||
- **No expiry, no lease, no DRM.** Downloaded files persist on-device until the user deletes them; the
|
||||
server can only revoke *future* downloads via `users.download_allowed`.
|
||||
- **No cross-device visibility** in v1 (a device sees its own entries, not other devices').
|
||||
|
||||
**Implementation update (2026-06-20):** the current client-facing contract is documented in
|
||||
`docs/downloads-api.md`. The public request field is `quality`, capabilities expose
|
||||
`quality_presets`, rows expose `quality`, `effective_quality`, `delivery_format`,
|
||||
`target_bitrate_kbps`, and `revision`, and batch manifests are implemented at
|
||||
`GET /api/v1/downloads/batches/{batch_id}/manifests`.
|
||||
|
||||
> Commands and paths in this document are repository-relative; assume the repository root is the cwd.
|
||||
|
||||
## Why this shape
|
||||
|
||||
- **One device-aware table, not two.** The original plan kept today's `downloads` table
|
||||
(`migrations/sql/042_downloads.sql`, keyed on `user_id`) frozen and added a separate device registry,
|
||||
purely to avoid breaking the existing contract. With that constraint lifted, the cleaner model is a
|
||||
single `downloads` table reshaped to carry `device_id` (and `profile_id`). `device_id` is **nullable**:
|
||||
- `device_id IS NULL` → an **ephemeral / account-level** download (today's web-app browser download:
|
||||
pick a version, pull the file; the row is a convenience record, not a managed library entry).
|
||||
- `device_id` set → a **managed device-library entry** (a phone "has" this file; durable; unique per
|
||||
`(user, profile, device, content, episode)`; the target of manifest + progress reconciliation).
|
||||
One table, one endpoint family, and the format/transcode machinery is shared by web and mobile alike.
|
||||
- **Device identity already exists in Postgres.** `migrations/sql/180_user_devices.sql` defines
|
||||
`public.user_devices` keyed `(user_id, profile_id, device_id)` with FKs to `users` and
|
||||
`user_profiles`. Device id arrives in the `X-Silo-Device-Id` header (see
|
||||
`internal/api/handlers/settings.go:37`). Managed entries reference it; ephemeral rows leave it NULL.
|
||||
- **The last-write-wins primitive already exists.** `userstore.UserStore.SetProgressIfNewer(... updatedAt) (bool, error)`
|
||||
(`internal/userstore/store.go:26`, implemented in `internal/userdb/progress.go:131`) is exactly the
|
||||
merge semantics offline reconciliation needs. Progress reconciliation is therefore **additive changes
|
||||
to the existing progress endpoints**, not new machinery, and it deliberately stays in the watch-state
|
||||
domain rather than being folded into the downloads package.
|
||||
- **Manifests must never carry presigned URLs.** `catalog.ItemDetail` (`internal/catalog/detail.go`)
|
||||
exposes `PosterURL`/`BackdropURL` as short-lived presigned S3 URLs — useless for an offline bundle
|
||||
that may be opened days later. The manifest carries **stable** references (thumbhashes inline +
|
||||
authenticated proxy endpoints the client fetches once at download time), so a stored manifest never
|
||||
expires.
|
||||
- **Remux and transcode both need a produced file, not a live pipe.** A clean, seekable, resumable
|
||||
download requires a finalized file (`-movflags +faststart` relocates the moov atom in a finalization
|
||||
pass — impossible over a pure pipe). `internal/playback`'s current `ServeRemux`/`StartTranscode`
|
||||
stream ephemeral output. So remux and transcode share one **prepare-to-file** job pipeline; only
|
||||
`original` is served straight from the source on disk.
|
||||
|
||||
## The replacement: `internal/downloads` (new package)
|
||||
|
||||
`internal/download` (singular) is replaced by `internal/downloads` (plural). The new package owns the
|
||||
whole downloads domain; everything below lives in it unless noted.
|
||||
|
||||
```
|
||||
internal/downloads/
|
||||
model.go — Download (web + device, one type), Artifact, OfflineManifest,
|
||||
quality/delivery constants, status constants, sentinel errors
|
||||
repo.go — Postgres CRUD: downloads (reshaped), download_artifacts
|
||||
service.go — orchestration: permission + quota checks, quality policy, create/list/serve
|
||||
policy.go — DownloadQualityResolver (quality ladder + delivery decision + server/user gates)
|
||||
bandwidth.go — BandwidthManager (ported from internal/download/bandwidth.go)
|
||||
limiter.go — QuantityLimiter (ported from internal/download/limiter.go)
|
||||
manifest.go — ManifestBuilder (assembles OfflineManifest; strips all presigned URLs)
|
||||
artifacts.go — ArtifactManager (async prepare-to-file jobs: remux + transcode, dedup, cleanup)
|
||||
serve.go — file serving for original source and prepared artifacts
|
||||
```
|
||||
|
||||
**What it absorbs / reshapes** from `internal/download`: the `downloads` table and its endpoints, now
|
||||
reshaped to be device-aware and quality/delivery-aware; `BandwidthManager`; `QuantityLimiter`; the `Service`
|
||||
permission/quota/file-serving logic; and the `config.DownloadConfig` integration.
|
||||
|
||||
**What stays out of the package (by design):**
|
||||
- **Progress reconciliation** is watch-state, not downloads. It is additive changes to
|
||||
`internal/api/handlers/progress.go` + `internal/userstore`/`internal/userdb`, reusing
|
||||
`SetProgressIfNewer`. Folding it into `internal/downloads` would couple two domains and duplicate the
|
||||
LWW write.
|
||||
- **The encode/remux primitive** stays in `internal/playback` (it owns ffmpeg). `internal/downloads`
|
||||
calls a new `playback.PrepareFile(...)`; it does not shell out to ffmpeg itself.
|
||||
|
||||
**Cutover strategy:** because the web app is the only consumer and is updated in lockstep, Phase 0 is a
|
||||
*reshape*, not a verbatim port: move the logic into `internal/downloads`, reshape the table + endpoints
|
||||
(add device-awareness plus quality/delivery fields), update the web app's download hooks/components, delete
|
||||
`internal/download`. The handler `internal/api/handlers/downloads.go` already depends only on a narrow
|
||||
`DownloadService` interface, so the swap is mechanical; its request/response DTOs gain the new fields.
|
||||
|
||||
## Data model (Goose migrations)
|
||||
|
||||
Migrations are timestamped, created with `make migrate-create NAME=...` (never hand-numbered; never
|
||||
`goose fix`).
|
||||
|
||||
### 1. Reshape `downloads` (device-aware, format-aware)
|
||||
|
||||
A single migration alters the existing table. Downloads ship default-off with little/no production data,
|
||||
so widening it in place is safe.
|
||||
|
||||
```sql
|
||||
-- +goose Up
|
||||
-- +goose StatementBegin
|
||||
ALTER TABLE public.downloads
|
||||
ADD COLUMN profile_id text, -- NULL for ephemeral/web rows
|
||||
ADD COLUMN device_id text, -- NULL = ephemeral; set = managed device entry
|
||||
ADD COLUMN format text NOT NULL DEFAULT 'original',
|
||||
ADD COLUMN artifact_id text; -- set for remux/transcode once prepared
|
||||
|
||||
ALTER TABLE public.downloads
|
||||
ADD CONSTRAINT downloads_format_check CHECK (format IN ('original','remux','transcode'));
|
||||
|
||||
-- Widen the status enum to cover the managed-entry lifecycle alongside the existing serve states.
|
||||
ALTER TABLE public.downloads DROP CONSTRAINT downloads_status_check;
|
||||
ALTER TABLE public.downloads ADD CONSTRAINT downloads_status_check
|
||||
CHECK (status IN ('queued','downloading','completed','failed','cancelled', -- existing
|
||||
'registered','preparing','ready','revoked')); -- managed entries
|
||||
|
||||
-- One managed entry per (user, profile, device, content, episode). Ephemeral rows (NULL device_id)
|
||||
-- are exempt via the partial index.
|
||||
CREATE UNIQUE INDEX downloads_device_entry_uidx
|
||||
ON public.downloads (user_id, profile_id, device_id, content_id, COALESCE(episode_id, ''))
|
||||
WHERE device_id IS NOT NULL;
|
||||
|
||||
CREATE INDEX downloads_device_idx ON public.downloads (user_id, profile_id, device_id) WHERE device_id IS NOT NULL;
|
||||
CREATE INDEX downloads_artifact_idx ON public.downloads (artifact_id) WHERE artifact_id IS NOT NULL;
|
||||
|
||||
-- Composite FK to user_devices. MATCH SIMPLE (the default) skips the check whenever any column is NULL,
|
||||
-- so ephemeral rows (NULL device_id) are not constrained while managed entries are.
|
||||
ALTER TABLE public.downloads
|
||||
ADD CONSTRAINT downloads_device_fkey
|
||||
FOREIGN KEY (user_id, profile_id, device_id)
|
||||
REFERENCES public.user_devices(user_id, profile_id, device_id) ON DELETE CASCADE;
|
||||
-- +goose StatementEnd
|
||||
|
||||
-- +goose Down
|
||||
-- +goose StatementBegin
|
||||
ALTER TABLE public.downloads DROP CONSTRAINT IF EXISTS downloads_device_fkey;
|
||||
DROP INDEX IF EXISTS downloads_artifact_idx;
|
||||
DROP INDEX IF EXISTS downloads_device_idx;
|
||||
DROP INDEX IF EXISTS downloads_device_entry_uidx;
|
||||
ALTER TABLE public.downloads DROP CONSTRAINT IF EXISTS downloads_format_check;
|
||||
ALTER TABLE public.downloads DROP CONSTRAINT IF EXISTS downloads_status_check;
|
||||
ALTER TABLE public.downloads ADD CONSTRAINT downloads_status_check
|
||||
CHECK (status IN ('queued','downloading','completed','failed','cancelled'));
|
||||
ALTER TABLE public.downloads
|
||||
DROP COLUMN artifact_id, DROP COLUMN format, DROP COLUMN device_id, DROP COLUMN profile_id;
|
||||
-- +goose StatementEnd
|
||||
```
|
||||
|
||||
Status lifecycle:
|
||||
- **ephemeral / web** (`device_id IS NULL`): `queued → downloading → completed` / `failed` /
|
||||
`cancelled` — exactly today's behavior, now also able to request `remux`/`transcode`.
|
||||
- **managed entry** (`device_id` set):
|
||||
- `original`: `registered → ready` immediately (client pulls the source), then client reports
|
||||
`completed`.
|
||||
- `remux` / `transcode`: `registered → preparing` (artifact job runs) `→ ready → completed`; or
|
||||
`failed`.
|
||||
- `revoked`: admin/permission revocation of *future* access. Cannot delete the on-device file (no DRM
|
||||
by decision); it only stops the server serving new bytes.
|
||||
|
||||
The unique index uses `COALESCE(episode_id,'')` so movies (NULL `episode_id`) collapse to one managed
|
||||
entry per `(device, content)` without NULL-comparison surprises.
|
||||
|
||||
### 2. `download_artifacts` — prepared (remux/transcode) files, deduplicated
|
||||
|
||||
```sql
|
||||
-- +goose Up
|
||||
-- +goose StatementBegin
|
||||
CREATE TABLE public.download_artifacts (
|
||||
id text NOT NULL,
|
||||
media_file_id integer NOT NULL REFERENCES public.media_files(id) ON DELETE CASCADE,
|
||||
format text NOT NULL, -- 'remux' | 'transcode'
|
||||
params_hash text NOT NULL, -- sha256 of the encode parameters (see below)
|
||||
container text NOT NULL DEFAULT 'mp4',
|
||||
codec_video text NOT NULL DEFAULT '',
|
||||
codec_audio text NOT NULL DEFAULT '',
|
||||
resolution text NOT NULL DEFAULT '',
|
||||
audio_track_index integer NOT NULL DEFAULT -1,
|
||||
output_path text NOT NULL DEFAULT '', -- absolute path on the server's artifact volume
|
||||
file_size bigint NOT NULL DEFAULT 0,
|
||||
status text NOT NULL DEFAULT 'queued',
|
||||
error_message text NOT NULL DEFAULT '',
|
||||
-- Durable-queue / crash-recovery columns (see "Durable artifact queue" below).
|
||||
attempts integer NOT NULL DEFAULT 0,
|
||||
max_attempts integer NOT NULL DEFAULT 3,
|
||||
lease_owner text, -- worker/node id holding the running lease
|
||||
lease_expires_at timestamptz, -- NULL unless status='running'
|
||||
next_retry_at timestamptz, -- backoff gate for re-enqueue after a failure
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
completed_at timestamptz,
|
||||
last_used_at timestamptz NOT NULL DEFAULT now(),
|
||||
CONSTRAINT download_artifacts_pkey PRIMARY KEY (id),
|
||||
CONSTRAINT download_artifacts_unique UNIQUE (media_file_id, format, params_hash),
|
||||
CONSTRAINT download_artifacts_status_check CHECK (status IN ('queued','running','ready','failed'))
|
||||
);
|
||||
|
||||
CREATE INDEX download_artifacts_lru_idx ON public.download_artifacts (last_used_at) WHERE status = 'ready';
|
||||
-- Claimable work: queued rows whose backoff has elapsed, plus running rows whose lease has expired.
|
||||
CREATE INDEX download_artifacts_claimable_idx ON public.download_artifacts (status, next_retry_at);
|
||||
CREATE INDEX download_artifacts_lease_idx ON public.download_artifacts (lease_expires_at) WHERE status = 'running';
|
||||
-- +goose StatementEnd
|
||||
|
||||
-- +goose Down
|
||||
-- +goose StatementBegin
|
||||
DROP TABLE IF EXISTS public.download_artifacts;
|
||||
-- +goose StatementEnd
|
||||
```
|
||||
|
||||
`params_hash = sha256(format | container | codec_video | codec_audio | resolution | audio_track_index | subtitle_burn_in)`.
|
||||
Two devices (or a web user and a phone) requesting the same 1080p H.264/AAC of the same source share one
|
||||
artifact. `last_used_at` drives LRU cleanup; `output_path` is on a configured artifact volume. The
|
||||
`attempts`/`lease_*`/`next_retry_at` columns make the table a **durable, recoverable job queue** (see the
|
||||
"Durable artifact queue" subsection) so a crash mid-encode cannot strand a download in `preparing`.
|
||||
|
||||
### 3. Server settings (no DDL)
|
||||
|
||||
`server_settings` is key/value, so new keys need no migration — written when an admin saves them, parsed
|
||||
in `internal/config/db_loader.go`:
|
||||
- `download.transcode_enabled` (bool, default **false**) — server gate for transcode-to-file.
|
||||
- `download.artifact_dir` (string) — artifact output volume; defaults under the existing transcode/temp root.
|
||||
- `download.max_concurrent_prepares` (int, default 2) — encode/remux worker-pool size.
|
||||
- `download.artifact_max_bytes` (int64, default 0 = unlimited) — LRU eviction budget for artifacts.
|
||||
|
||||
`config.DownloadConfig` (`internal/config/config.go:299`) gains the matching fields; they hot-reload via
|
||||
the existing 30s-TTL cache (`internal/download/service.go:107` → ported), and the keys are registered as
|
||||
non-restart in `internal/config/restart_keys.go`.
|
||||
|
||||
## Quality policy
|
||||
|
||||
Resolved in `internal/downloads/policy.go` at create time, for both web and device flows:
|
||||
|
||||
```go
|
||||
const (
|
||||
QualityOriginal = "original"
|
||||
Quality20Mbps = "20mbps"
|
||||
Quality10Mbps = "10mbps"
|
||||
Quality5Mbps = "5mbps"
|
||||
Quality2Mbps = "2mbps"
|
||||
Quality1Mbps = "1mbps"
|
||||
)
|
||||
|
||||
type QualityDecision struct {
|
||||
RequestedQuality string
|
||||
EffectiveQuality string
|
||||
DeliveryFormat string // original | remux | transcode
|
||||
TargetBitrateKbps int
|
||||
RequiresArtifact bool
|
||||
}
|
||||
|
||||
func (DownloadQualityResolver) Resolve(requested string, user *models.User,
|
||||
cfg config.DownloadConfig, file *models.MediaFile, caps playback.ClientCapabilities,
|
||||
artifactsAvailable bool) (QualityDecision, error)
|
||||
```
|
||||
|
||||
Decision tree:
|
||||
1. Empty quality defaults to `original`.
|
||||
2. `original` direct-plays the source unless supplied device caps require a compatibility artifact.
|
||||
Container/audio-only compatibility records `delivery_format=remux`; video transcode fallback records
|
||||
`effective_quality=20mbps`, `delivery_format=transcode`.
|
||||
3. Bitrate presets require `cfg.TranscodeEnabled` (server), `user.DownloadTranscodeAllowed`, and an
|
||||
available artifact pipeline. Server gate off → `ErrTranscodeDisabled` (403). User flag off →
|
||||
`ErrDownloadNotAllowed` (403). Missing artifact pipeline → `ErrQualityUnavailable` (501).
|
||||
4. `remux` and `transcode` as request values are invalid public qualities.
|
||||
|
||||
Target codec/resolution for compatibility artifacts and bitrate transcodes is chosen by reusing `playback.Resolve` /
|
||||
`playback.SelectVersion` (`internal/playback/resolver.go:60`) against the device's declared capabilities
|
||||
and the admin transcode ceilings, so download encoding matches streaming decisions exactly (no
|
||||
duplicated codec logic).
|
||||
|
||||
## Transcode-to-file pipeline (v1)
|
||||
|
||||
`internal/downloads/artifacts.go` — `ArtifactManager`. Remux and transcode share this; `original` never
|
||||
enters it.
|
||||
|
||||
**On create of a remux/transcode download (web or device):**
|
||||
1. Resolve target params (codec/res/container/audio track) → compute `params_hash`.
|
||||
2. `INSERT ... ON CONFLICT (media_file_id, format, params_hash) DO NOTHING` into `download_artifacts`;
|
||||
read the row. If `status = ready`, link `downloads.artifact_id`, set `status = ready`,
|
||||
`file_size = artifact.file_size`, bump `last_used_at`. Done — no new encode.
|
||||
3. Otherwise the download row is `preparing`; if the artifact is freshly `queued`, enqueue an encode job.
|
||||
|
||||
**Encode worker** (bounded pool of `download.max_concurrent_prepares`, hosted on the existing
|
||||
`internal/taskmanager` worker machinery so jobs are tracked, observable in admin, and cancellable):
|
||||
1. **Claim a job transactionally** (see "Durable artifact queue" below) — never just "enqueue freshly
|
||||
queued rows," which loses work on restart.
|
||||
2. Call a new `playback.PrepareFile(ctx, opts)` that builds **single-file** ffmpeg args and writes to
|
||||
`output_path + ".part"`, then atomically renames to `output_path`:
|
||||
- **remux:** `-map ... -c copy -movflags +faststart -f mp4` (fast, near disk-throughput).
|
||||
- **transcode:** the existing encode args (reuse the `buildFFmpegArgs` logic / `TranscodeOpts` at
|
||||
`internal/playback/transcode.go:30`, incl. HW-accel, audio-track selection, optional subtitle
|
||||
burn-in) but targeting `-movflags +faststart -f mp4` to a file instead of `-f hls`.
|
||||
`TranscodeOpts` already carries every needed field (`InputPath`, `TargetCodecVideo/Audio`,
|
||||
`TargetResolution`, `HWAccel`, `AudioTrackIndex`, `SubtitleBurnIn`, `TotalDuration`).
|
||||
3. On success: artifact → `ready` (`file_size`/`completed_at`, clear lease); flip every linked download
|
||||
row → `ready`; publish a `user_state`/`downloads` event on the events hub (`internal/events`) so a
|
||||
waiting client pulls without polling.
|
||||
4. On failure: increment `attempts`, set `error_message`/`next_retry_at` (backoff). If
|
||||
`attempts >= max_attempts` → artifact `failed` and linked rows `failed`; otherwise return it to
|
||||
`queued` for retry. Either way, delete the stale `.part` file.
|
||||
|
||||
**Durable artifact queue (crash/restart recovery).** The artifact table is the queue of record, so a
|
||||
process exit mid-encode cannot strand a download in `preparing` forever:
|
||||
- **Transactional claim:** a worker claims a job with a single statement —
|
||||
`UPDATE download_artifacts SET status='running', lease_owner=$me, lease_expires_at=now()+$lease, attempts=attempts+1
|
||||
WHERE id = (SELECT id FROM download_artifacts
|
||||
WHERE (status='queued' AND (next_retry_at IS NULL OR next_retry_at <= now()))
|
||||
OR (status='running' AND lease_expires_at < now()) -- steal expired leases
|
||||
ORDER BY created_at LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING *`.
|
||||
`FOR UPDATE SKIP LOCKED` makes concurrent workers (and multiple nodes) safe without double-encoding.
|
||||
- **Lease heartbeat:** a running worker periodically extends `lease_expires_at`; if it dies, the lease
|
||||
expires and another worker (or the startup sweep) reclaims the job.
|
||||
- **Startup sweep:** on boot, reset `running` rows with an expired lease back to `queued` (bounded by
|
||||
`max_attempts`), and re-link any `preparing` download rows to their artifact's real state. A `ready`
|
||||
artifact whose `output_path` is missing on disk is reset to `queued` (the file was lost).
|
||||
- **Idempotency:** the encode writes to `output_path + ".part"` and atomically renames; a reclaimed job
|
||||
overwrites its own `.part`. `output_path` is deterministic from `(media_file_id, format, params_hash)`.
|
||||
- **No silent stranding:** a download stuck in `preparing` past a bound is surfaced (admin task + a
|
||||
`failed` transition with reason), never left invisibly hung.
|
||||
|
||||
**Serving** (`internal/downloads/serve.go`, `GET /downloads/{id}/file`):
|
||||
- `original`: open `media_files.file_path`, `http.ServeContent` (Range/resume) wrapped in the ported
|
||||
`BandwidthManager.ThrottledReader`. Same path as today's `serveFileDownload`.
|
||||
- `remux`/`transcode`: require `status = ready`; serve `artifact.output_path` the same way (Range/resume
|
||||
+ throttle); bump `last_used_at`.
|
||||
|
||||
**Cleanup:** an admin/cron job (mirroring `internal/playback/transcode_cleanup.go`) evicts `ready`
|
||||
artifacts whose `last_used_at` is older than a retention window or when the total exceeds
|
||||
`download.artifact_max_bytes` (LRU), but never one still linked by an active managed download row,
|
||||
including completed rows that represent a device's local library state.
|
||||
Whatever is dropped is logged (no silent truncation).
|
||||
|
||||
## Offline playback manifest
|
||||
|
||||
`internal/downloads/manifest.go` builds `OfflineManifest` from `catalog`'s existing detail path (reusing
|
||||
`catalog.ItemDetail` assembly — markers, chapters, subtitle list, versions) and then **strips every
|
||||
presigned URL**. The manifest is stable: a client can store it and re-read it offline indefinitely.
|
||||
Manifest + asset endpoints are only meaningful for managed device entries (`device_id` set).
|
||||
|
||||
```go
|
||||
type OfflineManifest struct {
|
||||
DownloadID string `json:"download_id"`
|
||||
ContentID string `json:"content_id"`
|
||||
EpisodeID string `json:"episode_id,omitempty"`
|
||||
Type string `json:"type"` // movie | episode
|
||||
Revision int `json:"revision"`
|
||||
Quality string `json:"quality"`
|
||||
EffectiveQuality string `json:"effective_quality"`
|
||||
DeliveryFormat string `json:"delivery_format"`
|
||||
TargetBitrateKbps int `json:"target_bitrate_kbps"`
|
||||
MediaFileID int `json:"media_file_id"`
|
||||
FileSize int64 `json:"file_size"`
|
||||
|
||||
Title string `json:"title"`
|
||||
Year int `json:"year,omitempty"`
|
||||
Overview string `json:"overview,omitempty"`
|
||||
Runtime int `json:"runtime,omitempty"`
|
||||
ContentRating string `json:"content_rating,omitempty"`
|
||||
Genres []string `json:"genres,omitempty"`
|
||||
SeriesID string `json:"series_id,omitempty"`
|
||||
SeriesTitle string `json:"series_title,omitempty"`
|
||||
SeasonNumber *int `json:"season_number,omitempty"`
|
||||
EpisodeNumber *int `json:"episode_number,omitempty"`
|
||||
|
||||
// Artwork: stable thumbhashes inline (immediate offline placeholder) + authenticated proxy URLs
|
||||
// the client downloads ONCE at download time. Never presigned S3 URLs.
|
||||
PosterThumbhash string `json:"poster_thumbhash,omitempty"`
|
||||
BackdropThumbhash string `json:"backdrop_thumbhash,omitempty"`
|
||||
ArtworkURLs struct {
|
||||
Poster string `json:"poster,omitempty"`
|
||||
Backdrop string `json:"backdrop,omitempty"`
|
||||
Logo string `json:"logo,omitempty"`
|
||||
} `json:"artwork_urls"`
|
||||
|
||||
// Playback metadata
|
||||
Container string `json:"container"`
|
||||
CodecVideo string `json:"codec_video"`
|
||||
CodecAudio string `json:"codec_audio"`
|
||||
Resolution string `json:"resolution"`
|
||||
HDR bool `json:"hdr"`
|
||||
Duration int `json:"duration_seconds"`
|
||||
|
||||
Chapters []OfflineChapter `json:"chapters,omitempty"`
|
||||
Intro *Marker `json:"intro,omitempty"`
|
||||
Credits *Marker `json:"credits,omitempty"`
|
||||
Recap *Marker `json:"recap,omitempty"`
|
||||
Preview *Marker `json:"preview,omitempty"`
|
||||
|
||||
Subtitles []OfflineSubtitle `json:"subtitles"`
|
||||
|
||||
// Stable identity so the client can re-resolve content_id after a server-side rescan
|
||||
// (mirrors userstore.WatchIdentity at internal/userstore/types.go:130).
|
||||
StableIdentity OfflineIdentity `json:"stable_identity"`
|
||||
|
||||
ManifestVersion int `json:"manifest_version"` // bump on DTO shape change
|
||||
GeneratedAt string `json:"generated_at"` // RFC3339
|
||||
}
|
||||
|
||||
type OfflineSubtitle struct {
|
||||
Language string `json:"language"`
|
||||
Format string `json:"format"` // srt | ass | vtt
|
||||
Forced bool `json:"forced"`
|
||||
HearingImpaired bool `json:"hearing_impaired"`
|
||||
External bool `json:"external"`
|
||||
FetchURL string `json:"fetch_url"` // authenticated proxy; client downloads at download time
|
||||
FileSize int64 `json:"file_size,omitempty"`
|
||||
}
|
||||
```
|
||||
|
||||
**Artwork/subtitle proxies** (replace presigning):
|
||||
- `GET /downloads/{id}/artwork/{kind}` (`kind` ∈ `poster|backdrop|logo`) reads the raw stored path and
|
||||
**streams the bytes** through the existing image resolver instead of redirecting to a presigned URL.
|
||||
The client caches these once.
|
||||
- `GET /downloads/{id}/subtitles/{ref}` (`ref` encodes `external:{index}` or `downloaded:{id}`) streams
|
||||
the subtitle file (external sub from disk; downloaded sub from S3 via the existing subtitle manager).
|
||||
|
||||
Both are authorized under the **full managed-entry rule** (see "Security & reliability invariants"):
|
||||
the row must match `(user_id, profile_id, header device_id)`, and content/library access is **re-checked
|
||||
for the requesting profile** before any bytes are served (a download id alone never authorizes access).
|
||||
They are session-authenticated rather than time-limited tokens, so a manifest stored on-device stays valid.
|
||||
|
||||
## Progress reconciliation
|
||||
|
||||
Two **additive** changes. The naïve version (route the client's wall-clock straight into
|
||||
`SetProgressIfNewer` and drive `?since=` off the same column) is **unsafe**: a skewed or malicious device
|
||||
can submit a far-future timestamp, win LWW permanently over genuinely-newer server activity, and poison
|
||||
the cursor (a future-dated row makes a reader advance its cursor past real writes). So the design
|
||||
**separates client event time from server ordering**.
|
||||
|
||||
The watch-progress row gains two server-owned facets (LWW key vs. cursor ordering):
|
||||
- **`event_at`** — the client-observed wall-clock for the progress event, used as the **LWW comparison
|
||||
key**, but **bounded on ingest**: `event_at = min(client_updated_at, server_now + skew)` with a small
|
||||
skew window (e.g. 2m); a value past the window is clamped to `server_now` (and logged). For an online
|
||||
write `event_at = server_now`. Bounding means a client can at most claim "now" for its **own profile's**
|
||||
progress — which it is already entitled to do — and can never lock in a far-future value.
|
||||
- **`synced_seq`** — a **server-assigned monotonic** marker set on every write (a sequence, or
|
||||
server-ingest timestamp with a tie-breaker). It is **never** client-influenced and is the sole basis
|
||||
for the `?since=` cursor.
|
||||
|
||||
**Write — `POST /api/v1/sync/progress`** (`internal/api/handlers/progress.go:230`): add an optional
|
||||
`updated_at` (RFC3339) per item = the client event time. The handler clamps it (above) and merges via an
|
||||
extended `SetProgressIfNewer` whose comparison is on the **bounded `event_at`**, while the server stamps
|
||||
`synced_seq` itself. Absent `updated_at` → behavior is exactly as today (server `now()`), so existing
|
||||
callers (incl. watchsync) are unaffected. Completion is never inferred from the timestamp alone — the
|
||||
existing watched-threshold logic still gates `completed`.
|
||||
|
||||
```go
|
||||
type syncProgressItem struct {
|
||||
MediaItemID string `json:"media_item_id"`
|
||||
Position float64 `json:"position"`
|
||||
Duration float64 `json:"duration"`
|
||||
ForceOverwrite bool `json:"force_overwrite"`
|
||||
UpdatedAt *string `json:"updated_at,omitempty"` // NEW: RFC3339; client EVENT time (server clamps it)
|
||||
}
|
||||
```
|
||||
|
||||
**Read — `GET /api/v1/progress?since=<cursor>`** (`internal/api/handlers/progress.go:86`): the `since`
|
||||
cursor is an **opaque server token over `synced_seq`**, not a client timestamp. When present, only rows
|
||||
with `synced_seq > cursor` are returned (use a sequence, or `>=` + id de-dup if a timestamp marker is
|
||||
used, to avoid tie loss). The response carries the next cursor. This makes delta delivery immune to
|
||||
client clock skew. Requires `UserStore.ListProgressSince(ctx, profileID, cursor)` in both backends
|
||||
(`internal/userdb`, `internal/userstore/pgstore`) ordering/filtering on `synced_seq`. Absent `since` →
|
||||
identical to today.
|
||||
|
||||
Adding a `UserStore` method (and the `event_at`/`synced_seq` columns to the per-user `watch_progress`
|
||||
schema) touches every stub implementation; ship the implementations, the schema change, and the shared
|
||||
store test suite update (`internal/userstore/storetest`) in the same PR.
|
||||
|
||||
**Net safety:** honest clients still get correct "most-recent actual watch wins" LWW; a dishonest clock
|
||||
is bounded to claiming "now" for its own profile (no authority it didn't already have), and cursors —
|
||||
the cross-device delivery mechanism — never depend on a client clock at all.
|
||||
|
||||
## API surface (one `/downloads/*` family; existing endpoints reshaped)
|
||||
|
||||
The web app is updated in lockstep, so existing endpoints are reshaped rather than frozen.
|
||||
|
||||
**Reshaped (existing):**
|
||||
```
|
||||
POST /downloads body {content_id, episode_id?, file_id?, quality?, series?, caps?};
|
||||
X-Silo-Device-Id present → managed entry; absent → ephemeral/web row.
|
||||
`quality` is original by default; bitrate presets are single-item only.
|
||||
GET /downloads managed entries for the CALLING device (device from header); ephemeral rows otherwise
|
||||
DELETE /downloads/{id} remove a row owned by (user, profile, header device)
|
||||
GET /downloads/{id}/file serve original (source) or prepared artifact (Range/resume + throttle)
|
||||
GET|HEAD /direct-download one-shot browser download, original-only (browser-friendly, token-in-URL)
|
||||
```
|
||||
The response DTO gains `device_id`, `quality`, `effective_quality`, `delivery_format`,
|
||||
`target_bitrate_kbps`, `revision`, artifact-derived readiness, and the new statuses.
|
||||
**`device_id` is authoritative only from the `X-Silo-Device-Id` header — never from the body or query**
|
||||
(a body/query value could only ever be a display hint, and is not accepted as authority).
|
||||
|
||||
**Bitrate qualities are single-item only.** A series batch (`series:true`) and
|
||||
`/direct-download` are **original-quality only**: `/direct-download` streams
|
||||
synchronously and cannot wait on an async encode, and a series batch would otherwise
|
||||
fan out N encode jobs. A client that wants prepared transcodes for a series requests
|
||||
each episode individually. `DownloadQualityResolver` rejects bulk bitrate requests
|
||||
with `ErrBulkQualityUnavailable` / HTTP 501.
|
||||
|
||||
**New — capability / feature detection** (pattern from `internal/api/handlers/notifications.go:324`):
|
||||
```
|
||||
GET /downloads/capability
|
||||
→ { "enabled": bool, "download_allowed": bool,
|
||||
"quality_presets": ["original","20mbps","10mbps","5mbps","2mbps","1mbps"],
|
||||
"transcode_enabled": bool, "transcode_user_allowed": bool,
|
||||
"season_download": bool, "series_monitoring": bool }
|
||||
```
|
||||
|
||||
**New — managed-entry lifecycle + offline bundle:**
|
||||
```
|
||||
PATCH /downloads/{id} body {status:"downloading"|"completed"}
|
||||
GET /downloads/{id}/manifest → OfflineManifest
|
||||
GET /downloads/batches/{batch_id}/manifests → {manifests:[OfflineManifest]}
|
||||
GET /downloads/{id}/artwork/{kind} → image bytes (poster|backdrop|logo)
|
||||
GET /downloads/{id}/subtitles/{ref} → subtitle bytes
|
||||
```
|
||||
|
||||
**Progress reconciliation:** additive fields on existing `POST /sync/progress` and `GET /progress`
|
||||
(above). No new routes.
|
||||
|
||||
Routes mount in the existing authenticated group where the current download routes live
|
||||
(`internal/api/router.go:2121`). **Every managed-entry endpoint — create, list, `PATCH`, `DELETE`,
|
||||
`/file`, `/manifest`, `/artwork`, `/subtitles` — is profile-scoped (`RequireProfile`) and authorizes the
|
||||
row on `(user_id, profile_id, header device_id)`** (not `user_id` alone — household profiles share a
|
||||
`user_id`, so a user-only check would leak one profile's downloads to another). A managed call requires
|
||||
`X-Silo-Device-Id` (missing header on a managed create → `400 device_id_required`); ephemeral web rows
|
||||
omit it and stay account-scoped. Byte-serving and asset endpoints additionally **re-check per-profile
|
||||
content/library access** (the existing access filter / `EnsureAccessible`) before serving, so a stale or
|
||||
out-of-scope row — e.g. a child profile, or a profile that lost library access — cannot pull restricted
|
||||
media by download id even if the row exists.
|
||||
|
||||
## Data flows
|
||||
|
||||
**Ephemeral web download (reshaped existing flow, quality-aware)**
|
||||
```
|
||||
POST /downloads {content_id, quality:"original", caps:{...}} (no device id → ephemeral)
|
||||
→ Resolve quality/delivery · permission · quota
|
||||
→ compatibility artifact: ArtifactManager.Ensure → preparing|ready ; direct original: ready immediately
|
||||
client → GET /downloads/{id}/file → serveContent(source|artifact, Range, throttle)
|
||||
```
|
||||
|
||||
**Register a managed bitrate download (mobile)**
|
||||
```
|
||||
POST /downloads {content_id, quality:"5mbps", caps:{...}} (+X-Silo-Device-Id, +X-Profile-Id)
|
||||
→ Resolve: cfg.TranscodeEnabled && user.DownloadTranscodeAllowed → delivery_format=transcode
|
||||
→ upsert downloads (device entry, status=preparing)
|
||||
→ ArtifactManager.Ensure(media_file_id, params): ON CONFLICT dedup → enqueue encode if new
|
||||
encode worker → playback.PrepareFile(... -movflags +faststart) → rename → artifact=ready
|
||||
→ linked rows=ready → events hub notify
|
||||
client (on notify) → GET /downloads/{id}/file → serveContent(artifact, Range, throttle)
|
||||
→ GET /downloads/{id}/manifest → download ArtworkURLs.* and Subtitles[].FetchURL once
|
||||
→ PATCH /downloads/{id} {status:"completed"}
|
||||
```
|
||||
|
||||
**Offline progress, reconnect**
|
||||
```
|
||||
(offline) client queues {media_item_id, position, duration, updated_at} per stop
|
||||
(online) POST /sync/progress {items:[... updated_at ...]} → SetProgressIfNewer per item
|
||||
GET /progress?since=<cursor> → rows changed elsewhere since last sync
|
||||
```
|
||||
|
||||
## Phased build sequence (each phase independently shippable)
|
||||
|
||||
- **Phase 0 — Reshape + scaffolding (coordinated web update).** Create `internal/downloads`; reshape the
|
||||
`downloads` table (device plus quality/delivery columns, widened status) and the `/downloads` +
|
||||
`/direct-download` endpoints to be quality-aware; port `BandwidthManager`/`QuantityLimiter`/serving;
|
||||
extend `config.DownloadConfig` (+ new keys, default-off); add quality/delivery constants + sentinel
|
||||
errors; add `GET /downloads/capability`. Update the web app's download hooks/components
|
||||
(`web/src/hooks/queries/downloads.ts`, `web/src/components/DownloadVersionPicker.tsx`) to the reshaped
|
||||
DTOs. Delete `internal/download`.
|
||||
- **Phase 1 — Managed device entries.** Device-aware create/list/patch/delete; managed-entry serving for
|
||||
`original`; `X-Silo-Device-Id` handling + the device-entry unique constraint. **Invariant 2** lands
|
||||
here: profile+device authorization on every managed endpoint, header-only `device_id` authority.
|
||||
*Acceptance tests:* a second profile on the same `user_id` is denied a managed row's `/file` and
|
||||
`PATCH`/`DELETE`; a body/query `device_id` cannot override the header.
|
||||
- **Phase 2 — Offline manifest.** `ManifestBuilder`; manifest + artwork + subtitle proxy endpoints, each
|
||||
under the invariant-2 authorization + a per-profile content-access re-check before serving.
|
||||
*Acceptance test:* a child/library-restricted profile is denied artwork/subtitle/manifest for an
|
||||
out-of-scope download id.
|
||||
- **Phase 3 — Prepare-to-file (remux + transcode) [v1-critical].** `download_artifacts` migration (incl.
|
||||
lease/attempt columns); `ArtifactManager` as a **durable leased queue** (transactional claim, lease
|
||||
heartbeat, startup sweep, `.part` cleanup) per **invariant 3**; `playback.PrepareFile`; encode worker on
|
||||
`taskmanager`; ready-state serving; events notification; cleanup job. Turn `download.transcode_enabled`
|
||||
on. *Acceptance test:* kill the process mid-encode → on restart the artifact re-enqueues and the linked
|
||||
download reaches `ready` (no permanent `preparing`); concurrent workers never double-encode.
|
||||
- **Phase 4 — Progress reconciliation.** `event_at` (clamped) + server-assigned `synced_seq` columns on
|
||||
`watch_progress`; clamped `updated_at` on `POST /sync/progress`; server-token `?since=` on
|
||||
`GET /progress`; `ListProgressSince` in both stores + store test suite, per **invariant 1**.
|
||||
*Acceptance test:* a future-dated client `updated_at` is clamped (cannot win over a later real write)
|
||||
and never advances another device's cursor.
|
||||
|
||||
Phase 0 precedes all and is the coordinated server+web reshape. Phases 1/2/4 are independent; Phase 3
|
||||
depends on 1. Remux can ship in Phase 3 even if transcode stays admin-off.
|
||||
|
||||
## Client impact
|
||||
|
||||
- **Android (`silo-android`):** persist a stable `device_id` (UUID in app storage) and send
|
||||
`X-Silo-Device-Id`; call `/downloads/capability` on login to gate UI; register downloads, poll/listen
|
||||
for `ready`, pull the file, store the manifest + artwork + subtitle files locally; play the local file
|
||||
with ExoPlayer; queue offline progress with timestamps in a Room table and flush via
|
||||
`POST /sync/progress`, then pull deltas with `GET /progress?since=`.
|
||||
- **Apple (`silo-apple`):** same flow; `device_id` from `identifierForVendor` (or stored UUID on macOS);
|
||||
`URLSession` background download against `GET /downloads/{id}/file`; AVFoundation plays the local file;
|
||||
offline progress queue drained on reconnect. The manifest's markers/chapters reuse the types already
|
||||
used for online playback.
|
||||
- **Web (`web/`):** updated in lockstep in **Phase 0** — the download hooks/components move to the
|
||||
reshaped `/downloads` DTOs (quality/effective quality/delivery fields, new statuses). New admin additions: a
|
||||
`download.transcode_enabled` toggle in admin settings, plus the existing per-user
|
||||
`download_transcode_allowed` switch in `web/src/pages/AdminUserDetail.tsx`. No offline-library UI on web.
|
||||
|
||||
## Security & reliability invariants (must-hold)
|
||||
|
||||
These three are correctness requirements, not nice-to-haves; each ships with the acceptance test named in
|
||||
the phase plan. They were hardened in response to an adversarial design review.
|
||||
|
||||
1. **Server-owned sync ordering.** Cross-device delta delivery (`?since=`) is driven only by the
|
||||
server-assigned `synced_seq`; the client clock is bounded metadata (`event_at`, clamped to
|
||||
`now + skew`) used only for LWW conflict resolution on the caller's own profile. A client-supplied
|
||||
timestamp can never advance another reader's cursor or lock in a far-future "win."
|
||||
2. **Full profile + device authorization.** Every managed-entry endpoint authorizes on
|
||||
`(user_id, profile_id, header device_id)` and re-checks per-profile content/library access before
|
||||
serving bytes/assets. A download id is never sufficient; `device_id` authority is the header only.
|
||||
This holds the "no cross-device visibility" and "no cross-profile leakage" invariants given that
|
||||
household profiles share a `user_id`.
|
||||
3. **Durable artifact recovery.** Artifact preparation is a leased, attempt-counted, transactionally
|
||||
claimed queue with a startup sweep, so no crash/restart can strand a download in `preparing` or
|
||||
double-encode the same artifact.
|
||||
|
||||
## Risks & trade-offs
|
||||
|
||||
- **Coordinated server+web reshape (Phase 0).** Reshaping `/downloads` requires the web app to land in
|
||||
the same milestone. Mitigated because both are first-party and the reshape is mechanical (added fields,
|
||||
not removed behavior for the existing web path).
|
||||
- **One table, two lifecycles.** A nullable `device_id` plus a wide status enum carries both ephemeral
|
||||
and managed rows. Deliberate and managed (partial unique index + MATCH-SIMPLE FK keep the two modes'
|
||||
invariants clean); the alternative two-table split was only justified by the now-lifted compat
|
||||
constraint.
|
||||
- **Artifact disk + CPU cost.** Mitigated by `(media_file_id, format, params_hash)` dedup (shared across
|
||||
web/devices/users), a bounded encode pool, LRU/byte-budget cleanup, and transcode being **off by
|
||||
default** (admin opt-in).
|
||||
- **Remux is not free for a clean downloadable file.** `+faststart` needs a finalization pass, so even
|
||||
remux is a prepare-to-file job. It is fast (`-c copy`) but not instant; `preparing → ready` + push
|
||||
notification covers the UX.
|
||||
- **`content_id` can change on rescan.** The manifest carries `StableIdentity` (provider IDs +
|
||||
season/episode) mirroring `WatchIdentity` so the client re-resolves after a catalog rebind; the stale
|
||||
row doesn't break the on-device file.
|
||||
- **No expiry by decision.** A revoked `download_allowed` stops *future* serves (re-checked each serve,
|
||||
as today at `internal/download/service.go:345`) but cannot remove an already-downloaded file. Call this
|
||||
out in admin copy.
|
||||
- **LWW vs. client clocks (resolved, see invariant 1).** Conflict resolution uses a clamped `event_at`
|
||||
and delivery uses the server-owned `synced_seq`, so a skewed/malicious clock is bounded to "now" on its
|
||||
own profile and cannot poison cursors. Residual: an honest-but-wrong client clock within the skew
|
||||
window can still mis-order two of *its own* near-simultaneous events — acceptable.
|
||||
- **Authorization breadth (resolved, see invariant 2).** The per-endpoint profile+device authorization
|
||||
and pre-serve access re-check are load-bearing; the acceptance suite must include a cross-profile and a
|
||||
cross-device denial test, since a regression here is a content-access leak, not just a bug.
|
||||
- **`UserStore` gains a method + `watch_progress` gains columns.** `ListProgressSince` plus the
|
||||
`event_at`/`synced_seq` columns touch every store implementor/stub and the per-user schema — all in the
|
||||
Phase 4 PR.
|
||||
- **Quota semantics.** v1: all download rows (web + device) count toward the **same** per-user
|
||||
`QuantityLimiter`; revisit if too coarse.
|
||||
|
||||
## Out of scope (deferred)
|
||||
|
||||
Cross-device download visibility; download expiry/lease/DRM; cumulative per-user storage-byte quotas;
|
||||
push-to-client "your transcode is ready" beyond the existing events hub; server-initiated deletion of
|
||||
client-side files.
|
||||
|
||||
## Appendix — proposal-ready summary (maps to the v1 capability template)
|
||||
|
||||
- **Summary:** Mobile apps can download movies/episodes for fully offline playback — choosing original
|
||||
or an admin-gated bitrate quality — and watch progress stays consistent when the device reconnects.
|
||||
The web app's existing download feature is folded into the same reshaped contract. Remux remains an
|
||||
internal delivery format, not a client-facing quality preset.
|
||||
- **User value / why v1:** Offline playback is table-stakes for mobile media clients (commutes, flights,
|
||||
spotty networks) and the client teams need the download + sync contract locked to build against.
|
||||
Reshaping the existing download contract now — before lock and before any mobile client ships against
|
||||
it — is the right time; the only current consumer is the first-party web app, updated in lockstep.
|
||||
- **API surface:** the reshaped + new `/downloads/*` endpoints listed in "API surface" above (capability,
|
||||
device-aware create/list/patch, file serving, manifest/artwork/subtitle proxies) plus
|
||||
`updated_at`/`?since=` on the progress endpoints. **Note for reviewers:** this *changes* the existing
|
||||
`/downloads` contract (additive-only is intentionally waived pre-lock for this first-party endpoint).
|
||||
- **Client impact:** Android — full offline UI + sync; Apple — full offline UI + sync; Web — lockstep
|
||||
Phase-0 DTO update + admin toggles.
|
||||
- **Rough size (server-side):** **L** (new package replacing the old one, table reshape + artifacts
|
||||
table, async encode pipeline, manifest assembly, additive progress sync).
|
||||
@@ -0,0 +1,220 @@
|
||||
# Download limits & restrictions (downloads v2) — design
|
||||
|
||||
**Goal:** Extend downloads v2 with the restriction knobs admins actually need: (1) a **quality
|
||||
ceiling** — server-wide and per-user — so a user can be allowed to download, even transcode, but
|
||||
only up to a capped bitrate; (2) a **batch size cap** protecting the server from a single series
|
||||
request fanning out into hundreds of rows and transcode jobs; (3) **per-user overrides** for the
|
||||
existing global quantity and bandwidth limits, following the established `users.max_streams` /
|
||||
`users.max_transcodes` pattern, so households can tier users without changing the global policy.
|
||||
|
||||
**Status:** Draft — design input for a v1 capability proposal (Silo v1 scope gate,
|
||||
`docs/architecture/v1-scope.md`). All API changes are additive within `/api/v1`.
|
||||
|
||||
Decisions locked with the requester:
|
||||
- **No device/user storage caps.** Device disk is the client's job (per-subscription
|
||||
`max_storage_bytes` already covers the monitoring case); server disk is already covered by
|
||||
`download.artifact_max_bytes`; serving volume is already covered by bandwidth + per-period quotas.
|
||||
- **No expiry / lease / revocation of already-downloaded files.** Unchanged from the downloads v2
|
||||
design (`docs/superpowers/specs/2026-06-18-offline-sync-mobile-design.md`).
|
||||
- **Transcode restriction already exists** (`download.transcode_enabled` +
|
||||
`users.download_transcode_allowed`) and is kept as-is; the quality ceiling composes with it
|
||||
("transcode allowed, but only up to 5mbps").
|
||||
- **Admin observability (downloads dashboard, admin list/revoke endpoints) is a separate effort**,
|
||||
not part of this spec.
|
||||
|
||||
> Commands and paths in this document are repository-relative; assume the repository root is the cwd.
|
||||
|
||||
## Current state (what this builds on)
|
||||
|
||||
Enforcement today lives in `internal/downloads`:
|
||||
|
||||
- **Policy** — `policy.go`: `DownloadQualityResolver.Resolve` validates a requested quality from the
|
||||
ladder `original > 20mbps > 10mbps > 5mbps > 2mbps > 1mbps` and gates bitrate presets on
|
||||
`cfg.TranscodeEnabled && user.DownloadTranscodeAllowed`. `PresetsFor` advertises the fulfillable
|
||||
ladder through `GET /api/v1/downloads/capability` (`quality_presets`), so clients render only what
|
||||
the server will honor.
|
||||
- **Quantity** — `limiter.go`: `QuantityLimiter.Check(ctx, userID, batchSize)` enforces the global
|
||||
`download.max_concurrent_per_user` and `download.max_per_period` / `download.period_duration`
|
||||
settings (0 = unlimited) against `CountActiveByUser` / `CountByUserSince`.
|
||||
- **Bandwidth** — `bandwidth.go`: `BandwidthManager` token-buckets file serving with one server-wide
|
||||
rate and one per-user rate (`download.server_bandwidth_mbps`, `download.user_bandwidth_mbps`),
|
||||
caching one `rate.Limiter` per user id.
|
||||
- **Per-user flags** — `users.download_allowed`, `users.download_transcode_allowed`
|
||||
(`internal/auth/repository.go`), exposed on the admin users API (`internal/api/handlers/admin.go`)
|
||||
and web UI (`web/src/pages/AdminUserDetail.tsx`).
|
||||
- **Bulk** — `service.go` `createSeriesScoped` fans a series/season request into managed entries with
|
||||
no upper bound; `resolveBulkQuality` intentionally keeps batches **original-only**
|
||||
(`ErrBulkQualityUnavailable`). Subscription sync auto-registers in-scope episodes without
|
||||
consuming the quantity limiter (by design).
|
||||
- **Serve-time re-check** — `Service.ServeFile` re-resolves policy on every file request, so
|
||||
disabling downloads or revoking a user flag takes effect immediately for anything not yet pulled.
|
||||
|
||||
Settings are freeform keys with defaults and validation in `internal/config/db_loader.go`, hot-
|
||||
reloaded by `Service.loadConfig` (which also reloads the limiter and bandwidth manager on change),
|
||||
and edited in `web/src/pages/admin-settings/DownloadSettings.tsx`.
|
||||
|
||||
## 1. Quality ceiling
|
||||
|
||||
### Semantics
|
||||
|
||||
- New server setting **`download.max_quality`** (string; `""` = no cap; otherwise one of the quality
|
||||
ladder values). New nullable per-user column **`users.download_max_quality`** (`NULL` = inherit the
|
||||
server setting; otherwise one of the ladder values).
|
||||
- The **effective ceiling** for a user is the *lower* of the server setting and the user column on
|
||||
the ordered ladder (`original` is the top). A ceiling below `original` deliberately forbids
|
||||
`original` — that is the point: "this user gets at most a 5mbps transcode, never the full file."
|
||||
- Enforcement is **presets-first**: `PresetsFor` truncates the advertised ladder at the effective
|
||||
ceiling, so compliant clients never offer a forbidden quality and **no client changes are
|
||||
required** (clients already read `quality_presets` from the capability endpoint).
|
||||
- `Resolve` independently rejects any request above the ceiling with a new sentinel
|
||||
`ErrQualityNotPermitted` (handler maps it alongside the existing limit errors as HTTP 403,
|
||||
distinct from `ErrInvalidQuality`), so a non-compliant client cannot bypass the cap.
|
||||
- The **original-compatibility fallback** inside `Resolve` (original → remux/transcode when the
|
||||
device can't play the source) must clamp its chosen `EffectiveQuality`/bitrate to the ceiling.
|
||||
In practice a sub-original ceiling makes the fallback moot — `original` is already rejected — but
|
||||
the clamp keeps the invariant local to `Resolve` rather than depending on call-site ordering.
|
||||
- **Serve-time**: the existing `ServeFile` policy re-check additionally denies rows whose
|
||||
`effective_quality` exceeds the user's *current* ceiling. Consistent with today's revocation
|
||||
semantics: lowering a cap stops future serving of higher-quality registered entries; files already
|
||||
on-device are untouched (no expiry).
|
||||
|
||||
### Interactions (documented behavior, not bugs)
|
||||
|
||||
- **Ceiling below `original` + `download_transcode_allowed = false`** → the fulfillable preset list
|
||||
is empty and the capability endpoint reports no presets; the user effectively cannot download.
|
||||
This is legal admin configuration. The admin UI shows a hint on the per-user field when this
|
||||
combination is selected ("a sub-original cap requires the transcode permission").
|
||||
- **Series/season/monitoring downloads are original-only today** (`resolveBulkQuality`). A user
|
||||
capped below `original` therefore cannot create series/season batches or receive subscription
|
||||
auto-registrations; those paths return `ErrQualityNotPermitted` (create) or skip registration with
|
||||
a `SkippedDownload` reason `quality_not_permitted` (subscription sync). This is accepted for now
|
||||
and called out here so it is not rediscovered as a bug. Lifting it means bulk transcode — a
|
||||
deliberate follow-up, not smuggled into this change.
|
||||
|
||||
### Implementation
|
||||
|
||||
- `Resolve`/`PresetsFor` gain the ceiling input. Rather than growing the parameter lists further,
|
||||
introduce a small `PolicyInput`/`userPolicy` struct (user + cfg + artifactsAvailable) if the
|
||||
signatures get unwieldy — implementer's choice; behavior is what is specified above.
|
||||
- Ladder comparison helper in `internal/downloads/model.go` (e.g. `QualityAtMost(q, ceiling) bool`
|
||||
using the `QualityPresets` order) — no string parsing of bitrates.
|
||||
- Validation: `db_loader.go` rejects a `download.max_quality` value not on the ladder; the users
|
||||
API rejects invalid `download_max_quality` the same way (shared validation via
|
||||
`downloads.ValidQuality`, with `""`/`NULL` allowed).
|
||||
|
||||
## 2. Batch size cap
|
||||
|
||||
- New server setting **`download.max_batch_size`** (int; default **100**; 0 = unlimited): the
|
||||
maximum number of managed entries one explicit series/season create may register, and the maximum
|
||||
auto-registered per subscription-sync pass.
|
||||
- **Explicit series/season create** (`createSeriesScoped`): if the in-scope episode count (after
|
||||
access filtering and skip computation) exceeds the cap, **reject** the whole request with a new
|
||||
sentinel `ErrBatchTooLarge` (HTTP 400, error body includes the cap so clients can message "try a
|
||||
season at a time or subscribe"). Rejection is chosen over truncation because a silently partial
|
||||
one-shot batch never self-heals — there is no later pass that picks up the remainder.
|
||||
- **Subscription sync** (`subscription_service.go`): **truncate** each pass at the cap (stable
|
||||
episode order). Truncation is correct here because sync recomputes in-scope-but-unregistered
|
||||
episodes every pass, so the remainder registers on subsequent syncs — natural pagination, and one
|
||||
monster series cannot monopolize a sync pass or the artifact queue.
|
||||
- The capability response additively gains **`max_batch_size`** (int, 0 = unlimited) so clients can
|
||||
preflight ("this series has 340 episodes; the server accepts 100 per request") instead of
|
||||
discovering the limit by error.
|
||||
|
||||
## 3. Per-user quantity & bandwidth overrides
|
||||
|
||||
- Three new nullable columns on `users`, mirroring the `max_streams` pattern
|
||||
(`internal/models/user.go`, `internal/auth/repository.go`):
|
||||
- `download_max_concurrent int` — overrides `download.max_concurrent_per_user`
|
||||
- `download_max_per_period int` — overrides `download.max_per_period` (window stays the global
|
||||
`download.period_duration`; a per-user window is deliberately out of scope)
|
||||
- `download_bandwidth_mbps int` — overrides `download.user_bandwidth_mbps`
|
||||
- Semantics: `NULL` = inherit the global setting; **0 = explicitly unlimited** (matches the global
|
||||
settings' zero-means-unlimited convention, and is the reason `NULL` and `0` must stay distinct).
|
||||
- **QuantityLimiter**: `Check` gains the caller-resolved user overrides — e.g.
|
||||
`Check(ctx, userID, batchSize, overrides Overrides)` where `Overrides` carries the two optional
|
||||
ints. Every existing call site already has the `*models.User` loaded (the create paths load it
|
||||
for policy), so resolution stays in `service.go` and the limiter remains storage-free.
|
||||
- **BandwidthManager**: the per-user limiter cache currently assumes one global rate. Store the
|
||||
rate alongside each cached limiter and recreate the entry when the effective rate for that user
|
||||
changes (`ThrottledReader` gains the resolved per-user bps from the caller; `ServeFile` already
|
||||
loads the user for its policy re-check). `Reload` keeps its existing clear-all behavior.
|
||||
- **Admin API**: the three fields (plus `download_max_quality` from §1) are added to the admin user
|
||||
create/update/response DTOs in `internal/api/handlers/admin.go` and to
|
||||
`UpdateUserInput`/`CreateUserInput` in `internal/models/user.go` +
|
||||
`internal/auth/repository.go` — the exact `max_streams` additive pattern.
|
||||
|
||||
## Data model (Goose migration)
|
||||
|
||||
One timestamped migration (`make migrate-create NAME=user_download_limits`):
|
||||
|
||||
```sql
|
||||
ALTER TABLE users
|
||||
ADD COLUMN download_max_quality text,
|
||||
ADD COLUMN download_max_concurrent integer,
|
||||
ADD COLUMN download_max_per_period integer,
|
||||
ADD COLUMN download_bandwidth_mbps integer;
|
||||
```
|
||||
|
||||
All nullable, no defaults, no backfill — `NULL` means "inherit global" for every column. The two new
|
||||
settings keys (`download.max_quality`, `download.max_batch_size`) are settings rows, not schema.
|
||||
|
||||
## API surface (all additive)
|
||||
|
||||
- `GET /api/v1/downloads/capability` — `quality_presets` now reflects the effective ceiling
|
||||
(existing field, narrowed values); new field `max_batch_size`.
|
||||
- `POST /api/v1/downloads` — new 403 error case (`quality_not_permitted`) and new 400 error case
|
||||
(`batch_too_large`, body includes the configured cap).
|
||||
- `POST /api/v1/downloads/subscriptions/sync` — skipped registrations may carry the new
|
||||
`SkippedDownload` reason `quality_not_permitted`.
|
||||
- Admin users endpoints — four new optional request fields / response fields (§1, §3).
|
||||
- Settings endpoints — two new keys; both hot-reloadable (no restart-required entry in
|
||||
`internal/config/restart_keys.go`).
|
||||
|
||||
## Web admin UI
|
||||
|
||||
- `web/src/pages/admin-settings/DownloadSettings.tsx`: "Max Download Quality" (select over the
|
||||
ladder + "No limit") under Quantity Limits or a new Restrictions group; "Max Batch Size" numeric
|
||||
field with the 0-=-unlimited hint.
|
||||
- `web/src/pages/AdminUserDetail.tsx` (+ the user edit dialog): rows for the four per-user fields,
|
||||
each showing the inherited global value when unset (as the streams/transcodes fields do), plus the
|
||||
§1 hint when a sub-original cap is combined with transcode-not-allowed.
|
||||
|
||||
## Enforcement summary
|
||||
|
||||
| Restriction | Hard/soft | Enforced at |
|
||||
|---|---|---|
|
||||
| Quality ceiling | Hard | `Resolve` (create), `PresetsFor` (capability), `ServeFile` re-check |
|
||||
| Batch size cap | Hard | `createSeriesScoped` (reject), subscription sync (truncate per pass) |
|
||||
| Concurrent / per-period overrides | Hard | `QuantityLimiter.Check` on every create path |
|
||||
| Bandwidth override | Hard | `ThrottledReader` on every byte served |
|
||||
|
||||
Nothing in this spec is client-enforced; the client-pull architecture is unchanged.
|
||||
|
||||
## Client follow-up (silo-android, silo-apple)
|
||||
|
||||
- **Quality ceiling / presets:** zero work if the client renders `quality_presets` from the
|
||||
capability endpoint (the downloads v2 contract). Audit for hardcoded ladders.
|
||||
- **Batch errors:** handle `batch_too_large` with a useful prompt (download per season / subscribe
|
||||
instead of a raw error toast). Handle empty `quality_presets` as "downloads unavailable for this
|
||||
account", which the capability contract already implies.
|
||||
- No manifest or sync-protocol changes.
|
||||
|
||||
## Testing
|
||||
|
||||
- `policy_test.go`: ceiling truncation of `PresetsFor` (server-only, user-only, both, ties at
|
||||
`original`); `Resolve` rejection above the ceiling; fallback clamp; the
|
||||
cap-without-transcode-permission → empty presets case.
|
||||
- `limiter_test.go` (new cases): override precedence NULL/0/positive vs. global for both limits.
|
||||
- `service_*_test.go`: batch reject at cap boundary (cap, cap+1); subscription sync truncation and
|
||||
next-pass continuation; serve-time denial after a cap is lowered.
|
||||
- Handler tests for the new error mappings and admin-user field round-trips — run in a
|
||||
libvips-capable container (bare-host `go test ./...` skips `internal/api/handlers`).
|
||||
|
||||
## Out of scope / follow-ups
|
||||
|
||||
- Admin downloads dashboard + admin list/revoke endpoints (observability; separate spec).
|
||||
- Bulk (series/season) transcode quality — prerequisite for capped users to use series downloads
|
||||
and monitoring; lifts the `resolveBulkQuality` original-only rule.
|
||||
- Profile-level download permission (parental-controls integration; downloads are already keyed by
|
||||
`profile_id`, so the data model is ready).
|
||||
- Per-user managed-device count cap — revisit only if storage/quantity limits prove insufficient.
|
||||
@@ -86,9 +86,15 @@ func (s stubStore) GetProgress(context.Context, string, string) (*userstore.Watc
|
||||
func (s stubStore) ListProgress(context.Context, string, string, int, int) ([]userstore.WatchProgress, error) {
|
||||
panic("unused")
|
||||
}
|
||||
func (s stubStore) ListProgressFiltered(context.Context, string, string, []string, *int, int, int) ([]userstore.WatchProgress, error) {
|
||||
panic("unused")
|
||||
}
|
||||
func (s stubStore) ListProgressByMediaItems(context.Context, string, []string) (map[string]userstore.WatchProgress, error) {
|
||||
panic("unused")
|
||||
}
|
||||
func (s stubStore) ListProgressSince(context.Context, string, string) ([]userstore.WatchProgress, string, error) {
|
||||
panic("unused")
|
||||
}
|
||||
func (s stubStore) AddHistory(context.Context, userstore.WatchHistoryEntry) error { panic("unused") }
|
||||
func (s stubStore) AddHistoryIfMissing(context.Context, userstore.WatchHistoryEntry) (bool, error) {
|
||||
panic("unused")
|
||||
@@ -130,9 +136,18 @@ func (s stubStore) ListFavorites(context.Context, string, int, int) ([]userstore
|
||||
func (s stubStore) ListFavoritesByMediaItems(context.Context, string, []string) (map[string]bool, error) {
|
||||
panic("unused")
|
||||
}
|
||||
func (s stubStore) IsFavorite(context.Context, string, string) (bool, error) { panic("unused") }
|
||||
func (s stubStore) AddToWatchlist(context.Context, string, string) error { panic("unused") }
|
||||
func (s stubStore) IsFavorite(context.Context, string, string) (bool, error) { panic("unused") }
|
||||
func (s stubStore) AddToWatchlist(context.Context, string, string) error { panic("unused") }
|
||||
func (s stubStore) AddToWatchlistAt(context.Context, string, string, time.Time) error {
|
||||
panic("unused")
|
||||
}
|
||||
func (s stubStore) RemoveWatchedFromWatchlist(context.Context, string) (bool, error) {
|
||||
return true, nil
|
||||
}
|
||||
func (s stubStore) RemoveFromWatchlist(context.Context, string, string) error { panic("unused") }
|
||||
func (s stubStore) ReplaceWatchlistOrder(context.Context, string, []string) error {
|
||||
panic("unused")
|
||||
}
|
||||
func (s stubStore) ListWatchlist(context.Context, string, int, int) ([]userstore.WatchlistEntry, error) {
|
||||
panic("unused")
|
||||
}
|
||||
|
||||
@@ -89,6 +89,7 @@ func (l *PGLibraryRefreshItemLister) ListLibraryItems(ctx context.Context, libra
|
||||
query := `
|
||||
SELECT mi.content_id, COALESCE(mi.tmdb_id, ''), COALESCE(mi.tvdb_id, ''), COALESCE(mi.imdb_id, '')
|
||||
FROM media_item_libraries mil
|
||||
JOIN media_folders f ON f.id = mil.media_folder_id
|
||||
JOIN media_items mi ON mi.content_id = mil.content_id
|
||||
WHERE mil.media_folder_id = $1`
|
||||
args := []any{libraryID}
|
||||
@@ -123,6 +124,11 @@ func (l *PGLibraryRefreshItemLister) ListLibraryItems(ctx context.Context, libra
|
||||
FROM stale_media_ids smi
|
||||
WHERE smi.content_id = mi.content_id
|
||||
)
|
||||
OR (
|
||||
COALESCE(mi.default_metadata_language, '') <> ''
|
||||
AND LOWER(TRIM(mi.default_metadata_language))
|
||||
<> LOWER(TRIM(COALESCE(NULLIF(f.metadata_language, ''), 'en')))
|
||||
)
|
||||
)`
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
package adminjob
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// TestQuickRefreshListsLanguageMismatchedItems covers the listing half of
|
||||
// issue #211: quick-mode library refresh must include items whose stamped
|
||||
// default_metadata_language differs from the library's configured metadata
|
||||
// language, even when the item is otherwise complete (overview, poster,
|
||||
// backdrop all present) — otherwise a library language change never revisits
|
||||
// already-complete items.
|
||||
func TestQuickRefreshListsLanguageMismatchedItems(t *testing.T) {
|
||||
dsn := os.Getenv("SILO_TEST_DATABASE_URL")
|
||||
if dsn == "" {
|
||||
t.Skip("SILO_TEST_DATABASE_URL is not set")
|
||||
}
|
||||
ctx := context.Background()
|
||||
pool, err := pgxpool.New(ctx, dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("connect test database: %v", err)
|
||||
}
|
||||
t.Cleanup(pool.Close)
|
||||
|
||||
suffix := time.Now().UnixNano()
|
||||
mismatchID := fmt.Sprintf("lang-mismatch-%d", suffix)
|
||||
matchedID := fmt.Sprintf("lang-matched-%d", suffix)
|
||||
|
||||
var folderID int
|
||||
if err := pool.QueryRow(ctx, `
|
||||
INSERT INTO media_folders (type, name, enabled, metadata_language)
|
||||
VALUES ('movies', 'Lang Refresh Test', true, 'da')
|
||||
RETURNING id
|
||||
`).Scan(&folderID); err != nil {
|
||||
t.Fatalf("seed folder: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
_, _ = pool.Exec(ctx, `DELETE FROM media_items WHERE content_id = ANY($1)`, []string{mismatchID, matchedID})
|
||||
_, _ = pool.Exec(ctx, `DELETE FROM media_folders WHERE id = $1`, folderID)
|
||||
})
|
||||
|
||||
for _, row := range []struct {
|
||||
id, lang string
|
||||
}{
|
||||
{mismatchID, "zh"},
|
||||
{matchedID, "da"},
|
||||
} {
|
||||
if _, err := pool.Exec(ctx, `
|
||||
INSERT INTO media_items (
|
||||
content_id, type, title, status, genres, tmdb_id,
|
||||
default_metadata_language, overview, poster_path, backdrop_path,
|
||||
last_refreshed, refresh_failures, episode_metadata_incomplete
|
||||
) VALUES ($1, 'movie', 'Complete Item', 'matched', '{}'::text[], '42',
|
||||
$2, 'An overview', '/p.jpg', '/b.jpg', NOW(), 0, FALSE)
|
||||
`, row.id, row.lang); err != nil {
|
||||
t.Fatalf("seed media item %s: %v", row.id, err)
|
||||
}
|
||||
if _, err := pool.Exec(ctx, `
|
||||
INSERT INTO media_item_libraries (content_id, media_folder_id)
|
||||
VALUES ($1, $2)
|
||||
`, row.id, folderID); err != nil {
|
||||
t.Fatalf("link media item %s: %v", row.id, err)
|
||||
}
|
||||
}
|
||||
|
||||
lister := NewPGLibraryRefreshItemLister(pool)
|
||||
items, err := lister.ListLibraryItems(ctx, folderID, LibraryRefreshModeQuick)
|
||||
if err != nil {
|
||||
t.Fatalf("ListLibraryItems: %v", err)
|
||||
}
|
||||
|
||||
var sawMismatch, sawMatched bool
|
||||
for _, item := range items {
|
||||
switch item.ContentID {
|
||||
case mismatchID:
|
||||
sawMismatch = true
|
||||
case matchedID:
|
||||
sawMatched = true
|
||||
}
|
||||
}
|
||||
if !sawMismatch {
|
||||
t.Errorf("quick refresh must include complete item with stamped language differing from the library language")
|
||||
}
|
||||
if sawMatched {
|
||||
t.Errorf("quick refresh must not include complete item whose stamped language matches the library language")
|
||||
}
|
||||
}
|
||||
@@ -2141,6 +2141,21 @@ func (h *AdminHandler) HandleUpdateSetting(w http.ResponseWriter, r *http.Reques
|
||||
"subtitle_ai.transcribe_quota_period must be day, week, or month")
|
||||
return
|
||||
}
|
||||
case notifications.SettingApplePushDeliveryEnabled:
|
||||
enabled, err := strconv.ParseBool(strings.TrimSpace(req.Value))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "notifications.apple_push_delivery_enabled must be true or false")
|
||||
return
|
||||
}
|
||||
req.Value = strconv.FormatBool(enabled)
|
||||
case notifications.SettingPushRelayURL, notifications.SettingPushRelayDeploymentID, notifications.SettingPushRelayAPIKey:
|
||||
// The registration flow persists the relay URL, deployment id, and API
|
||||
// key together; a direct write to any of them desyncs the stored URL
|
||||
// from the credentials the relay minted for it (and feeds an arbitrary
|
||||
// id into the next rotation request).
|
||||
writeError(w, http.StatusBadRequest, "bad_request",
|
||||
key+" is managed by the push relay registration flow; use POST /admin/notifications/push/relay/register")
|
||||
return
|
||||
case catalog.SearchSettingProvider:
|
||||
switch strings.TrimSpace(strings.ToLower(req.Value)) {
|
||||
case catalog.SearchProviderPostgres, catalog.SearchProviderMeilisearch:
|
||||
|
||||
@@ -0,0 +1,279 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/notifications"
|
||||
)
|
||||
|
||||
type AdminApplePushHandler struct {
|
||||
system *notifications.System
|
||||
settings ServerSettingsStore
|
||||
client httpDoer
|
||||
}
|
||||
|
||||
type httpDoer interface {
|
||||
Do(*http.Request) (*http.Response, error)
|
||||
}
|
||||
|
||||
func NewAdminApplePushHandler(system *notifications.System, settings ServerSettingsStore) *AdminApplePushHandler {
|
||||
return &AdminApplePushHandler{
|
||||
system: system,
|
||||
settings: settings,
|
||||
client: &http.Client{Timeout: 10 * time.Second},
|
||||
}
|
||||
}
|
||||
|
||||
type adminApplePushTestRequest struct {
|
||||
ProfileID string `json:"profile_id"`
|
||||
ServerDeviceID string `json:"server_device_id"`
|
||||
}
|
||||
|
||||
type adminApplePushTestResponse struct {
|
||||
AttemptID string `json:"attempt_id"`
|
||||
PushDeviceID string `json:"push_device_id"`
|
||||
ServerDeviceID string `json:"server_device_id"`
|
||||
Outcome string `json:"outcome"`
|
||||
RelayRequestID string `json:"relay_request_id,omitempty"`
|
||||
UpstreamStatus *int `json:"upstream_status,omitempty"`
|
||||
UpstreamReason string `json:"upstream_reason,omitempty"`
|
||||
FailureMessage string `json:"failure_message,omitempty"`
|
||||
}
|
||||
|
||||
type adminPushRelayRegisterRequest struct {
|
||||
RelayURL string `json:"relay_url"`
|
||||
}
|
||||
|
||||
type pushRelayRegisterRequest struct {
|
||||
DeploymentID string `json:"deployment_id,omitempty"`
|
||||
}
|
||||
|
||||
type pushRelayRegisterResponse struct {
|
||||
RequestID string `json:"request_id"`
|
||||
DeploymentID string `json:"deployment_id"`
|
||||
APIKey string `json:"api_key"`
|
||||
KeyPrefix string `json:"key_prefix"`
|
||||
APNsTopics []string `json:"apns_topics"`
|
||||
}
|
||||
|
||||
type pushRelayNestedError struct {
|
||||
Error struct {
|
||||
Code string `json:"code"`
|
||||
Message string `json:"message"`
|
||||
RequestID string `json:"request_id"`
|
||||
} `json:"error"`
|
||||
}
|
||||
|
||||
type adminPushRelayRegisterResponse struct {
|
||||
RelayURL string `json:"relay_url"`
|
||||
DeploymentID string `json:"deployment_id"`
|
||||
KeyPrefix string `json:"key_prefix"`
|
||||
APIKeyConfigured bool `json:"api_key_configured"`
|
||||
RelayRequestID string `json:"relay_request_id,omitempty"`
|
||||
APNsTopics []string `json:"apns_topics,omitempty"`
|
||||
}
|
||||
|
||||
// HandleTest handles POST /admin/notifications/push/apple/test.
|
||||
func (h *AdminApplePushHandler) HandleTest(w http.ResponseWriter, r *http.Request) {
|
||||
if h == nil || h.system == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Apple push delivery is not available")
|
||||
return
|
||||
}
|
||||
var req adminApplePushTestRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Invalid request body")
|
||||
return
|
||||
}
|
||||
result, err := h.system.SendApplePushTest(r.Context(), req.ProfileID, req.ServerDeviceID)
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, notifications.ErrPushDeliveryInvalid):
|
||||
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
||||
case errors.Is(err, notifications.ErrPushDeliveryNotFound):
|
||||
writeError(w, http.StatusNotFound, "not_found", "Apple push device not found")
|
||||
case errors.Is(err, notifications.ErrPushDeliveryUnavailable):
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Apple push delivery is not available")
|
||||
default:
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to send Apple push test")
|
||||
}
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, adminApplePushTestResponse{
|
||||
AttemptID: result.AttemptID,
|
||||
PushDeviceID: result.PushDeviceID,
|
||||
ServerDeviceID: result.ServerDeviceID,
|
||||
Outcome: result.Outcome,
|
||||
RelayRequestID: result.RelayRequestID,
|
||||
UpstreamStatus: result.UpstreamStatus,
|
||||
UpstreamReason: result.UpstreamReason,
|
||||
FailureMessage: result.FailureMessage,
|
||||
})
|
||||
}
|
||||
|
||||
// HandleRegisterRelay handles POST /admin/notifications/push/relay/register.
|
||||
func (h *AdminApplePushHandler) HandleRegisterRelay(w http.ResponseWriter, r *http.Request) {
|
||||
if h == nil || h.settings == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Settings store is not available")
|
||||
return
|
||||
}
|
||||
var req adminPushRelayRegisterRequest
|
||||
dec := json.NewDecoder(r.Body)
|
||||
dec.DisallowUnknownFields()
|
||||
if err := dec.Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Invalid request body")
|
||||
return
|
||||
}
|
||||
relayURL, err := normalizePushRelayURL(req.RelayURL)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
existingDeploymentID, err := h.settings.Get(r.Context(), notifications.SettingPushRelayDeploymentID)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "settings_error", "Failed to load relay deployment id")
|
||||
return
|
||||
}
|
||||
relayResp, err := h.registerWithRelay(r, relayURL, strings.TrimSpace(existingDeploymentID))
|
||||
if err != nil {
|
||||
status, code, message := mapRelayRegistrationError(err)
|
||||
writeError(w, status, code, message)
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(relayResp.DeploymentID) == "" || strings.TrimSpace(relayResp.APIKey) == "" {
|
||||
writeError(w, http.StatusBadGateway, "relay_bad_response", "Push relay returned an incomplete registration response")
|
||||
return
|
||||
}
|
||||
|
||||
if err := h.settings.Set(r.Context(), notifications.SettingPushRelayURL, relayURL); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "settings_error", "Failed to save push relay URL")
|
||||
return
|
||||
}
|
||||
if err := h.settings.Set(r.Context(), notifications.SettingPushRelayDeploymentID, relayResp.DeploymentID); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "settings_error", "Failed to save push relay deployment id")
|
||||
return
|
||||
}
|
||||
if err := h.settings.Set(r.Context(), notifications.SettingPushRelayAPIKey, relayResp.APIKey); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "settings_error", "Failed to save push relay API key")
|
||||
return
|
||||
}
|
||||
if h.system != nil && h.system.Settings != nil {
|
||||
h.system.Settings.Invalidate(
|
||||
notifications.SettingPushRelayURL,
|
||||
notifications.SettingPushRelayDeploymentID,
|
||||
notifications.SettingPushRelayAPIKey,
|
||||
)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, adminPushRelayRegisterResponse{
|
||||
RelayURL: relayURL,
|
||||
DeploymentID: relayResp.DeploymentID,
|
||||
KeyPrefix: relayResp.KeyPrefix,
|
||||
APIKeyConfigured: true,
|
||||
RelayRequestID: relayResp.RequestID,
|
||||
APNsTopics: relayResp.APNsTopics,
|
||||
})
|
||||
}
|
||||
|
||||
func (h *AdminApplePushHandler) registerWithRelay(r *http.Request, relayURL, deploymentID string) (*pushRelayRegisterResponse, error) {
|
||||
body, err := json.Marshal(pushRelayRegisterRequest{
|
||||
DeploymentID: deploymentID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
httpReq, err := http.NewRequestWithContext(r.Context(), http.MethodPost, relayURL+"/v1/deployments/register", bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
httpReq.Header.Set("Content-Type", "application/json")
|
||||
httpReq.Header.Set("User-Agent", "Silo-Server/PushRelayRegistration")
|
||||
|
||||
resp, err := h.client.Do(httpReq)
|
||||
if err != nil {
|
||||
return nil, relayRegistrationError{status: http.StatusBadGateway, code: "relay_unreachable", message: "Push relay could not be reached"}
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
|
||||
data, _ := io.ReadAll(io.LimitReader(resp.Body, 16<<10))
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
var parsed pushRelayNestedError
|
||||
_ = json.Unmarshal(data, &parsed)
|
||||
code := parsed.Error.Code
|
||||
if code == "" {
|
||||
code = fmt.Sprintf("relay_http_%d", resp.StatusCode)
|
||||
}
|
||||
message := parsed.Error.Message
|
||||
if message == "" {
|
||||
message = http.StatusText(resp.StatusCode)
|
||||
}
|
||||
return nil, relayRegistrationError{status: resp.StatusCode, code: code, message: message}
|
||||
}
|
||||
var parsed pushRelayRegisterResponse
|
||||
if err := json.Unmarshal(data, &parsed); err != nil {
|
||||
return nil, relayRegistrationError{status: http.StatusBadGateway, code: "relay_bad_response", message: "Push relay returned invalid JSON"}
|
||||
}
|
||||
return &parsed, nil
|
||||
}
|
||||
|
||||
type relayRegistrationError struct {
|
||||
status int
|
||||
code string
|
||||
message string
|
||||
}
|
||||
|
||||
func (e relayRegistrationError) Error() string {
|
||||
return e.code
|
||||
}
|
||||
|
||||
func mapRelayRegistrationError(err error) (int, string, string) {
|
||||
var relayErr relayRegistrationError
|
||||
if !errors.As(err, &relayErr) {
|
||||
return http.StatusInternalServerError, "internal_error", "Failed to register push relay"
|
||||
}
|
||||
switch relayErr.status {
|
||||
case http.StatusForbidden:
|
||||
return http.StatusUnprocessableEntity, "relay_deployment_rejected", "Push relay rejected this deployment"
|
||||
case http.StatusTooManyRequests:
|
||||
return http.StatusTooManyRequests, "relay_rate_limited", relayErr.message
|
||||
case http.StatusServiceUnavailable:
|
||||
return http.StatusServiceUnavailable, "relay_unavailable", relayErr.message
|
||||
default:
|
||||
if relayErr.status >= 500 {
|
||||
return http.StatusBadGateway, "relay_error", relayErr.message
|
||||
}
|
||||
return http.StatusBadGateway, relayErr.code, relayErr.message
|
||||
}
|
||||
}
|
||||
|
||||
// normalizePushRelayURLValue trims a relay URL and enforces https with a
|
||||
// host. Empty input stays empty so callers choose their own default.
|
||||
func normalizePushRelayURLValue(raw string) (string, error) {
|
||||
value := strings.TrimRight(strings.TrimSpace(raw), "/")
|
||||
if value == "" {
|
||||
return "", nil
|
||||
}
|
||||
parsed, err := url.Parse(value)
|
||||
if err != nil || parsed.Scheme != "https" || parsed.Host == "" {
|
||||
return "", errors.New("relay_url must be an https URL")
|
||||
}
|
||||
return value, nil
|
||||
}
|
||||
|
||||
func normalizePushRelayURL(raw string) (string, error) {
|
||||
value, err := normalizePushRelayURLValue(raw)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if value == "" {
|
||||
return notifications.DefaultPushRelayURL, nil
|
||||
}
|
||||
return value, nil
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/notifications"
|
||||
)
|
||||
|
||||
func TestAdminApplePushHandlerUnavailableWithoutSystem(t *testing.T) {
|
||||
h := NewAdminApplePushHandler(nil, nil)
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleTest(rec, httptest.NewRequest(http.MethodPost, "/admin/notifications/push/apple/test", strings.NewReader(`{}`)))
|
||||
if rec.Code != http.StatusServiceUnavailable {
|
||||
t.Fatalf("HandleTest without system = %d, want 503", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminApplePushHandlerRejectsInvalidJSON(t *testing.T) {
|
||||
h := NewAdminApplePushHandler(¬ifications.System{}, nil)
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleTest(rec, httptest.NewRequest(http.MethodPost, "/admin/notifications/push/apple/test", strings.NewReader(`{`)))
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("HandleTest invalid JSON = %d, want 400", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminApplePushHandlerRegistersRelayAndStoresKey(t *testing.T) {
|
||||
settings := &fakeServerSettingsStore{values: map[string]string{
|
||||
notifications.SettingPushRelayDeploymentID: "01EXISTING",
|
||||
}}
|
||||
var relayReq map[string]string
|
||||
relay := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/v1/deployments/register" {
|
||||
t.Fatalf("relay path = %s", r.URL.Path)
|
||||
}
|
||||
if err := json.NewDecoder(r.Body).Decode(&relayReq); err != nil {
|
||||
t.Fatalf("decode relay request: %v", err)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, pushRelayRegisterResponse{
|
||||
RequestID: "relay-request",
|
||||
DeploymentID: "01RETURNED",
|
||||
APIKey: "rk_live_raw-key",
|
||||
KeyPrefix: "rk_live_raw",
|
||||
APNsTopics: []string{"org.siloserver.silo"},
|
||||
})
|
||||
}))
|
||||
t.Cleanup(relay.Close)
|
||||
|
||||
h := NewAdminApplePushHandler(¬ifications.System{Settings: notifications.NewSettings(settings)}, settings)
|
||||
h.client = relay.Client()
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleRegisterRelay(rec, httptest.NewRequest(http.MethodPost, "/admin/notifications/push/relay/register", strings.NewReader(`{
|
||||
"relay_url":"`+relay.URL+`"
|
||||
}`)))
|
||||
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("http relay URL status = %d, want 400 because HTTPS is required", rec.Code)
|
||||
}
|
||||
|
||||
httpsRelay := httptest.NewTLSServer(relay.Config.Handler)
|
||||
t.Cleanup(httpsRelay.Close)
|
||||
h.client = httpsRelay.Client()
|
||||
rec = httptest.NewRecorder()
|
||||
h.HandleRegisterRelay(rec, httptest.NewRequest(http.MethodPost, "/admin/notifications/push/relay/register", strings.NewReader(`{
|
||||
"relay_url":"`+httpsRelay.URL+`"
|
||||
}`)))
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d (%s), want 200", rec.Code, rec.Body.String())
|
||||
}
|
||||
if relayReq["deployment_id"] != "01EXISTING" || len(relayReq) != 1 {
|
||||
t.Fatalf("relay request = %+v", relayReq)
|
||||
}
|
||||
if settings.values[notifications.SettingPushRelayURL] != httpsRelay.URL {
|
||||
t.Fatalf("stored relay URL = %q", settings.values[notifications.SettingPushRelayURL])
|
||||
}
|
||||
if settings.values[notifications.SettingPushRelayDeploymentID] != "01RETURNED" {
|
||||
t.Fatalf("stored deployment id = %q", settings.values[notifications.SettingPushRelayDeploymentID])
|
||||
}
|
||||
if settings.values[notifications.SettingPushRelayAPIKey] != "rk_live_raw-key" {
|
||||
t.Fatalf("stored api key = %q", settings.values[notifications.SettingPushRelayAPIKey])
|
||||
}
|
||||
|
||||
var body map[string]any
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
|
||||
t.Fatalf("decode response: %v", err)
|
||||
}
|
||||
if _, ok := body["api_key"]; ok {
|
||||
t.Fatal("response leaked raw api_key")
|
||||
}
|
||||
if body["api_key_configured"] != true || body["deployment_id"] != "01RETURNED" {
|
||||
t.Fatalf("response = %+v", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminApplePushHandlerMapsRelayRateLimit(t *testing.T) {
|
||||
settings := &fakeServerSettingsStore{values: map[string]string{}}
|
||||
relay := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
writeJSON(w, http.StatusTooManyRequests, map[string]any{
|
||||
"error": map[string]string{"code": "rate_limited", "message": "too many deployment registrations from this network"},
|
||||
})
|
||||
}))
|
||||
t.Cleanup(relay.Close)
|
||||
|
||||
h := NewAdminApplePushHandler(¬ifications.System{}, settings)
|
||||
h.client = relay.Client()
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleRegisterRelay(rec, httptest.NewRequest(http.MethodPost, "/admin/notifications/push/relay/register", strings.NewReader(`{
|
||||
"relay_url":"`+relay.URL+`"
|
||||
}`)))
|
||||
|
||||
if rec.Code != http.StatusTooManyRequests {
|
||||
t.Fatalf("status = %d (%s), want 429", rec.Code, rec.Body.String())
|
||||
}
|
||||
if settings.values[notifications.SettingPushRelayAPIKey] != "" {
|
||||
t.Fatal("api key was stored after relay rejected registration")
|
||||
}
|
||||
}
|
||||
@@ -161,7 +161,11 @@ func (h *AdminMatchHandler) HandleApplyItemMatch(w http.ResponseWriter, r *http.
|
||||
return
|
||||
}
|
||||
|
||||
if len(req.ProviderIDs) == 0 {
|
||||
// Normalize keys and values the same way the search endpoint does so a
|
||||
// caller-supplied key like "TMDB" or " tmdb " cannot bypass downstream
|
||||
// provider-id handling (e.g. stale-ID suppression in metadata.Process).
|
||||
providerIDs := normalizeMatchProviderIDs(req.ProviderIDs)
|
||||
if len(providerIDs) == 0 {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "At least one provider ID is required")
|
||||
return
|
||||
}
|
||||
@@ -191,7 +195,7 @@ func (h *AdminMatchHandler) HandleApplyItemMatch(w http.ResponseWriter, r *http.
|
||||
|
||||
result, err := h.metadata.Process(r.Context(), metadata.ProcessRequest{
|
||||
ContentID: contentID,
|
||||
ProviderIDs: req.ProviderIDs,
|
||||
ProviderIDs: providerIDs,
|
||||
FolderID: folderIDStr,
|
||||
Mode: metadata.ModeIdentify,
|
||||
})
|
||||
|
||||
@@ -326,6 +326,67 @@ func TestAdminMatchApply_RequiresProviderIDs(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminMatchApply_NormalizesProviderIDKeys(t *testing.T) {
|
||||
items := &fakeMatchItemLookup{
|
||||
items: map[string]*models.MediaItem{
|
||||
"item-5": {ContentID: "item-5", Title: "Test", Type: "movie"},
|
||||
},
|
||||
}
|
||||
metaSvc := &fakeMatchMetadataService{
|
||||
processResult: &metadata.ProcessResult{ContentID: "item-5", Updated: true},
|
||||
}
|
||||
h := NewAdminMatchHandler(items, nil, metaSvc)
|
||||
router := buildMatchRouter(h)
|
||||
|
||||
body, _ := json.Marshal(matchApplyRequest{
|
||||
ProviderIDs: map[string]string{
|
||||
" TMDB ": " 157336 ",
|
||||
"blank": " ",
|
||||
},
|
||||
})
|
||||
req := httptest.NewRequest(http.MethodPost, "/admin/items/item-5/match/apply", bytes.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
router.ServeHTTP(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200, got %d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if got := metaSvc.lastProcess.ProviderIDs["tmdb"]; got != "157336" {
|
||||
t.Errorf("Process provider_ids[tmdb] = %q, want 157336 (keys/values must be normalized)", got)
|
||||
}
|
||||
if _, ok := metaSvc.lastProcess.ProviderIDs[" TMDB "]; ok {
|
||||
t.Errorf("Process received raw key %q, want normalized keys only: %v", " TMDB ", metaSvc.lastProcess.ProviderIDs)
|
||||
}
|
||||
if _, ok := metaSvc.lastProcess.ProviderIDs["blank"]; ok {
|
||||
t.Errorf("blank provider IDs should be dropped, got %v", metaSvc.lastProcess.ProviderIDs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminMatchApply_RejectsAllBlankProviderIDs(t *testing.T) {
|
||||
items := &fakeMatchItemLookup{
|
||||
items: map[string]*models.MediaItem{
|
||||
"item-6": {ContentID: "item-6", Title: "Test", Type: "movie"},
|
||||
},
|
||||
}
|
||||
h := NewAdminMatchHandler(items, nil, &fakeMatchMetadataService{})
|
||||
router := buildMatchRouter(h)
|
||||
|
||||
body, _ := json.Marshal(matchApplyRequest{
|
||||
ProviderIDs: map[string]string{"tmdb": " ", " ": "123"},
|
||||
})
|
||||
req := httptest.NewRequest(http.MethodPost, "/admin/items/item-6/match/apply", bytes.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
router.ServeHTTP(rec, req)
|
||||
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("expected 400 for all-blank provider_ids, got %d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminMatchApply_ItemNotFound(t *testing.T) {
|
||||
items := &fakeMatchItemLookup{items: map[string]*models.MediaItem{}}
|
||||
h := NewAdminMatchHandler(items, nil, &fakeMatchMetadataService{})
|
||||
|
||||
@@ -179,7 +179,7 @@ func (h *AdminPlaybackControlHandler) handleSessionCommand(w http.ResponseWriter
|
||||
|
||||
fallback := func() {
|
||||
h.playback.forgetRealtimeCommand(commandID)
|
||||
_ = h.playback.stopPlaybackSessionByID(context.Background(), sessionID)
|
||||
_ = h.playback.stopPlaybackSessionByID(context.Background(), sessionID, true)
|
||||
}
|
||||
|
||||
h.playback.rememberRealtimeCommand(commandID, sessionID, name)
|
||||
|
||||
@@ -41,6 +41,8 @@ type serverChannelResponse struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
NotifyNewMovies bool `json:"notify_new_movies"`
|
||||
NotifyNewEpisodes bool `json:"notify_new_episodes"`
|
||||
NotifyNewAudiobooks bool `json:"notify_new_audiobooks"`
|
||||
NotifyNewEbooks bool `json:"notify_new_ebooks"`
|
||||
NotifyRequestSubmitted bool `json:"notify_request_submitted"`
|
||||
NotifyRequestApproved bool `json:"notify_request_approved"`
|
||||
NotifyRequestDeclined bool `json:"notify_request_declined"`
|
||||
@@ -65,6 +67,8 @@ func serverChannelToResponse(ch notifications.ServerChannel) serverChannelRespon
|
||||
Enabled: ch.Enabled,
|
||||
NotifyNewMovies: ch.NotifyNewMovies,
|
||||
NotifyNewEpisodes: ch.NotifyNewEpisodes,
|
||||
NotifyNewAudiobooks: ch.NotifyNewAudiobooks,
|
||||
NotifyNewEbooks: ch.NotifyNewEbooks,
|
||||
NotifyRequestSubmitted: ch.NotifyRequestSubmitted,
|
||||
NotifyRequestApproved: ch.NotifyRequestApproved,
|
||||
NotifyRequestDeclined: ch.NotifyRequestDeclined,
|
||||
@@ -86,6 +90,8 @@ type serverChannelRequest struct {
|
||||
Enabled *bool `json:"enabled"`
|
||||
NotifyNewMovies *bool `json:"notify_new_movies"`
|
||||
NotifyNewEpisodes *bool `json:"notify_new_episodes"`
|
||||
NotifyNewAudiobooks *bool `json:"notify_new_audiobooks"`
|
||||
NotifyNewEbooks *bool `json:"notify_new_ebooks"`
|
||||
NotifyRequestSubmitted *bool `json:"notify_request_submitted"`
|
||||
NotifyRequestApproved *bool `json:"notify_request_approved"`
|
||||
NotifyRequestDeclined *bool `json:"notify_request_declined"`
|
||||
@@ -100,6 +106,8 @@ func (r serverChannelRequest) toInput() notifications.ServerChannelInput {
|
||||
Enabled: r.Enabled,
|
||||
NotifyNewMovies: r.NotifyNewMovies,
|
||||
NotifyNewEpisodes: r.NotifyNewEpisodes,
|
||||
NotifyNewAudiobooks: r.NotifyNewAudiobooks,
|
||||
NotifyNewEbooks: r.NotifyNewEbooks,
|
||||
NotifyRequestSubmitted: r.NotifyRequestSubmitted,
|
||||
NotifyRequestApproved: r.NotifyRequestApproved,
|
||||
NotifyRequestDeclined: r.NotifyRequestDeclined,
|
||||
|
||||
@@ -0,0 +1,233 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/downloads"
|
||||
)
|
||||
|
||||
// subscriptionRequest is the JSON body for POST /downloads/subscriptions.
|
||||
type subscriptionRequest struct {
|
||||
SeriesID string `json:"series_id"`
|
||||
Mode string `json:"mode"` // all | future | latest_season | specific_seasons
|
||||
SeasonNumbers []int `json:"season_numbers,omitempty"`
|
||||
DeleteWatched bool `json:"delete_watched,omitempty"`
|
||||
MaxStorageBytes int64 `json:"max_storage_bytes,omitempty"`
|
||||
}
|
||||
|
||||
// patchSubscriptionRequest is the JSON body for PATCH /downloads/subscriptions/{id}.
|
||||
// Pointer fields distinguish "absent" from a zero value so a partial update only
|
||||
// touches the fields the client sent.
|
||||
type patchSubscriptionRequest struct {
|
||||
Mode *string `json:"mode,omitempty"`
|
||||
SeasonNumbers *[]int `json:"season_numbers,omitempty"`
|
||||
DeleteWatched *bool `json:"delete_watched,omitempty"`
|
||||
MaxStorageBytes *int64 `json:"max_storage_bytes,omitempty"`
|
||||
Active *bool `json:"active,omitempty"`
|
||||
}
|
||||
|
||||
// subscriptionResponse is one subscription in API responses.
|
||||
type subscriptionResponse struct {
|
||||
ID string `json:"id"`
|
||||
SeriesID string `json:"series_id"`
|
||||
Mode string `json:"mode"`
|
||||
SeasonNumbers []int `json:"season_numbers,omitempty"`
|
||||
TargetSeason *int `json:"target_season,omitempty"`
|
||||
DeleteWatched bool `json:"delete_watched"`
|
||||
MaxStorageBytes int64 `json:"max_storage_bytes"`
|
||||
Active bool `json:"active"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
UpdatedAt string `json:"updated_at"`
|
||||
}
|
||||
|
||||
type subscriptionsListResponse struct {
|
||||
Subscriptions []subscriptionResponse `json:"subscriptions"`
|
||||
}
|
||||
|
||||
// subscriptionResultResponse wraps a created/updated subscription plus the count
|
||||
// of in-scope episodes registered as managed downloads.
|
||||
type subscriptionResultResponse struct {
|
||||
Subscription subscriptionResponse `json:"subscription"`
|
||||
Registered int `json:"registered"`
|
||||
}
|
||||
|
||||
// subscriptionSyncResponse reports how many newly in-scope episodes a sync
|
||||
// registered across the calling device's monitors.
|
||||
type subscriptionSyncResponse struct {
|
||||
Registered int `json:"registered"`
|
||||
}
|
||||
|
||||
func toSubscriptionResponse(s *downloads.Subscription) subscriptionResponse {
|
||||
return subscriptionResponse{
|
||||
ID: s.ID,
|
||||
SeriesID: s.SeriesID,
|
||||
Mode: s.Mode,
|
||||
SeasonNumbers: s.SeasonNumbers,
|
||||
TargetSeason: s.TargetSeason,
|
||||
DeleteWatched: s.DeleteWatched,
|
||||
MaxStorageBytes: s.MaxStorageBytes,
|
||||
Active: s.Active,
|
||||
CreatedAt: s.CreatedAt.Format("2006-01-02T15:04:05Z07:00"),
|
||||
UpdatedAt: s.UpdatedAt.Format("2006-01-02T15:04:05Z07:00"),
|
||||
}
|
||||
}
|
||||
|
||||
func toSubscriptionResultResponse(result *downloads.SubscriptionResult) subscriptionResultResponse {
|
||||
return subscriptionResultResponse{
|
||||
Subscription: toSubscriptionResponse(result.Subscription),
|
||||
Registered: result.Registered,
|
||||
}
|
||||
}
|
||||
|
||||
// HandleCreateSubscription handles POST /downloads/subscriptions (managed-only).
|
||||
func (h *DownloadHandler) HandleCreateSubscription(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, deviceName, devicePlatform, ok := h.requireManaged(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req subscriptionRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Invalid request body")
|
||||
return
|
||||
}
|
||||
if req.SeriesID == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "series_id is required")
|
||||
return
|
||||
}
|
||||
|
||||
result, err := h.svc.CreateSubscription(r.Context(), userID, downloads.SubscriptionRequest{
|
||||
SeriesID: req.SeriesID,
|
||||
Mode: req.Mode,
|
||||
SeasonNumbers: req.SeasonNumbers,
|
||||
DeleteWatched: req.DeleteWatched,
|
||||
MaxStorageBytes: req.MaxStorageBytes,
|
||||
ProfileID: profileID,
|
||||
DeviceID: deviceID,
|
||||
DeviceName: deviceName,
|
||||
DevicePlatform: devicePlatform,
|
||||
}, requestAccessFilter(r))
|
||||
if err != nil {
|
||||
h.writeSubscriptionError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusAccepted, toSubscriptionResultResponse(result))
|
||||
}
|
||||
|
||||
// HandleListSubscriptions handles GET /downloads/subscriptions (managed-only).
|
||||
func (h *DownloadHandler) HandleListSubscriptions(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, _, _, ok := h.requireManaged(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
subs, err := h.svc.ListSubscriptions(r.Context(), userID, profileID, deviceID)
|
||||
if err != nil {
|
||||
h.writeSubscriptionError(w, err)
|
||||
return
|
||||
}
|
||||
responses := make([]subscriptionResponse, 0, len(subs))
|
||||
for _, s := range subs {
|
||||
responses = append(responses, toSubscriptionResponse(s))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, subscriptionsListResponse{Subscriptions: responses})
|
||||
}
|
||||
|
||||
// HandleGetSubscription handles GET /downloads/subscriptions/{id} (managed-only).
|
||||
func (h *DownloadHandler) HandleGetSubscription(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, _, _, ok := h.requireManaged(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
sub, err := h.svc.GetSubscription(r.Context(), userID, profileID, deviceID, chi.URLParam(r, "id"))
|
||||
if err != nil {
|
||||
h.writeSubscriptionError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, toSubscriptionResponse(sub))
|
||||
}
|
||||
|
||||
// HandlePatchSubscription handles PATCH /downloads/subscriptions/{id} (managed-only).
|
||||
func (h *DownloadHandler) HandlePatchSubscription(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, _, _, ok := h.requireManaged(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req patchSubscriptionRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Invalid request body")
|
||||
return
|
||||
}
|
||||
result, err := h.svc.UpdateSubscription(r.Context(), userID, profileID, deviceID, chi.URLParam(r, "id"), downloads.SubscriptionPatch{
|
||||
Mode: req.Mode,
|
||||
SeasonNumbers: req.SeasonNumbers,
|
||||
DeleteWatched: req.DeleteWatched,
|
||||
MaxStorageBytes: req.MaxStorageBytes,
|
||||
Active: req.Active,
|
||||
}, requestAccessFilter(r))
|
||||
if err != nil {
|
||||
h.writeSubscriptionError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, toSubscriptionResultResponse(result))
|
||||
}
|
||||
|
||||
// HandleDeleteSubscription handles DELETE /downloads/subscriptions/{id} (managed-only).
|
||||
func (h *DownloadHandler) HandleDeleteSubscription(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, _, _, ok := h.requireManaged(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if err := h.svc.DeleteSubscription(r.Context(), userID, profileID, deviceID, chi.URLParam(r, "id")); err != nil {
|
||||
h.writeSubscriptionError(w, err)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// HandleSyncSubscriptions handles POST /downloads/subscriptions/sync (managed-only):
|
||||
// the client triggers it on open / background refresh to register newly available
|
||||
// episodes for its monitors, then pulls the files on its own schedule.
|
||||
func (h *DownloadHandler) HandleSyncSubscriptions(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, _, _, ok := h.requireManaged(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
registered, err := h.svc.SyncSubscriptions(r.Context(), userID, profileID, deviceID, requestAccessFilter(r))
|
||||
if err != nil {
|
||||
h.writeSubscriptionError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, subscriptionSyncResponse{Registered: registered})
|
||||
}
|
||||
|
||||
// writeSubscriptionError maps series-monitoring errors to HTTP responses.
|
||||
func (h *DownloadHandler) writeSubscriptionError(w http.ResponseWriter, err error) {
|
||||
switch {
|
||||
case errors.Is(err, downloads.ErrSubscriptionNotFound):
|
||||
writeError(w, http.StatusNotFound, "not_found", "Subscription not found")
|
||||
case errors.Is(err, downloads.ErrSubscriptionsUnavailable):
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Series monitoring is not available")
|
||||
case errors.Is(err, downloads.ErrInvalidSubscriptionMode):
|
||||
writeError(w, http.StatusBadRequest, "invalid_mode", "Unknown monitor mode")
|
||||
case errors.Is(err, downloads.ErrSeasonsRequired):
|
||||
writeError(w, http.StatusBadRequest, "seasons_required", "season_numbers is required for specific_seasons mode")
|
||||
case errors.Is(err, downloads.ErrInvalidSeasonNumbers):
|
||||
writeError(w, http.StatusBadRequest, "invalid_season_numbers", "season_numbers values must be between 0 and 9999")
|
||||
case errors.Is(err, downloads.ErrNotSeries):
|
||||
writeError(w, http.StatusBadRequest, "not_series", "content_id is not a series")
|
||||
case errors.Is(err, downloads.ErrProfileRequired):
|
||||
writeError(w, http.StatusBadRequest, "profile_required", "A profile and device are required for series monitoring")
|
||||
case errors.Is(err, downloads.ErrFeatureDisabled), errors.Is(err, downloads.ErrDownloadNotAllowed):
|
||||
writeError(w, http.StatusForbidden, "forbidden", "You are not allowed to download")
|
||||
case errors.Is(err, catalog.ErrItemNotFound):
|
||||
writeError(w, http.StatusNotFound, "not_found", "Series not found")
|
||||
default:
|
||||
slog.Error("download subscription operation failed", "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to process subscription")
|
||||
}
|
||||
}
|
||||
@@ -12,17 +12,33 @@ import (
|
||||
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/download"
|
||||
"github.com/Silo-Server/silo-server/internal/downloads"
|
||||
"github.com/Silo-Server/silo-server/internal/playback"
|
||||
)
|
||||
|
||||
// DownloadService is the interface that the download handler depends on.
|
||||
// DownloadService is the interface that the download handler depends on. A
|
||||
// non-empty deviceID (from the X-Silo-Device-Id header) selects the managed
|
||||
// device-library lifecycle; empty is the ephemeral/account-level path.
|
||||
type DownloadService interface {
|
||||
CreateQueued(ctx context.Context, userID int, req download.CreateRequest, filter catalog.AccessFilter) (*download.Download, error)
|
||||
CreateQueuedBatch(ctx context.Context, userID int, seriesContentID string, filter catalog.AccessFilter) ([]*download.Download, string, error)
|
||||
ServeDirect(ctx context.Context, w http.ResponseWriter, r *http.Request, userID int, fileID int, filter catalog.AccessFilter) error
|
||||
ServeFile(ctx context.Context, w http.ResponseWriter, r *http.Request, userID int, downloadID string) error
|
||||
List(ctx context.Context, userID int) ([]*download.Download, error)
|
||||
Cancel(ctx context.Context, userID int, downloadID string) error
|
||||
Capability(ctx context.Context, userID int) (downloads.Capability, error)
|
||||
Create(ctx context.Context, userID int, req downloads.CreateRequest, filter catalog.AccessFilter) (*downloads.Download, error)
|
||||
CreateSeries(ctx context.Context, userID int, req downloads.CreateRequest, filter catalog.AccessFilter) ([]*downloads.Download, string, []downloads.SkippedDownload, error)
|
||||
CreateSeason(ctx context.Context, userID int, req downloads.CreateRequest, seasonNumber int, filter catalog.AccessFilter) ([]*downloads.Download, string, []downloads.SkippedDownload, error)
|
||||
CreateSubscription(ctx context.Context, userID int, req downloads.SubscriptionRequest, filter catalog.AccessFilter) (*downloads.SubscriptionResult, error)
|
||||
ListSubscriptions(ctx context.Context, userID int, profileID, deviceID string) ([]*downloads.Subscription, error)
|
||||
GetSubscription(ctx context.Context, userID int, profileID, deviceID, id string) (*downloads.Subscription, error)
|
||||
UpdateSubscription(ctx context.Context, userID int, profileID, deviceID, id string, patch downloads.SubscriptionPatch, filter catalog.AccessFilter) (*downloads.SubscriptionResult, error)
|
||||
DeleteSubscription(ctx context.Context, userID int, profileID, deviceID, id string) error
|
||||
SyncSubscriptions(ctx context.Context, userID int, profileID, deviceID string, filter catalog.AccessFilter) (int, error)
|
||||
ServeDirect(ctx context.Context, w http.ResponseWriter, r *http.Request, userID, fileID int, format string, filter catalog.AccessFilter) error
|
||||
ServeFile(ctx context.Context, w http.ResponseWriter, r *http.Request, userID int, profileID, deviceID, downloadID string, filter catalog.AccessFilter) error
|
||||
List(ctx context.Context, userID int, profileID, deviceID string) ([]*downloads.Download, error)
|
||||
Delete(ctx context.Context, userID int, profileID, deviceID, downloadID string) error
|
||||
PatchStatus(ctx context.Context, userID int, profileID, deviceID, downloadID, status string) error
|
||||
BuildManifest(ctx context.Context, userID int, profileID, deviceID, downloadID string, filter catalog.AccessFilter) (*downloads.OfflineManifest, error)
|
||||
BuildBatchManifests(ctx context.Context, userID int, profileID, deviceID, batchID string, filter catalog.AccessFilter) ([]*downloads.OfflineManifest, []downloads.SkippedManifest, error)
|
||||
ServeArtwork(ctx context.Context, w http.ResponseWriter, r *http.Request, userID int, profileID, deviceID, downloadID, kind string, filter catalog.AccessFilter) error
|
||||
ServeSubtitle(ctx context.Context, w http.ResponseWriter, r *http.Request, userID int, profileID, deviceID, downloadID, ref string, filter catalog.AccessFilter) error
|
||||
}
|
||||
|
||||
// DownloadHandler handles download endpoints.
|
||||
@@ -37,44 +53,96 @@ func NewDownloadHandler(svc DownloadService) *DownloadHandler {
|
||||
|
||||
// downloadRequest represents the JSON body for POST /downloads.
|
||||
type downloadRequest struct {
|
||||
ContentID string `json:"content_id"`
|
||||
EpisodeID string `json:"episode_id,omitempty"`
|
||||
FileID int `json:"file_id,omitempty"`
|
||||
Series bool `json:"series,omitempty"` // if true, downloads all episodes
|
||||
ContentID string `json:"content_id"`
|
||||
EpisodeID string `json:"episode_id,omitempty"`
|
||||
FileID int `json:"file_id,omitempty"`
|
||||
Quality string `json:"quality,omitempty"` // original (default) | 20mbps | 10mbps | 5mbps | 2mbps | 1mbps
|
||||
Series bool `json:"series,omitempty"` // if true, downloads all episodes
|
||||
Season *int `json:"season_number,omitempty"` // with series=true, downloads only this season (0 = Specials)
|
||||
Caps *downloadCaps `json:"caps,omitempty"` // device decode capability (original fallback / transcode target)
|
||||
}
|
||||
|
||||
// downloadCaps mirrors playback.ClientCapabilities for the request body.
|
||||
type downloadCaps struct {
|
||||
CodecsVideo []string `json:"codecs_video,omitempty"`
|
||||
CodecsAudio []string `json:"codecs_audio,omitempty"`
|
||||
AudioPassthroughCodecs []string `json:"audio_passthrough_codecs,omitempty"`
|
||||
Containers []string `json:"containers,omitempty"`
|
||||
MaxResolution string `json:"max_resolution,omitempty"`
|
||||
HDR bool `json:"hdr,omitempty"`
|
||||
}
|
||||
|
||||
// patchDownloadRequest is the JSON body for PATCH /downloads/{id}.
|
||||
type patchDownloadRequest struct {
|
||||
Status string `json:"status"`
|
||||
}
|
||||
|
||||
// downloadResponse represents a download entry in API responses.
|
||||
type downloadResponse struct {
|
||||
ID string `json:"id"`
|
||||
ContentID string `json:"content_id"`
|
||||
EpisodeID string `json:"episode_id,omitempty"`
|
||||
BatchID string `json:"batch_id,omitempty"`
|
||||
MediaFileID int `json:"media_file_id"`
|
||||
FileSize int64 `json:"file_size"`
|
||||
BytesSent int64 `json:"bytes_sent"`
|
||||
Kind string `json:"kind"`
|
||||
Status string `json:"status"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
CompletedAt *string `json:"completed_at,omitempty"`
|
||||
ID string `json:"id"`
|
||||
ContentID string `json:"content_id"`
|
||||
EpisodeID string `json:"episode_id,omitempty"`
|
||||
BatchID string `json:"batch_id,omitempty"`
|
||||
DeviceID string `json:"device_id,omitempty"`
|
||||
MediaFileID int `json:"media_file_id"`
|
||||
FileSize int64 `json:"file_size"`
|
||||
BytesSent int64 `json:"bytes_sent"`
|
||||
Kind string `json:"kind"`
|
||||
Status string `json:"status"`
|
||||
Quality string `json:"quality"`
|
||||
EffectiveQuality string `json:"effective_quality"`
|
||||
DeliveryFormat string `json:"delivery_format"`
|
||||
TargetBitrateKbps int `json:"target_bitrate_kbps"`
|
||||
Revision int `json:"revision"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
CompletedAt *string `json:"completed_at,omitempty"`
|
||||
}
|
||||
|
||||
// downloadsListResponse wraps the downloads list for JSON serialization.
|
||||
type downloadsListResponse struct {
|
||||
Downloads []downloadResponse `json:"downloads"`
|
||||
Downloads []downloadResponse `json:"downloads"`
|
||||
Skipped []downloads.SkippedDownload `json:"skipped,omitempty"`
|
||||
}
|
||||
|
||||
func toDownloadResponse(d *download.Download) downloadResponse {
|
||||
type batchManifestsResponse struct {
|
||||
Manifests []*downloads.OfflineManifest `json:"manifests"`
|
||||
// Skipped lists batch entries whose manifest could not be built (revoked,
|
||||
// deleted from the catalog, access-filtered); the rest of the batch is
|
||||
// still delivered.
|
||||
Skipped []downloads.SkippedManifest `json:"skipped,omitempty"`
|
||||
}
|
||||
|
||||
// downloadCapabilityResponse is the GET /downloads/capability payload clients
|
||||
// use for feature detection instead of introspecting admin settings.
|
||||
type downloadCapabilityResponse struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
DownloadAllowed bool `json:"download_allowed"`
|
||||
QualityPresets []string `json:"quality_presets"`
|
||||
TranscodeEnabled bool `json:"transcode_enabled"`
|
||||
TranscodeUserAllowed bool `json:"transcode_user_allowed"`
|
||||
SeasonDownload bool `json:"season_download"`
|
||||
SeriesMonitoring bool `json:"series_monitoring"`
|
||||
MonitoringModes []string `json:"monitoring_modes,omitempty"`
|
||||
}
|
||||
|
||||
func toDownloadResponse(d *downloads.Download) downloadResponse {
|
||||
resp := downloadResponse{
|
||||
ID: d.ID,
|
||||
ContentID: d.ContentID,
|
||||
EpisodeID: d.EpisodeID,
|
||||
BatchID: d.BatchID,
|
||||
MediaFileID: d.MediaFileID,
|
||||
FileSize: d.FileSize,
|
||||
BytesSent: d.BytesSent,
|
||||
Kind: d.Kind,
|
||||
Status: d.Status,
|
||||
CreatedAt: d.CreatedAt.Format("2006-01-02T15:04:05Z07:00"),
|
||||
ID: d.ID,
|
||||
ContentID: d.ContentID,
|
||||
EpisodeID: d.EpisodeID,
|
||||
BatchID: d.BatchID,
|
||||
DeviceID: d.DeviceID,
|
||||
MediaFileID: d.MediaFileID,
|
||||
FileSize: d.FileSize,
|
||||
BytesSent: d.BytesSent,
|
||||
Kind: d.Kind,
|
||||
Status: d.Status,
|
||||
Quality: d.Quality,
|
||||
EffectiveQuality: d.EffectiveQuality,
|
||||
DeliveryFormat: d.Format,
|
||||
TargetBitrateKbps: d.TargetBitrateKbps,
|
||||
Revision: d.Revision,
|
||||
CreatedAt: d.CreatedAt.Format("2006-01-02T15:04:05Z07:00"),
|
||||
}
|
||||
if d.CompletedAt != nil {
|
||||
s := d.CompletedAt.Format("2006-01-02T15:04:05Z07:00")
|
||||
@@ -83,14 +151,53 @@ func toDownloadResponse(d *download.Download) downloadResponse {
|
||||
return resp
|
||||
}
|
||||
|
||||
// HandleCreateDownload handles POST /downloads.
|
||||
// managedIdentity returns the (profileID, deviceID) the request is acting as.
|
||||
// deviceID comes ONLY from the X-Silo-Device-Id header (never the body/query);
|
||||
// profileID is resolved by the viewer-access middleware from X-Profile-Id.
|
||||
func managedIdentity(r *http.Request) (profileID, deviceID, deviceName, devicePlatform string) {
|
||||
device := deviceMetadataFromRequest(r)
|
||||
return apimw.GetProfileID(r.Context()), device.DeviceID, device.DeviceName, device.DevicePlatform
|
||||
}
|
||||
|
||||
// HandleCapability handles GET /downloads/capability.
|
||||
func (h *DownloadHandler) HandleCapability(w http.ResponseWriter, r *http.Request) {
|
||||
userID := apimw.GetUserID(r.Context())
|
||||
if userID == 0 {
|
||||
writeError(w, http.StatusUnauthorized, "unauthorized", "Authentication required")
|
||||
return
|
||||
}
|
||||
if h.svc == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Downloads not configured")
|
||||
return
|
||||
}
|
||||
|
||||
capInfo, err := h.svc.Capability(r.Context(), userID)
|
||||
if err != nil {
|
||||
slog.Error("failed to load download capability", "user_id", userID, "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to load download capability")
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, downloadCapabilityResponse{
|
||||
Enabled: capInfo.Enabled,
|
||||
DownloadAllowed: capInfo.DownloadAllowed,
|
||||
QualityPresets: capInfo.QualityPresets,
|
||||
TranscodeEnabled: capInfo.TranscodeEnabled,
|
||||
TranscodeUserAllowed: capInfo.TranscodeUserAllowed,
|
||||
SeasonDownload: capInfo.SeasonDownload,
|
||||
SeriesMonitoring: capInfo.SeriesMonitoring,
|
||||
MonitoringModes: capInfo.MonitoringModes,
|
||||
})
|
||||
}
|
||||
|
||||
// HandleCreateDownload handles POST /downloads. The X-Silo-Device-Id header
|
||||
// (if present) makes this a managed device entry; otherwise it is ephemeral.
|
||||
func (h *DownloadHandler) HandleCreateDownload(w http.ResponseWriter, r *http.Request) {
|
||||
userID := apimw.GetUserID(r.Context())
|
||||
if userID == 0 {
|
||||
writeError(w, http.StatusUnauthorized, "unauthorized", "Authentication required")
|
||||
return
|
||||
}
|
||||
|
||||
if h.svc == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Downloads not configured")
|
||||
return
|
||||
@@ -101,65 +208,102 @@ func (h *DownloadHandler) HandleCreateDownload(w http.ResponseWriter, r *http.Re
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Invalid request body")
|
||||
return
|
||||
}
|
||||
|
||||
if req.ContentID == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "content_id is required")
|
||||
return
|
||||
}
|
||||
|
||||
profileID, deviceID, deviceName, devicePlatform := managedIdentity(r)
|
||||
filter := requestAccessFilter(r)
|
||||
createReq := downloads.CreateRequest{
|
||||
ContentID: req.ContentID,
|
||||
EpisodeID: req.EpisodeID,
|
||||
FileID: req.FileID,
|
||||
Quality: req.Quality,
|
||||
ProfileID: profileID,
|
||||
DeviceID: deviceID,
|
||||
DeviceName: deviceName,
|
||||
DevicePlatform: devicePlatform,
|
||||
}
|
||||
if req.Caps != nil {
|
||||
createReq.Caps = playback.ClientCapabilities{
|
||||
CodecsVideo: req.Caps.CodecsVideo,
|
||||
CodecsAudio: req.Caps.CodecsAudio,
|
||||
AudioPassthroughCodecs: req.Caps.AudioPassthroughCodecs,
|
||||
Containers: req.Caps.Containers,
|
||||
MaxResolution: req.Caps.MaxResolution,
|
||||
HDR: req.Caps.HDR,
|
||||
}
|
||||
}
|
||||
|
||||
if req.Series {
|
||||
downloads, batchID, err := h.svc.CreateQueuedBatch(r.Context(), userID, req.ContentID, filter)
|
||||
var (
|
||||
list []*downloads.Download
|
||||
batchID string
|
||||
skipped []downloads.SkippedDownload
|
||||
err error
|
||||
)
|
||||
// Dispatch on presence, not value: season 0 is the Specials season.
|
||||
if req.Season != nil {
|
||||
if *req.Season < 0 {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "season_number must be >= 0")
|
||||
return
|
||||
}
|
||||
list, batchID, skipped, err = h.svc.CreateSeason(r.Context(), userID, createReq, *req.Season, filter)
|
||||
} else {
|
||||
list, batchID, skipped, err = h.svc.CreateSeries(r.Context(), userID, createReq, filter)
|
||||
}
|
||||
if err != nil {
|
||||
h.writeDownloadError(w, err)
|
||||
return
|
||||
}
|
||||
responses := make([]downloadResponse, 0, len(downloads))
|
||||
for _, d := range downloads {
|
||||
responses := make([]downloadResponse, 0, len(list))
|
||||
for _, d := range list {
|
||||
resp := toDownloadResponse(d)
|
||||
resp.BatchID = batchID
|
||||
if resp.BatchID == "" {
|
||||
resp.BatchID = batchID
|
||||
}
|
||||
responses = append(responses, resp)
|
||||
}
|
||||
writeJSON(w, http.StatusAccepted, downloadsListResponse{Downloads: responses})
|
||||
writeJSON(w, http.StatusAccepted, downloadsListResponse{Downloads: responses, Skipped: skipped})
|
||||
return
|
||||
}
|
||||
|
||||
dl, err := h.svc.CreateQueued(r.Context(), userID, download.CreateRequest{
|
||||
ContentID: req.ContentID,
|
||||
EpisodeID: req.EpisodeID,
|
||||
FileID: req.FileID,
|
||||
}, filter)
|
||||
dl, err := h.svc.Create(r.Context(), userID, createReq, filter)
|
||||
if err != nil {
|
||||
h.writeDownloadError(w, err)
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusAccepted, toDownloadResponse(dl))
|
||||
}
|
||||
|
||||
// HandleListDownloads handles GET /downloads.
|
||||
// HandleListDownloads handles GET /downloads. With a device header it returns
|
||||
// the calling device's managed entries; otherwise the user's ephemeral rows.
|
||||
func (h *DownloadHandler) HandleListDownloads(w http.ResponseWriter, r *http.Request) {
|
||||
userID := apimw.GetUserID(r.Context())
|
||||
if userID == 0 {
|
||||
writeError(w, http.StatusUnauthorized, "unauthorized", "Authentication required")
|
||||
return
|
||||
}
|
||||
|
||||
if h.svc == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Downloads not configured")
|
||||
return
|
||||
}
|
||||
|
||||
downloads, err := h.svc.List(r.Context(), userID)
|
||||
profileID, deviceID, _, _ := managedIdentity(r)
|
||||
list, err := h.svc.List(r.Context(), userID, profileID, deviceID)
|
||||
if err != nil {
|
||||
if errors.Is(err, downloads.ErrProfileRequired) {
|
||||
writeError(w, http.StatusBadRequest, "profile_required", "A profile is required for managed downloads")
|
||||
return
|
||||
}
|
||||
slog.Error("failed to list downloads", "user_id", userID, "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to list downloads")
|
||||
return
|
||||
}
|
||||
|
||||
responses := make([]downloadResponse, 0, len(downloads))
|
||||
for _, d := range downloads {
|
||||
responses := make([]downloadResponse, 0, len(list))
|
||||
for _, d := range list {
|
||||
responses = append(responses, toDownloadResponse(d))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, downloadsListResponse{Downloads: responses})
|
||||
@@ -172,28 +316,66 @@ func (h *DownloadHandler) HandleDeleteDownload(w http.ResponseWriter, r *http.Re
|
||||
writeError(w, http.StatusUnauthorized, "unauthorized", "Authentication required")
|
||||
return
|
||||
}
|
||||
id := chi.URLParam(r, "id")
|
||||
if id == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Download ID is required")
|
||||
return
|
||||
}
|
||||
if h.svc == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Downloads not configured")
|
||||
return
|
||||
}
|
||||
|
||||
profileID, deviceID, _, _ := managedIdentity(r)
|
||||
if err := h.svc.Delete(r.Context(), userID, profileID, deviceID, id); err != nil {
|
||||
if errors.Is(err, downloads.ErrNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Download not found")
|
||||
return
|
||||
}
|
||||
if errors.Is(err, downloads.ErrProfileRequired) {
|
||||
writeError(w, http.StatusBadRequest, "profile_required", "A profile is required for managed downloads")
|
||||
return
|
||||
}
|
||||
slog.Error("failed to delete download", "download_id", id, "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to delete download")
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// HandlePatchDownload handles PATCH /downloads/{id} — a managed-only endpoint
|
||||
// where a client confirms local state (downloading|completed).
|
||||
func (h *DownloadHandler) HandlePatchDownload(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, _, _, ok := h.requireManaged(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
id := chi.URLParam(r, "id")
|
||||
if id == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Download ID is required")
|
||||
return
|
||||
}
|
||||
|
||||
if h.svc == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Downloads not configured")
|
||||
var req patchDownloadRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Invalid request body")
|
||||
return
|
||||
}
|
||||
|
||||
if err := h.svc.Cancel(r.Context(), userID, id); err != nil {
|
||||
if errors.Is(err, download.ErrNotFound) {
|
||||
if err := h.svc.PatchStatus(r.Context(), userID, profileID, deviceID, id, req.Status); err != nil {
|
||||
switch {
|
||||
case errors.Is(err, downloads.ErrNotFound):
|
||||
writeError(w, http.StatusNotFound, "not_found", "Download not found")
|
||||
return
|
||||
case errors.Is(err, downloads.ErrInvalidStatus):
|
||||
writeError(w, http.StatusBadRequest, "invalid_status", "status must be downloading or completed")
|
||||
case errors.Is(err, downloads.ErrProfileRequired):
|
||||
writeError(w, http.StatusBadRequest, "profile_required", "A profile is required for managed downloads")
|
||||
default:
|
||||
slog.Error("failed to patch download", "download_id", id, "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to update download")
|
||||
}
|
||||
slog.Error("failed to cancel download", "download_id", id, "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to cancel download")
|
||||
return
|
||||
}
|
||||
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
@@ -204,41 +386,49 @@ func (h *DownloadHandler) HandleDownloadFile(w http.ResponseWriter, r *http.Requ
|
||||
writeError(w, http.StatusUnauthorized, "unauthorized", "Authentication required")
|
||||
return
|
||||
}
|
||||
|
||||
id := chi.URLParam(r, "id")
|
||||
if id == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Download ID is required")
|
||||
return
|
||||
}
|
||||
|
||||
if h.svc == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Downloads not configured")
|
||||
return
|
||||
}
|
||||
|
||||
if err := h.svc.ServeFile(r.Context(), w, r, userID, id); err != nil {
|
||||
if errors.Is(err, download.ErrNotFound) {
|
||||
profileID, deviceID, _, _ := managedIdentity(r)
|
||||
filter := requestAccessFilter(r)
|
||||
if err := h.svc.ServeFile(r.Context(), w, r, userID, profileID, deviceID, id, filter); err != nil {
|
||||
if errors.Is(err, downloads.ErrNotFound) || errors.Is(err, catalog.ErrItemNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Download not found")
|
||||
return
|
||||
}
|
||||
if errors.Is(err, download.ErrDownloadNotActive) {
|
||||
if errors.Is(err, downloads.ErrDownloadNotActive) {
|
||||
writeError(w, http.StatusConflict, "download_inactive", "This download is no longer active")
|
||||
return
|
||||
}
|
||||
if errors.Is(err, downloads.ErrProfileRequired) {
|
||||
writeError(w, http.StatusBadRequest, "profile_required", "A profile is required for managed downloads")
|
||||
return
|
||||
}
|
||||
if errors.Is(err, downloads.ErrFeatureDisabled) || errors.Is(err, downloads.ErrDownloadNotAllowed) {
|
||||
writeError(w, http.StatusForbidden, "forbidden", "You are not allowed to download")
|
||||
return
|
||||
}
|
||||
slog.Error("failed to serve download file", "download_id", id, "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to serve download")
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// HandleDirectDownload handles GET /direct-download?file_id=N.
|
||||
// HandleDirectDownload handles GET /direct-download?file_id=N. A legacy
|
||||
// format=original query is accepted, but direct downloads remain original-only.
|
||||
func (h *DownloadHandler) HandleDirectDownload(w http.ResponseWriter, r *http.Request) {
|
||||
userID := apimw.GetUserID(r.Context())
|
||||
if userID == 0 {
|
||||
writeError(w, http.StatusUnauthorized, "unauthorized", "Authentication required")
|
||||
return
|
||||
}
|
||||
|
||||
if h.svc == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Downloads not configured")
|
||||
return
|
||||
@@ -252,23 +442,163 @@ func (h *DownloadHandler) HandleDirectDownload(w http.ResponseWriter, r *http.Re
|
||||
}
|
||||
|
||||
filter := requestAccessFilter(r)
|
||||
|
||||
if err := h.svc.ServeDirect(r.Context(), w, r, userID, fileID, filter); err != nil {
|
||||
if err := h.svc.ServeDirect(r.Context(), w, r, userID, fileID, r.URL.Query().Get("format"), filter); err != nil {
|
||||
h.writeDownloadError(w, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// requireManaged validates a managed (device-scoped) request: authentication, a
|
||||
// configured service, and the device + profile identity (device_id from the
|
||||
// X-Silo-Device-Id header only, never the body). On failure it writes the error
|
||||
// response and returns ok=false. Shared by every managed-only endpoint — the
|
||||
// offline assets and the series-monitoring subscriptions.
|
||||
func (h *DownloadHandler) requireManaged(w http.ResponseWriter, r *http.Request) (userID int, profileID, deviceID, deviceName, devicePlatform string, ok bool) {
|
||||
userID = apimw.GetUserID(r.Context())
|
||||
if userID == 0 {
|
||||
writeError(w, http.StatusUnauthorized, "unauthorized", "Authentication required")
|
||||
return
|
||||
}
|
||||
if h.svc == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Downloads not configured")
|
||||
return
|
||||
}
|
||||
profileID, deviceID, deviceName, devicePlatform = managedIdentity(r)
|
||||
if deviceID == "" {
|
||||
writeError(w, http.StatusBadRequest, "device_id_required", "X-Silo-Device-Id header is required")
|
||||
return
|
||||
}
|
||||
if profileID == "" {
|
||||
writeError(w, http.StatusBadRequest, "profile_required", "A profile is required for managed downloads")
|
||||
return
|
||||
}
|
||||
return userID, profileID, deviceID, deviceName, devicePlatform, true
|
||||
}
|
||||
|
||||
// managedAssetIdentity is requireManaged plus the URL {id} param, for the offline
|
||||
// asset endpoints (manifest/artwork/subtitle).
|
||||
func (h *DownloadHandler) managedAssetIdentity(w http.ResponseWriter, r *http.Request) (userID int, profileID, deviceID, id string, ok bool) {
|
||||
userID, profileID, deviceID, _, _, ok = h.requireManaged(w, r)
|
||||
if !ok {
|
||||
return userID, profileID, deviceID, "", false
|
||||
}
|
||||
id = chi.URLParam(r, "id")
|
||||
if id == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Download ID is required")
|
||||
return userID, profileID, deviceID, "", false
|
||||
}
|
||||
return userID, profileID, deviceID, id, true
|
||||
}
|
||||
|
||||
// HandleManifest handles GET /downloads/{id}/manifest (managed-only).
|
||||
func (h *DownloadHandler) HandleManifest(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, id, ok := h.managedAssetIdentity(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
manifest, err := h.svc.BuildManifest(r.Context(), userID, profileID, deviceID, id, requestAccessFilter(r))
|
||||
if err != nil {
|
||||
h.writeAssetError(w, "manifest", id, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, manifest)
|
||||
}
|
||||
|
||||
// HandleBatchManifests handles GET /downloads/batches/{batch_id}/manifests
|
||||
// (managed-only).
|
||||
func (h *DownloadHandler) HandleBatchManifests(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, _, _, ok := h.requireManaged(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
batchID := chi.URLParam(r, "batch_id")
|
||||
if batchID == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Batch ID is required")
|
||||
return
|
||||
}
|
||||
manifests, skipped, err := h.svc.BuildBatchManifests(r.Context(), userID, profileID, deviceID, batchID, requestAccessFilter(r))
|
||||
if err != nil {
|
||||
h.writeAssetError(w, "batch_manifests", batchID, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, batchManifestsResponse{Manifests: manifests, Skipped: skipped})
|
||||
}
|
||||
|
||||
// HandleArtwork handles GET /downloads/{id}/artwork/{kind} (managed-only).
|
||||
func (h *DownloadHandler) HandleArtwork(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, id, ok := h.managedAssetIdentity(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
kind := chi.URLParam(r, "kind")
|
||||
if err := h.svc.ServeArtwork(r.Context(), w, r, userID, profileID, deviceID, id, kind, requestAccessFilter(r)); err != nil {
|
||||
h.writeAssetError(w, "artwork", id, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// HandleSubtitle handles GET /downloads/{id}/subtitles/{ref} (managed-only).
|
||||
func (h *DownloadHandler) HandleSubtitle(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, id, ok := h.managedAssetIdentity(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
ref := chi.URLParam(r, "ref")
|
||||
if err := h.svc.ServeSubtitle(r.Context(), w, r, userID, profileID, deviceID, id, ref, requestAccessFilter(r)); err != nil {
|
||||
h.writeAssetError(w, "subtitle", id, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// writeAssetError maps offline asset (manifest/artwork/subtitle) errors. Access
|
||||
// denials and missing assets collapse to 404 so a download id never reveals the
|
||||
// existence of out-of-scope content.
|
||||
func (h *DownloadHandler) writeAssetError(w http.ResponseWriter, asset, id string, err error) {
|
||||
switch {
|
||||
case errors.Is(err, downloads.ErrNotFound),
|
||||
errors.Is(err, downloads.ErrAssetNotFound),
|
||||
errors.Is(err, catalog.ErrItemNotFound):
|
||||
writeError(w, http.StatusNotFound, "not_found", "Not found")
|
||||
case errors.Is(err, downloads.ErrInvalidSubtitleRef):
|
||||
writeError(w, http.StatusBadRequest, "invalid_subtitle_ref", "Invalid subtitle reference")
|
||||
case errors.Is(err, downloads.ErrDownloadNotActive):
|
||||
writeError(w, http.StatusConflict, "download_inactive", "This download is no longer active")
|
||||
case errors.Is(err, downloads.ErrProfileRequired):
|
||||
writeError(w, http.StatusBadRequest, "profile_required", "A profile is required for managed downloads")
|
||||
case errors.Is(err, downloads.ErrManifestUnavailable):
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Offline assets are not configured")
|
||||
case errors.Is(err, downloads.ErrFeatureDisabled), errors.Is(err, downloads.ErrDownloadNotAllowed):
|
||||
writeError(w, http.StatusForbidden, "forbidden", "You are not allowed to download")
|
||||
default:
|
||||
slog.Error("failed to serve download asset", "asset", asset, "download_id", id, "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to serve download asset")
|
||||
}
|
||||
}
|
||||
|
||||
func (h *DownloadHandler) writeDownloadError(w http.ResponseWriter, err error) {
|
||||
switch {
|
||||
case errors.Is(err, download.ErrFeatureDisabled):
|
||||
case errors.Is(err, downloads.ErrFeatureDisabled):
|
||||
writeError(w, http.StatusForbidden, "feature_disabled", "Downloads are disabled")
|
||||
case errors.Is(err, download.ErrDownloadNotAllowed):
|
||||
case errors.Is(err, downloads.ErrDownloadNotAllowed):
|
||||
writeError(w, http.StatusForbidden, "forbidden", "You are not allowed to download")
|
||||
case errors.Is(err, download.ErrConcurrentLimitReached):
|
||||
case errors.Is(err, downloads.ErrTranscodeDisabled):
|
||||
writeError(w, http.StatusForbidden, "transcode_disabled", "Download transcoding is disabled")
|
||||
case errors.Is(err, downloads.ErrInvalidQuality):
|
||||
writeError(w, http.StatusBadRequest, "invalid_quality", "Unknown download quality")
|
||||
case errors.Is(err, downloads.ErrProfileRequired):
|
||||
writeError(w, http.StatusBadRequest, "profile_required", "A profile is required for managed downloads")
|
||||
case errors.Is(err, downloads.ErrBulkQualityUnavailable):
|
||||
writeError(w, http.StatusNotImplemented, "bulk_quality_unavailable", "Bitrate quality is not available for bulk downloads yet")
|
||||
case errors.Is(err, downloads.ErrQualityUnavailable):
|
||||
writeError(w, http.StatusNotImplemented, "quality_unavailable", "This download quality is not available")
|
||||
case errors.Is(err, downloads.ErrFormatUnavailable):
|
||||
writeError(w, http.StatusNotImplemented, "format_unavailable", "This download format is not available yet")
|
||||
case errors.Is(err, downloads.ErrConcurrentLimitReached):
|
||||
writeError(w, http.StatusTooManyRequests, "download_limit_exceeded", "Maximum concurrent downloads reached")
|
||||
case errors.Is(err, download.ErrPeriodLimitReached):
|
||||
case errors.Is(err, downloads.ErrPeriodLimitReached):
|
||||
writeError(w, http.StatusTooManyRequests, "download_quota_exceeded", "Download quota exceeded for this period")
|
||||
case errors.Is(err, downloads.ErrNoDownloadableEpisodes):
|
||||
writeError(w, http.StatusNotFound, "no_downloadable_episodes", "No downloadable episodes found")
|
||||
case errors.Is(err, catalog.ErrItemNotFound):
|
||||
writeError(w, http.StatusNotFound, "not_found", "Media item not found")
|
||||
default:
|
||||
|
||||
@@ -0,0 +1,826 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
"github.com/Silo-Server/silo-server/internal/auth"
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/downloads"
|
||||
)
|
||||
|
||||
// fakeDownloadService is a programmable DownloadService for handler tests. It
|
||||
// records the identity (user/profile/device) it was called with so tests can
|
||||
// assert the handler threads header-only device authority correctly.
|
||||
type fakeDownloadService struct {
|
||||
capability downloads.Capability
|
||||
created *downloads.Download
|
||||
createErr error
|
||||
series []*downloads.Download
|
||||
seriesID string
|
||||
seriesErr error
|
||||
list []*downloads.Download
|
||||
listErr error
|
||||
deleteErr error
|
||||
patchErr error
|
||||
serveErr error
|
||||
directErr error
|
||||
manifest *downloads.OfflineManifest
|
||||
batchManifests []*downloads.OfflineManifest
|
||||
batchSkipped []downloads.SkippedManifest
|
||||
manifestErr error
|
||||
artworkErr error
|
||||
subtitleErr error
|
||||
|
||||
gotCreateReq downloads.CreateRequest
|
||||
gotSeriesReq downloads.CreateRequest
|
||||
gotList identityCall
|
||||
gotServe identityCall
|
||||
gotDelete identityCall
|
||||
gotPatch identityCall
|
||||
gotPatchStatus string
|
||||
gotManifest identityCall
|
||||
gotBatchManifest identityCall
|
||||
gotArtwork identityCall
|
||||
gotArtworkKind string
|
||||
gotSubtitle identityCall
|
||||
gotSubtitleRef string
|
||||
gotDirectFormat string
|
||||
gotDirectFileID int
|
||||
|
||||
// Season download + series-monitoring (subscription) fakes.
|
||||
season []*downloads.Download
|
||||
seasonID string
|
||||
seasonErr error
|
||||
gotSeasonReq downloads.CreateRequest
|
||||
gotSeasonNum int
|
||||
seasonCalled bool
|
||||
|
||||
subResult *downloads.SubscriptionResult
|
||||
subList []*downloads.Subscription
|
||||
sub *downloads.Subscription
|
||||
subErr error
|
||||
subDeleteErr error
|
||||
gotSubReq downloads.SubscriptionRequest
|
||||
gotSubPatch downloads.SubscriptionPatch
|
||||
gotSubIdent identityCall
|
||||
syncRegistered int
|
||||
}
|
||||
|
||||
type identityCall struct {
|
||||
userID int
|
||||
profileID string
|
||||
deviceID string
|
||||
downloadID string
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) Capability(context.Context, int) (downloads.Capability, error) {
|
||||
return f.capability, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) Create(_ context.Context, _ int, req downloads.CreateRequest, _ catalog.AccessFilter) (*downloads.Download, error) {
|
||||
f.gotCreateReq = req
|
||||
if f.createErr != nil {
|
||||
return nil, f.createErr
|
||||
}
|
||||
return f.created, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) CreateSeries(_ context.Context, _ int, req downloads.CreateRequest, _ catalog.AccessFilter) ([]*downloads.Download, string, []downloads.SkippedDownload, error) {
|
||||
f.gotSeriesReq = req
|
||||
if f.seriesErr != nil {
|
||||
return nil, "", nil, f.seriesErr
|
||||
}
|
||||
return f.series, f.seriesID, nil, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) CreateSeason(_ context.Context, _ int, req downloads.CreateRequest, seasonNumber int, _ catalog.AccessFilter) ([]*downloads.Download, string, []downloads.SkippedDownload, error) {
|
||||
f.gotSeasonReq = req
|
||||
f.gotSeasonNum = seasonNumber
|
||||
f.seasonCalled = true
|
||||
if f.seasonErr != nil {
|
||||
return nil, "", nil, f.seasonErr
|
||||
}
|
||||
return f.season, f.seasonID, nil, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) CreateSubscription(_ context.Context, _ int, req downloads.SubscriptionRequest, _ catalog.AccessFilter) (*downloads.SubscriptionResult, error) {
|
||||
f.gotSubReq = req
|
||||
if f.subErr != nil {
|
||||
return nil, f.subErr
|
||||
}
|
||||
return f.subResult, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) ListSubscriptions(_ context.Context, userID int, profileID, deviceID string) ([]*downloads.Subscription, error) {
|
||||
f.gotSubIdent = identityCall{userID: userID, profileID: profileID, deviceID: deviceID}
|
||||
return f.subList, f.subErr
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) GetSubscription(_ context.Context, userID int, profileID, deviceID, id string) (*downloads.Subscription, error) {
|
||||
f.gotSubIdent = identityCall{userID, profileID, deviceID, id}
|
||||
if f.subErr != nil {
|
||||
return nil, f.subErr
|
||||
}
|
||||
return f.sub, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) UpdateSubscription(_ context.Context, userID int, profileID, deviceID, id string, patch downloads.SubscriptionPatch, _ catalog.AccessFilter) (*downloads.SubscriptionResult, error) {
|
||||
f.gotSubIdent = identityCall{userID, profileID, deviceID, id}
|
||||
f.gotSubPatch = patch
|
||||
if f.subErr != nil {
|
||||
return nil, f.subErr
|
||||
}
|
||||
return f.subResult, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) DeleteSubscription(_ context.Context, userID int, profileID, deviceID, id string) error {
|
||||
f.gotSubIdent = identityCall{userID, profileID, deviceID, id}
|
||||
return f.subDeleteErr
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) SyncSubscriptions(_ context.Context, userID int, profileID, deviceID string, _ catalog.AccessFilter) (int, error) {
|
||||
f.gotSubIdent = identityCall{userID: userID, profileID: profileID, deviceID: deviceID}
|
||||
if f.subErr != nil {
|
||||
return 0, f.subErr
|
||||
}
|
||||
return f.syncRegistered, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) ServeDirect(_ context.Context, w http.ResponseWriter, _ *http.Request, _, fileID int, format string, _ catalog.AccessFilter) error {
|
||||
f.gotDirectFileID = fileID
|
||||
f.gotDirectFormat = format
|
||||
if f.directErr != nil {
|
||||
return f.directErr
|
||||
}
|
||||
_, _ = w.Write([]byte("served"))
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) ServeFile(_ context.Context, w http.ResponseWriter, _ *http.Request, userID int, profileID, deviceID, downloadID string, _ catalog.AccessFilter) error {
|
||||
f.gotServe = identityCall{userID, profileID, deviceID, downloadID}
|
||||
if f.serveErr != nil {
|
||||
return f.serveErr
|
||||
}
|
||||
_, _ = w.Write([]byte("served"))
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) List(_ context.Context, userID int, profileID, deviceID string) ([]*downloads.Download, error) {
|
||||
f.gotList = identityCall{userID: userID, profileID: profileID, deviceID: deviceID}
|
||||
return f.list, f.listErr
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) Delete(_ context.Context, userID int, profileID, deviceID, downloadID string) error {
|
||||
f.gotDelete = identityCall{userID, profileID, deviceID, downloadID}
|
||||
return f.deleteErr
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) PatchStatus(_ context.Context, userID int, profileID, deviceID, downloadID, status string) error {
|
||||
f.gotPatch = identityCall{userID, profileID, deviceID, downloadID}
|
||||
f.gotPatchStatus = status
|
||||
return f.patchErr
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) BuildManifest(_ context.Context, userID int, profileID, deviceID, downloadID string, _ catalog.AccessFilter) (*downloads.OfflineManifest, error) {
|
||||
f.gotManifest = identityCall{userID, profileID, deviceID, downloadID}
|
||||
if f.manifestErr != nil {
|
||||
return nil, f.manifestErr
|
||||
}
|
||||
return f.manifest, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) BuildBatchManifests(_ context.Context, userID int, profileID, deviceID, batchID string, _ catalog.AccessFilter) ([]*downloads.OfflineManifest, []downloads.SkippedManifest, error) {
|
||||
f.gotBatchManifest = identityCall{userID, profileID, deviceID, batchID}
|
||||
if f.manifestErr != nil {
|
||||
return nil, nil, f.manifestErr
|
||||
}
|
||||
return f.batchManifests, f.batchSkipped, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) ServeArtwork(_ context.Context, w http.ResponseWriter, _ *http.Request, userID int, profileID, deviceID, downloadID, kind string, _ catalog.AccessFilter) error {
|
||||
f.gotArtwork = identityCall{userID, profileID, deviceID, downloadID}
|
||||
f.gotArtworkKind = kind
|
||||
if f.artworkErr != nil {
|
||||
return f.artworkErr
|
||||
}
|
||||
_, _ = w.Write([]byte("img"))
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) ServeSubtitle(_ context.Context, w http.ResponseWriter, _ *http.Request, userID int, profileID, deviceID, downloadID, ref string, _ catalog.AccessFilter) error {
|
||||
f.gotSubtitle = identityCall{userID, profileID, deviceID, downloadID}
|
||||
f.gotSubtitleRef = ref
|
||||
if f.subtitleErr != nil {
|
||||
return f.subtitleErr
|
||||
}
|
||||
_, _ = w.Write([]byte("WEBVTT"))
|
||||
return nil
|
||||
}
|
||||
|
||||
// downloadTestRequest builds a request with auth claims and optional profile +
|
||||
// device identity (as the viewer-access middleware / client headers would set).
|
||||
func downloadTestRequest(method, target string, body []byte, userID int, profileID, deviceID string) *http.Request {
|
||||
var r *http.Request
|
||||
if body != nil {
|
||||
r = httptest.NewRequest(method, target, bytes.NewReader(body))
|
||||
} else {
|
||||
r = httptest.NewRequest(method, target, nil)
|
||||
}
|
||||
if userID != 0 {
|
||||
ctx := apimw.SetClaims(r.Context(), &auth.Claims{
|
||||
UserID: userID,
|
||||
Role: "user",
|
||||
TokenType: auth.TokenTypeAccess,
|
||||
})
|
||||
if profileID != "" {
|
||||
ctx = apimw.SetProfileID(ctx, profileID)
|
||||
}
|
||||
r = r.WithContext(ctx)
|
||||
}
|
||||
if deviceID != "" {
|
||||
r.Header.Set("X-Silo-Device-Id", deviceID)
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
func withChiID(r *http.Request, id string) *http.Request {
|
||||
rctx := chi.NewRouteContext()
|
||||
rctx.URLParams.Add("id", id)
|
||||
return r.WithContext(context.WithValue(r.Context(), chi.RouteCtxKey, rctx))
|
||||
}
|
||||
|
||||
func TestHandleCapability(t *testing.T) {
|
||||
svc := &fakeDownloadService{capability: downloads.Capability{
|
||||
Enabled: true,
|
||||
DownloadAllowed: true,
|
||||
QualityPresets: []string{downloads.QualityOriginal},
|
||||
TranscodeEnabled: false,
|
||||
TranscodeUserAllowed: true,
|
||||
}}
|
||||
h := NewDownloadHandler(svc)
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCapability(rec, downloadTestRequest(http.MethodGet, "/downloads/capability", nil, 7, "", ""))
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
var resp downloadCapabilityResponse
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
if !resp.Enabled || !resp.DownloadAllowed || resp.TranscodeEnabled || !resp.TranscodeUserAllowed {
|
||||
t.Fatalf("unexpected capability flags: %+v", resp)
|
||||
}
|
||||
if len(resp.QualityPresets) != 1 || resp.QualityPresets[0] != downloads.QualityOriginal {
|
||||
t.Fatalf("quality presets = %v, want [original]", resp.QualityPresets)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleCapabilityUnauthorized(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCapability(rec, httptest.NewRequest(http.MethodGet, "/downloads/capability", nil))
|
||||
if rec.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("status = %d, want 401", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleCapabilityNilService(t *testing.T) {
|
||||
h := NewDownloadHandler(nil)
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCapability(rec, downloadTestRequest(http.MethodGet, "/downloads/capability", nil, 7, "", ""))
|
||||
if rec.Code != http.StatusServiceUnavailable {
|
||||
t.Fatalf("status = %d, want 503", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleCreateDownloadThreadsQuality(t *testing.T) {
|
||||
svc := &fakeDownloadService{created: &downloads.Download{
|
||||
ID: "dl1", ContentID: "c1", Status: downloads.StatusQueued, Format: downloads.FormatTranscode,
|
||||
Quality: downloads.Quality5Mbps, EffectiveQuality: downloads.Quality5Mbps, TargetBitrateKbps: 5000, Revision: 2,
|
||||
}}
|
||||
h := NewDownloadHandler(svc)
|
||||
|
||||
body, _ := json.Marshal(downloadRequest{ContentID: "c1", Quality: downloads.Quality5Mbps})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateDownload(rec, downloadTestRequest(http.MethodPost, "/downloads", body, 7, "", ""))
|
||||
|
||||
if rec.Code != http.StatusAccepted {
|
||||
t.Fatalf("status = %d, want 202 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotCreateReq.Quality != downloads.Quality5Mbps {
|
||||
t.Fatalf("service received quality %q, want 5mbps", svc.gotCreateReq.Quality)
|
||||
}
|
||||
var resp downloadResponse
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
if resp.Quality != downloads.Quality5Mbps || resp.DeliveryFormat != downloads.FormatTranscode ||
|
||||
resp.TargetBitrateKbps != 5000 || resp.Revision != 2 {
|
||||
t.Fatalf("response = %+v, want quality/delivery/bitrate/revision", resp)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleCreateDownloadSeriesThreadsQuality(t *testing.T) {
|
||||
svc := &fakeDownloadService{
|
||||
series: []*downloads.Download{{ID: "dl1", ContentID: "s1", Format: downloads.FormatOriginal, Quality: downloads.QualityOriginal, EffectiveQuality: downloads.QualityOriginal, Revision: 1}},
|
||||
seriesID: "batch1",
|
||||
}
|
||||
h := NewDownloadHandler(svc)
|
||||
|
||||
body, _ := json.Marshal(downloadRequest{ContentID: "s1", Series: true, Quality: downloads.QualityOriginal})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateDownload(rec, downloadTestRequest(http.MethodPost, "/downloads", body, 7, "", ""))
|
||||
|
||||
if rec.Code != http.StatusAccepted {
|
||||
t.Fatalf("status = %d, want 202 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotSeriesReq.Quality != downloads.QualityOriginal {
|
||||
t.Fatalf("series service received quality %q, want original", svc.gotSeriesReq.Quality)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleCreateDownloadMissingContentID(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{})
|
||||
body, _ := json.Marshal(downloadRequest{Quality: downloads.QualityOriginal})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateDownload(rec, downloadTestRequest(http.MethodPost, "/downloads", body, 7, "", ""))
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleCreateDownloadErrorMapping(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
err error
|
||||
wantCode int
|
||||
}{
|
||||
{"feature disabled", downloads.ErrFeatureDisabled, http.StatusForbidden},
|
||||
{"not allowed", downloads.ErrDownloadNotAllowed, http.StatusForbidden},
|
||||
{"transcode disabled", downloads.ErrTranscodeDisabled, http.StatusForbidden},
|
||||
{"invalid quality", downloads.ErrInvalidQuality, http.StatusBadRequest},
|
||||
{"quality unavailable", downloads.ErrQualityUnavailable, http.StatusNotImplemented},
|
||||
{"bulk quality unavailable", downloads.ErrBulkQualityUnavailable, http.StatusNotImplemented},
|
||||
{"format unavailable", downloads.ErrFormatUnavailable, http.StatusNotImplemented},
|
||||
{"profile required", downloads.ErrProfileRequired, http.StatusBadRequest},
|
||||
{"concurrent limit", downloads.ErrConcurrentLimitReached, http.StatusTooManyRequests},
|
||||
{"period limit", downloads.ErrPeriodLimitReached, http.StatusTooManyRequests},
|
||||
{"item not found", catalog.ErrItemNotFound, http.StatusNotFound},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{createErr: tc.err})
|
||||
body, _ := json.Marshal(downloadRequest{ContentID: "c1"})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateDownload(rec, downloadTestRequest(http.MethodPost, "/downloads", body, 7, "", ""))
|
||||
if rec.Code != tc.wantCode {
|
||||
t.Fatalf("status = %d, want %d (body: %s)", rec.Code, tc.wantCode, rec.Body.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestManagedCreateUsesHeaderDeviceNotBody verifies invariant 2's "device_id
|
||||
// authority is the header only": a device_id placed in the JSON body is ignored,
|
||||
// and the service receives the X-Silo-Device-Id header value + the profile.
|
||||
func TestManagedCreateUsesHeaderDeviceNotBody(t *testing.T) {
|
||||
svc := &fakeDownloadService{created: &downloads.Download{ID: "dl1", ContentID: "c1", DeviceID: "devA"}}
|
||||
h := NewDownloadHandler(svc)
|
||||
|
||||
// Body smuggles a device_id; it is not a field of downloadRequest, so it is
|
||||
// structurally dropped and must never reach the service.
|
||||
body := []byte(`{"content_id":"c1","device_id":"EVIL","profile_id":"EVIL"}`)
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateDownload(rec, downloadTestRequest(http.MethodPost, "/downloads", body, 7, "pA", "devA"))
|
||||
|
||||
if rec.Code != http.StatusAccepted {
|
||||
t.Fatalf("status = %d, want 202 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotCreateReq.DeviceID != "devA" {
|
||||
t.Fatalf("service device id = %q, want header value devA", svc.gotCreateReq.DeviceID)
|
||||
}
|
||||
if svc.gotCreateReq.ProfileID != "pA" {
|
||||
t.Fatalf("service profile id = %q, want context value pA", svc.gotCreateReq.ProfileID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedFileThreadsIdentity(t *testing.T) {
|
||||
svc := &fakeDownloadService{}
|
||||
h := NewDownloadHandler(svc)
|
||||
req := withChiID(downloadTestRequest(http.MethodGet, "/downloads/dl1/file", nil, 7, "pA", "devA"), "dl1")
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleDownloadFile(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotServe != (identityCall{7, "pA", "devA", "dl1"}) {
|
||||
t.Fatalf("serve identity = %+v, want {7 pA devA dl1}", svc.gotServe)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedListThreadsIdentity(t *testing.T) {
|
||||
svc := &fakeDownloadService{}
|
||||
h := NewDownloadHandler(svc)
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleListDownloads(rec, downloadTestRequest(http.MethodGet, "/downloads", nil, 7, "pA", "devA"))
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200", rec.Code)
|
||||
}
|
||||
if svc.gotList.deviceID != "devA" || svc.gotList.profileID != "pA" {
|
||||
t.Fatalf("list identity = %+v, want profile pA device devA", svc.gotList)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedDeleteThreadsIdentity(t *testing.T) {
|
||||
svc := &fakeDownloadService{}
|
||||
h := NewDownloadHandler(svc)
|
||||
req := withChiID(downloadTestRequest(http.MethodDelete, "/downloads/dl1", nil, 7, "pA", "devA"), "dl1")
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleDeleteDownload(rec, req)
|
||||
|
||||
if rec.Code != http.StatusNoContent {
|
||||
t.Fatalf("status = %d, want 204 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotDelete != (identityCall{7, "pA", "devA", "dl1"}) {
|
||||
t.Fatalf("delete identity = %+v, want {7 pA devA dl1}", svc.gotDelete)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleDeleteDownloadNotFound(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{deleteErr: downloads.ErrNotFound})
|
||||
rec := httptest.NewRecorder()
|
||||
req := withChiID(downloadTestRequest(http.MethodDelete, "/downloads/dl1", nil, 7, "", ""), "dl1")
|
||||
h.HandleDeleteDownload(rec, req)
|
||||
if rec.Code != http.StatusNotFound {
|
||||
t.Fatalf("status = %d, want 404", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedPatchRequiresDeviceHeader(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{})
|
||||
// No device header → 400 device_id_required, even with a profile present.
|
||||
req := withChiID(downloadTestRequest(http.MethodPatch, "/downloads/dl1", []byte(`{"status":"completed"}`), 7, "pA", ""), "dl1")
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandlePatchDownload(rec, req)
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400 device_id_required (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedPatchThreadsIdentityAndStatus(t *testing.T) {
|
||||
svc := &fakeDownloadService{}
|
||||
h := NewDownloadHandler(svc)
|
||||
req := withChiID(downloadTestRequest(http.MethodPatch, "/downloads/dl1", []byte(`{"status":"completed"}`), 7, "pA", "devA"), "dl1")
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandlePatchDownload(rec, req)
|
||||
|
||||
if rec.Code != http.StatusNoContent {
|
||||
t.Fatalf("status = %d, want 204 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotPatch != (identityCall{7, "pA", "devA", "dl1"}) {
|
||||
t.Fatalf("patch identity = %+v, want {7 pA devA dl1}", svc.gotPatch)
|
||||
}
|
||||
if svc.gotPatchStatus != downloads.StatusCompleted {
|
||||
t.Fatalf("patch status = %q, want completed", svc.gotPatchStatus)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleDirectDownloadThreadsOriginalFormat(t *testing.T) {
|
||||
svc := &fakeDownloadService{}
|
||||
h := NewDownloadHandler(svc)
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleDirectDownload(rec, downloadTestRequest(http.MethodGet, "/direct-download?file_id=42&format=original", nil, 7, "", ""))
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotDirectFileID != 42 {
|
||||
t.Fatalf("file id = %d, want 42", svc.gotDirectFileID)
|
||||
}
|
||||
if svc.gotDirectFormat != downloads.FormatOriginal {
|
||||
t.Fatalf("direct format = %q, want original", svc.gotDirectFormat)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleDirectDownloadMissingFileID(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleDirectDownload(rec, downloadTestRequest(http.MethodGet, "/direct-download", nil, 7, "", ""))
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func withChiParams(r *http.Request, params map[string]string) *http.Request {
|
||||
rctx := chi.NewRouteContext()
|
||||
for k, v := range params {
|
||||
rctx.URLParams.Add(k, v)
|
||||
}
|
||||
return r.WithContext(context.WithValue(r.Context(), chi.RouteCtxKey, rctx))
|
||||
}
|
||||
|
||||
func TestManagedManifestRequiresDeviceHeader(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{})
|
||||
// Profile present but no device header → 400 device_id_required.
|
||||
req := withChiID(downloadTestRequest(http.MethodGet, "/downloads/dl1/manifest", nil, 7, "pA", ""), "dl1")
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleManifest(rec, req)
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedManifestThreadsIdentity(t *testing.T) {
|
||||
svc := &fakeDownloadService{manifest: &downloads.OfflineManifest{DownloadID: "dl1", Title: "Movie"}}
|
||||
h := NewDownloadHandler(svc)
|
||||
req := withChiID(downloadTestRequest(http.MethodGet, "/downloads/dl1/manifest", nil, 7, "pA", "devA"), "dl1")
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleManifest(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotManifest != (identityCall{7, "pA", "devA", "dl1"}) {
|
||||
t.Fatalf("manifest identity = %+v, want {7 pA devA dl1}", svc.gotManifest)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedBatchManifestsThreadsIdentity(t *testing.T) {
|
||||
svc := &fakeDownloadService{batchManifests: []*downloads.OfflineManifest{
|
||||
{DownloadID: "dl1", Title: "Episode 1"},
|
||||
}}
|
||||
h := NewDownloadHandler(svc)
|
||||
req := withChiParams(downloadTestRequest(http.MethodGet, "/downloads/batches/b1/manifests", nil, 7, "pA", "devA"),
|
||||
map[string]string{"batch_id": "b1"})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleBatchManifests(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotBatchManifest != (identityCall{7, "pA", "devA", "b1"}) {
|
||||
t.Fatalf("batch manifest identity = %+v, want {7 pA devA b1}", svc.gotBatchManifest)
|
||||
}
|
||||
var resp batchManifestsResponse
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
if len(resp.Manifests) != 1 || resp.Manifests[0].DownloadID != "dl1" {
|
||||
t.Fatalf("manifests = %+v, want one dl1 manifest", resp.Manifests)
|
||||
}
|
||||
}
|
||||
|
||||
// TestManagedAssetsDenyRestrictedProfile is the Phase 2 acceptance test: a
|
||||
// profile denied content access (the service returns ErrItemNotFound) gets a
|
||||
// 404 from manifest, artwork, and subtitle — a download id never reveals
|
||||
// out-of-scope content.
|
||||
func TestManagedAssetsDenyRestrictedProfile(t *testing.T) {
|
||||
t.Run("manifest", func(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{manifestErr: catalog.ErrItemNotFound})
|
||||
req := withChiID(downloadTestRequest(http.MethodGet, "/downloads/dl1/manifest", nil, 7, "child", "devC"), "dl1")
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleManifest(rec, req)
|
||||
if rec.Code != http.StatusNotFound {
|
||||
t.Fatalf("manifest status = %d, want 404", rec.Code)
|
||||
}
|
||||
})
|
||||
t.Run("artwork", func(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{artworkErr: catalog.ErrItemNotFound})
|
||||
req := withChiParams(downloadTestRequest(http.MethodGet, "/downloads/dl1/artwork/poster", nil, 7, "child", "devC"),
|
||||
map[string]string{"id": "dl1", "kind": "poster"})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleArtwork(rec, req)
|
||||
if rec.Code != http.StatusNotFound {
|
||||
t.Fatalf("artwork status = %d, want 404", rec.Code)
|
||||
}
|
||||
})
|
||||
t.Run("subtitle", func(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{subtitleErr: catalog.ErrItemNotFound})
|
||||
req := withChiParams(downloadTestRequest(http.MethodGet, "/downloads/dl1/subtitles/external:0", nil, 7, "child", "devC"),
|
||||
map[string]string{"id": "dl1", "ref": "external:0"})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleSubtitle(rec, req)
|
||||
if rec.Code != http.StatusNotFound {
|
||||
t.Fatalf("subtitle status = %d, want 404", rec.Code)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestManagedArtworkThreadsIdentity(t *testing.T) {
|
||||
svc := &fakeDownloadService{}
|
||||
h := NewDownloadHandler(svc)
|
||||
req := withChiParams(downloadTestRequest(http.MethodGet, "/downloads/dl1/artwork/backdrop", nil, 7, "pA", "devA"),
|
||||
map[string]string{"id": "dl1", "kind": "backdrop"})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleArtwork(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotArtwork != (identityCall{7, "pA", "devA", "dl1"}) || svc.gotArtworkKind != "backdrop" {
|
||||
t.Fatalf("artwork identity = %+v kind = %q", svc.gotArtwork, svc.gotArtworkKind)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedSubtitleInvalidRef(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{subtitleErr: downloads.ErrInvalidSubtitleRef})
|
||||
req := withChiParams(downloadTestRequest(http.MethodGet, "/downloads/dl1/subtitles/bogus", nil, 7, "pA", "devA"),
|
||||
map[string]string{"id": "dl1", "ref": "bogus"})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleSubtitle(rec, req)
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func intPtr(i int) *int { return &i }
|
||||
|
||||
// TestHandleCreateDownloadSeasonRoutes verifies that series=true + a season
|
||||
// number routes to CreateSeason (not CreateSeries) with the season threaded.
|
||||
func TestHandleCreateDownloadSeasonRoutes(t *testing.T) {
|
||||
svc := &fakeDownloadService{
|
||||
season: []*downloads.Download{{ID: "dl1", ContentID: "s1", EpisodeID: "e1", Format: downloads.FormatOriginal}},
|
||||
seasonID: "batch1",
|
||||
}
|
||||
h := NewDownloadHandler(svc)
|
||||
|
||||
body, _ := json.Marshal(downloadRequest{ContentID: "s1", Series: true, Season: intPtr(2)})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateDownload(rec, downloadTestRequest(http.MethodPost, "/downloads", body, 7, "pA", "devA"))
|
||||
|
||||
if rec.Code != http.StatusAccepted {
|
||||
t.Fatalf("status = %d, want 202 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotSeasonNum != 2 {
|
||||
t.Fatalf("season number = %d, want 2", svc.gotSeasonNum)
|
||||
}
|
||||
if svc.gotSeasonReq.ContentID != "s1" {
|
||||
t.Fatalf("season content id = %q, want s1", svc.gotSeasonReq.ContentID)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleCreateDownloadSpecialsSeason pins the Specials boundary: season 0
|
||||
// must route to CreateSeason(0), not silently broaden to a full-series
|
||||
// download, and negative seasons are rejected.
|
||||
func TestHandleCreateDownloadSpecialsSeason(t *testing.T) {
|
||||
svc := &fakeDownloadService{
|
||||
season: []*downloads.Download{{ID: "dl1", ContentID: "s1", EpisodeID: "e1", Format: downloads.FormatOriginal}},
|
||||
seasonID: "batch1",
|
||||
}
|
||||
h := NewDownloadHandler(svc)
|
||||
|
||||
body, _ := json.Marshal(downloadRequest{ContentID: "s1", Series: true, Season: intPtr(0)})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateDownload(rec, downloadTestRequest(http.MethodPost, "/downloads", body, 7, "pA", "devA"))
|
||||
|
||||
if rec.Code != http.StatusAccepted {
|
||||
t.Fatalf("status = %d, want 202 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if !svc.seasonCalled || svc.gotSeasonNum != 0 {
|
||||
t.Fatalf("seasonCalled=%v num=%d, want CreateSeason(0)", svc.seasonCalled, svc.gotSeasonNum)
|
||||
}
|
||||
if svc.gotSeriesReq.ContentID != "" {
|
||||
t.Fatal("season 0 must not fall through to CreateSeries")
|
||||
}
|
||||
|
||||
rec = httptest.NewRecorder()
|
||||
body, _ = json.Marshal(downloadRequest{ContentID: "s1", Series: true, Season: intPtr(-1)})
|
||||
h.HandleCreateDownload(rec, downloadTestRequest(http.MethodPost, "/downloads", body, 7, "pA", "devA"))
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("negative season status = %d, want 400", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleCreateSubscriptionThreadsRequest(t *testing.T) {
|
||||
svc := &fakeDownloadService{subResult: &downloads.SubscriptionResult{
|
||||
Subscription: &downloads.Subscription{ID: "sub1", SeriesID: "s1", Mode: downloads.SubModeLatestSeason, Active: true},
|
||||
Registered: 3,
|
||||
}}
|
||||
h := NewDownloadHandler(svc)
|
||||
|
||||
body, _ := json.Marshal(subscriptionRequest{SeriesID: "s1", Mode: downloads.SubModeLatestSeason, DeleteWatched: true, MaxStorageBytes: 1024})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateSubscription(rec, downloadTestRequest(http.MethodPost, "/downloads/subscriptions", body, 7, "pA", "devA"))
|
||||
|
||||
if rec.Code != http.StatusAccepted {
|
||||
t.Fatalf("status = %d, want 202 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotSubReq.SeriesID != "s1" || svc.gotSubReq.Mode != downloads.SubModeLatestSeason {
|
||||
t.Fatalf("subscription request = %+v", svc.gotSubReq)
|
||||
}
|
||||
if svc.gotSubReq.DeviceID != "devA" || svc.gotSubReq.ProfileID != "pA" {
|
||||
t.Fatalf("subscription identity = profile %q device %q, want pA/devA", svc.gotSubReq.ProfileID, svc.gotSubReq.DeviceID)
|
||||
}
|
||||
if !svc.gotSubReq.DeleteWatched || svc.gotSubReq.MaxStorageBytes != 1024 {
|
||||
t.Fatalf("subscription options not threaded: %+v", svc.gotSubReq)
|
||||
}
|
||||
var resp subscriptionResultResponse
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
if resp.Registered != 3 || resp.Subscription.ID != "sub1" {
|
||||
t.Fatalf("unexpected response: %+v", resp)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleCreateSubscriptionRequiresDevice enforces that monitoring is
|
||||
// device-scoped: a missing X-Silo-Device-Id header is a 400 even with a profile.
|
||||
func TestHandleCreateSubscriptionRequiresDevice(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{})
|
||||
body, _ := json.Marshal(subscriptionRequest{SeriesID: "s1", Mode: downloads.SubModeAll})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateSubscription(rec, downloadTestRequest(http.MethodPost, "/downloads/subscriptions", body, 7, "pA", ""))
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleListSubscriptionsThreadsIdentity(t *testing.T) {
|
||||
svc := &fakeDownloadService{subList: []*downloads.Subscription{{ID: "sub1", SeriesID: "s1", Mode: downloads.SubModeAll, Active: true}}}
|
||||
h := NewDownloadHandler(svc)
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleListSubscriptions(rec, downloadTestRequest(http.MethodGet, "/downloads/subscriptions", nil, 7, "pA", "devA"))
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotSubIdent.profileID != "pA" || svc.gotSubIdent.deviceID != "devA" {
|
||||
t.Fatalf("list identity = %+v, want profile pA device devA", svc.gotSubIdent)
|
||||
}
|
||||
var resp subscriptionsListResponse
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
if len(resp.Subscriptions) != 1 || resp.Subscriptions[0].ID != "sub1" {
|
||||
t.Fatalf("unexpected subscriptions: %+v", resp.Subscriptions)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleSubscriptionErrorMapping(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
err error
|
||||
wantCode int
|
||||
}{
|
||||
{"not found", downloads.ErrSubscriptionNotFound, http.StatusNotFound},
|
||||
{"unavailable", downloads.ErrSubscriptionsUnavailable, http.StatusServiceUnavailable},
|
||||
{"invalid mode", downloads.ErrInvalidSubscriptionMode, http.StatusBadRequest},
|
||||
{"seasons required", downloads.ErrSeasonsRequired, http.StatusBadRequest},
|
||||
{"not series", downloads.ErrNotSeries, http.StatusBadRequest},
|
||||
{"not allowed", downloads.ErrDownloadNotAllowed, http.StatusForbidden},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{subErr: tc.err})
|
||||
body, _ := json.Marshal(subscriptionRequest{SeriesID: "s1", Mode: downloads.SubModeAll})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateSubscription(rec, downloadTestRequest(http.MethodPost, "/downloads/subscriptions", body, 7, "pA", "devA"))
|
||||
if rec.Code != tc.wantCode {
|
||||
t.Fatalf("status = %d, want %d (body: %s)", rec.Code, tc.wantCode, rec.Body.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleSyncSubscriptionsThreadsIdentity(t *testing.T) {
|
||||
svc := &fakeDownloadService{syncRegistered: 5}
|
||||
h := NewDownloadHandler(svc)
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleSyncSubscriptions(rec, downloadTestRequest(http.MethodPost, "/downloads/subscriptions/sync", nil, 7, "pA", "devA"))
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotSubIdent.profileID != "pA" || svc.gotSubIdent.deviceID != "devA" {
|
||||
t.Fatalf("sync identity = %+v, want profile pA device devA", svc.gotSubIdent)
|
||||
}
|
||||
var resp subscriptionSyncResponse
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
if resp.Registered != 5 {
|
||||
t.Fatalf("registered = %d, want 5", resp.Registered)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleSyncSubscriptionsRequiresDevice(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{})
|
||||
rec := httptest.NewRecorder()
|
||||
// Profile present, no device header → 400 device_id_required.
|
||||
h.HandleSyncSubscriptions(rec, downloadTestRequest(http.MethodPost, "/downloads/subscriptions/sync", nil, 7, "pA", ""))
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
@@ -25,8 +25,8 @@ type personalDataItemRepository interface {
|
||||
EnsureAccessible(ctx context.Context, contentID string, filter catalog.AccessFilter) error
|
||||
}
|
||||
|
||||
type LocalFavoriteEventDispatcher interface {
|
||||
HandleLocalFavoriteEvent(ctx context.Context, event watchsync.LocalFavoriteEvent) error
|
||||
type LocalListEventDispatcher interface {
|
||||
HandleLocalListEvent(ctx context.Context, event watchsync.LocalListEvent) error
|
||||
}
|
||||
|
||||
// PersonalDataHandler handles favorites, watchlist, and history endpoints.
|
||||
@@ -37,7 +37,7 @@ type PersonalDataHandler struct {
|
||||
seasonRepo *catalog.SeasonRepository
|
||||
detailSvc *catalog.DetailService
|
||||
EventsHub *evt.Hub
|
||||
localFavoriteDispatcher LocalFavoriteEventDispatcher
|
||||
localListDispatcher LocalListEventDispatcher
|
||||
profileStaler ProfileStaler
|
||||
profileRefreshRequester ProfileRefreshRequester
|
||||
ebookProgressStore EbookReaderProgressLister
|
||||
@@ -78,8 +78,8 @@ func (h *PersonalDataHandler) SetSeasonRepo(repo *catalog.SeasonRepository) {
|
||||
h.seasonRepo = repo
|
||||
}
|
||||
|
||||
func (h *PersonalDataHandler) SetLocalFavoriteEventDispatcher(dispatcher LocalFavoriteEventDispatcher) {
|
||||
h.localFavoriteDispatcher = dispatcher
|
||||
func (h *PersonalDataHandler) SetLocalListEventDispatcher(dispatcher LocalListEventDispatcher) {
|
||||
h.localListDispatcher = dispatcher
|
||||
}
|
||||
|
||||
// --- Response types ---
|
||||
@@ -219,7 +219,7 @@ func (h *PersonalDataHandler) HandleAddFavorite(w http.ResponseWriter, r *http.R
|
||||
return
|
||||
}
|
||||
|
||||
h.dispatchLocalFavoriteEvent(r.Context(), watchsync.LocalFavoriteEventAdded, userID, profileID, itemID)
|
||||
h.dispatchLocalListEvent(r.Context(), watchsync.ListKindFavorites, watchsync.ListChangeAdded, userID, profileID, itemID)
|
||||
triggerProfileRefresh(r.Context(), h.profileStaler, h.profileRefreshRequester, userID, profileID)
|
||||
publishUserStateEvent(r.Context(), h.EventsHub, userID, profileID, itemID, "", "favorite", userStateEventState{
|
||||
IsFavorite: boolPtr(true),
|
||||
@@ -249,7 +249,7 @@ func (h *PersonalDataHandler) HandleRemoveFavorite(w http.ResponseWriter, r *htt
|
||||
return
|
||||
}
|
||||
|
||||
h.dispatchLocalFavoriteEvent(r.Context(), watchsync.LocalFavoriteEventRemoved, userID, profileID, itemID)
|
||||
h.dispatchLocalListEvent(r.Context(), watchsync.ListKindFavorites, watchsync.ListChangeRemoved, userID, profileID, itemID)
|
||||
triggerProfileRefresh(r.Context(), h.profileStaler, h.profileRefreshRequester, userID, profileID)
|
||||
publishUserStateEvent(r.Context(), h.EventsHub, userID, profileID, itemID, "", "favorite", userStateEventState{
|
||||
IsFavorite: boolPtr(false),
|
||||
@@ -257,15 +257,15 @@ func (h *PersonalDataHandler) HandleRemoveFavorite(w http.ResponseWriter, r *htt
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
func (h *PersonalDataHandler) dispatchLocalFavoriteEvent(ctx context.Context, kind watchsync.LocalFavoriteEventKind, userID int, profileID string, itemID string) {
|
||||
if h == nil || h.localFavoriteDispatcher == nil || h.itemRepo == nil {
|
||||
func (h *PersonalDataHandler) dispatchLocalListEvent(ctx context.Context, list watchsync.ListKind, change watchsync.ListChange, userID int, profileID string, itemID string) {
|
||||
if h == nil || h.localListDispatcher == nil || h.itemRepo == nil {
|
||||
return
|
||||
}
|
||||
item, err := h.itemRepo.GetByID(ctx, itemID)
|
||||
if err != nil || item == nil {
|
||||
return
|
||||
}
|
||||
favorite := watchsync.LocalFavorite{
|
||||
listItem := watchsync.LocalFavorite{
|
||||
MediaItemID: item.ContentID,
|
||||
Kind: item.Type,
|
||||
Title: item.Title,
|
||||
@@ -275,11 +275,12 @@ func (h *PersonalDataHandler) dispatchLocalFavoriteEvent(ctx context.Context, ki
|
||||
TVDBID: item.TvdbID,
|
||||
FavoritedAt: time.Now().UTC(),
|
||||
}
|
||||
_ = h.localFavoriteDispatcher.HandleLocalFavoriteEvent(ctx, watchsync.LocalFavoriteEvent{
|
||||
Kind: kind,
|
||||
_ = h.localListDispatcher.HandleLocalListEvent(ctx, watchsync.LocalListEvent{
|
||||
List: list,
|
||||
Change: change,
|
||||
UserID: userID,
|
||||
ProfileID: profileID,
|
||||
Favorites: []watchsync.LocalFavorite{favorite},
|
||||
Items: []watchsync.LocalFavorite{listItem},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -374,6 +375,7 @@ func (h *PersonalDataHandler) HandleAddToWatchlist(w http.ResponseWriter, r *htt
|
||||
return
|
||||
}
|
||||
|
||||
h.dispatchLocalListEvent(r.Context(), watchsync.ListKindWatchlist, watchsync.ListChangeAdded, userID, profileID, itemID)
|
||||
triggerProfileRefresh(r.Context(), h.profileStaler, h.profileRefreshRequester, userID, profileID)
|
||||
publishUserStateEvent(r.Context(), h.EventsHub, userID, profileID, itemID, "", "watchlist", userStateEventState{
|
||||
InWatchlist: boolPtr(true),
|
||||
@@ -403,6 +405,7 @@ func (h *PersonalDataHandler) HandleRemoveFromWatchlist(w http.ResponseWriter, r
|
||||
return
|
||||
}
|
||||
|
||||
h.dispatchLocalListEvent(r.Context(), watchsync.ListKindWatchlist, watchsync.ListChangeRemoved, userID, profileID, itemID)
|
||||
triggerProfileRefresh(r.Context(), h.profileStaler, h.profileRefreshRequester, userID, profileID)
|
||||
publishUserStateEvent(r.Context(), h.EventsHub, userID, profileID, itemID, "", "watchlist", userStateEventState{
|
||||
InWatchlist: boolPtr(false),
|
||||
|
||||
@@ -56,14 +56,27 @@ func (h *FilesystemHandler) HandleBrowse(w http.ResponseWriter, r *http.Request)
|
||||
|
||||
result := make([]filesystemBrowseEntry, 0, len(entries))
|
||||
for _, entry := range entries {
|
||||
name := entry.Name()
|
||||
full := filepath.Join(cleaned, name)
|
||||
|
||||
// os.ReadDir reports lstat semantics, so a symlink to a directory has
|
||||
// IsDir() == false. Follow symlink entries with os.Stat and keep them only
|
||||
// when the target is a directory; skip links whose target cannot be stat'd
|
||||
// (broken or looping). Non-symlink, non-directory entries (regular files)
|
||||
// are skipped without the extra stat so large media folders stay fast.
|
||||
if !entry.IsDir() {
|
||||
continue
|
||||
if entry.Type()&os.ModeSymlink == 0 {
|
||||
continue
|
||||
}
|
||||
target, err := os.Stat(full)
|
||||
if err != nil || !target.IsDir() {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
name := entry.Name()
|
||||
result = append(result, filesystemBrowseEntry{
|
||||
Name: name,
|
||||
Path: filepath.Join(cleaned, name),
|
||||
Path: full,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestHandleBrowseFollowsDirectorySymlinks verifies that the folder browser
|
||||
// lists symlinks that resolve to directories (issue #208). os.ReadDir returns
|
||||
// lstat-based entries, so a directory symlink reports IsDir() == false and was
|
||||
// silently dropped; the handler must os.Stat each entry to follow the link.
|
||||
func TestHandleBrowseFollowsDirectorySymlinks(t *testing.T) {
|
||||
if runtime.GOOS == "windows" {
|
||||
t.Skip("symlink creation is unreliable on Windows CI")
|
||||
}
|
||||
t.Parallel()
|
||||
|
||||
root := t.TempDir()
|
||||
|
||||
// A real directory that should always be listed.
|
||||
realDir := filepath.Join(root, "real-dir")
|
||||
if err := os.Mkdir(realDir, 0o755); err != nil {
|
||||
t.Fatalf("mkdir real-dir: %v", err)
|
||||
}
|
||||
|
||||
// The symlink-to-directory target lives outside root so the browser cannot
|
||||
// reach it except by following the link — mirrors the reported setup where
|
||||
// entries under a bind mount point into a separate FUSE mount.
|
||||
linkTarget := filepath.Join(t.TempDir(), "link-target")
|
||||
if err := os.Mkdir(linkTarget, 0o755); err != nil {
|
||||
t.Fatalf("mkdir link-target: %v", err)
|
||||
}
|
||||
dirSymlink := filepath.Join(root, "symlinked-dir")
|
||||
if err := os.Symlink(linkTarget, dirSymlink); err != nil {
|
||||
t.Fatalf("symlink dir: %v", err)
|
||||
}
|
||||
|
||||
// A regular file: must never be listed by the folder browser.
|
||||
regularFile := filepath.Join(root, "note.txt")
|
||||
if err := os.WriteFile(regularFile, []byte("x"), 0o644); err != nil {
|
||||
t.Fatalf("write file: %v", err)
|
||||
}
|
||||
|
||||
// A symlink to a file: resolves to a non-directory, must not be listed.
|
||||
fileSymlink := filepath.Join(root, "note-link")
|
||||
if err := os.Symlink(regularFile, fileSymlink); err != nil {
|
||||
t.Fatalf("symlink file: %v", err)
|
||||
}
|
||||
|
||||
// A broken symlink: os.Stat fails; it must be skipped, not error the request.
|
||||
brokenSymlink := filepath.Join(root, "broken-link")
|
||||
if err := os.Symlink(filepath.Join(root, "does-not-exist"), brokenSymlink); err != nil {
|
||||
t.Fatalf("symlink broken: %v", err)
|
||||
}
|
||||
|
||||
handler := NewFilesystemHandler()
|
||||
req := httptest.NewRequest(http.MethodGet, "/filesystem/browse?path="+root, nil)
|
||||
rec := httptest.NewRecorder()
|
||||
handler.HandleBrowse(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body = %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
|
||||
var resp filesystemBrowseResponse
|
||||
if err := json.NewDecoder(rec.Body).Decode(&resp); err != nil {
|
||||
t.Fatalf("decode response: %v", err)
|
||||
}
|
||||
|
||||
got := make(map[string]string, len(resp.Entries))
|
||||
for _, e := range resp.Entries {
|
||||
got[e.Name] = e.Path
|
||||
}
|
||||
|
||||
if _, ok := got["real-dir"]; !ok {
|
||||
t.Errorf("real-dir missing from entries: %+v", resp.Entries)
|
||||
}
|
||||
if path, ok := got["symlinked-dir"]; !ok {
|
||||
t.Errorf("symlinked-dir (directory symlink) missing from entries: %+v", resp.Entries)
|
||||
} else if want := filepath.Join(root, "symlinked-dir"); path != want {
|
||||
t.Errorf("symlinked-dir path = %q, want %q", path, want)
|
||||
}
|
||||
if _, ok := got["note.txt"]; ok {
|
||||
t.Errorf("regular file note.txt should not be listed: %+v", resp.Entries)
|
||||
}
|
||||
if _, ok := got["note-link"]; ok {
|
||||
t.Errorf("file symlink note-link should not be listed: %+v", resp.Entries)
|
||||
}
|
||||
if _, ok := got["broken-link"]; ok {
|
||||
t.Errorf("broken symlink broken-link should not be listed: %+v", resp.Entries)
|
||||
}
|
||||
}
|
||||
@@ -107,6 +107,15 @@ func NewItemsHandler(
|
||||
}
|
||||
}
|
||||
|
||||
// SetCompletionObserver wires an optional observer notified when a watch
|
||||
// completes (used to auto-remove fully-watched items from the watchlist).
|
||||
func (h *ItemsHandler) SetCompletionObserver(obs watchstate.CompletionObserver) {
|
||||
if h == nil || h.watchState == nil || obs == nil {
|
||||
return
|
||||
}
|
||||
h.watchState.WithCompletionObserver(obs)
|
||||
}
|
||||
|
||||
// SetProfileStaler configures an optional staleness trigger for taste profiles.
|
||||
func (h *ItemsHandler) SetProfileStaler(ps ProfileStaler) {
|
||||
h.profileStaler = ps
|
||||
|
||||
@@ -687,6 +687,26 @@ func (h *LibraryHandler) HandleUpdateLibrary(w http.ResponseWriter, r *http.Requ
|
||||
}
|
||||
}
|
||||
|
||||
// Re-fetch metadata when the library's metadata language changed, so
|
||||
// existing items adopt the new language instead of keeping the one
|
||||
// stamped at first match. Quick mode suffices: the refresh item lister
|
||||
// includes complete-but-language-mismatched items.
|
||||
if h.JobRepo != nil && !strings.EqualFold(strings.TrimSpace(oldFolder.MetadataLanguage), strings.TrimSpace(folder.MetadataLanguage)) {
|
||||
job, jobErr := h.JobRepo.CreateLibraryRefresh(r.Context(), currentAdminUserID(r), adminjob.LibraryRefreshRequest{
|
||||
LibraryID: folder.ID,
|
||||
LibraryName: folder.Name,
|
||||
Mode: adminjob.LibraryRefreshModeQuick,
|
||||
}, "Queued metadata refresh after library language change")
|
||||
if jobErr != nil {
|
||||
var conflict *adminjob.ActiveJobConflictError
|
||||
if !errors.As(jobErr, &conflict) {
|
||||
slog.Warn("queue language-change metadata refresh failed", "library_id", folder.ID, "error", jobErr)
|
||||
}
|
||||
} else {
|
||||
publishEventJob(r.Context(), h.EventsHub, "job.created", job)
|
||||
}
|
||||
}
|
||||
|
||||
// Rescan when paths have changed (folders added or removed).
|
||||
if req.Paths != nil && !slices.Equal(oldFolder.Paths, *req.Paths) {
|
||||
if h.ScanQueue != nil {
|
||||
@@ -2495,10 +2515,15 @@ func (h *LibraryHandler) HandleListUnmatchedItems(w http.ResponseWriter, r *http
|
||||
)`
|
||||
}
|
||||
|
||||
// Manga chapters carry their series' match state; the chapter rows
|
||||
// themselves stay 'pending' and are resolved through the manga series,
|
||||
// so they must not surface as actionable unmatched items here.
|
||||
mangaChapterGuard := ` AND ` + catalog.MangaChapterExclusionWhere("mi")
|
||||
|
||||
countSQL := `
|
||||
SELECT COUNT(*)
|
||||
FROM media_items mi
|
||||
WHERE mi.status IN ('unmatched', 'pending', 'ambiguous')`
|
||||
WHERE mi.status IN ('unmatched', 'pending', 'ambiguous')` + mangaChapterGuard
|
||||
countSQL += filter
|
||||
|
||||
var total int
|
||||
@@ -2521,10 +2546,10 @@ func (h *LibraryHandler) HandleListUnmatchedItems(w http.ResponseWriter, r *http
|
||||
WHERE mil.content_id = mi.content_id
|
||||
LIMIT 1
|
||||
) lib ON true
|
||||
WHERE mi.status IN ('unmatched', 'pending', 'ambiguous')%s
|
||||
WHERE mi.status IN ('unmatched', 'pending', 'ambiguous')%s%s
|
||||
ORDER BY mi.title ASC, mi.content_id ASC
|
||||
LIMIT $%d OFFSET $%d
|
||||
`, filter, len(filterArgs)+1, len(filterArgs)+2)
|
||||
`, mangaChapterGuard, filter, len(filterArgs)+1, len(filterArgs)+2)
|
||||
|
||||
rows, err := h.pool.Query(r.Context(), listSQL, listArgs...)
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/adminjob"
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
)
|
||||
|
||||
type fakeAdminJobCreator struct {
|
||||
mu sync.Mutex
|
||||
refreshs []adminjob.LibraryRefreshRequest
|
||||
}
|
||||
|
||||
func (f *fakeAdminJobCreator) Create(context.Context, adminjob.CreateJobInput) (*models.AdminJob, error) {
|
||||
return &models.AdminJob{}, nil
|
||||
}
|
||||
|
||||
func (f *fakeAdminJobCreator) CreateLibraryRefresh(_ context.Context, _ int, req adminjob.LibraryRefreshRequest, _ string) (*models.AdminJob, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.refreshs = append(f.refreshs, req)
|
||||
return &models.AdminJob{}, nil
|
||||
}
|
||||
|
||||
func (f *fakeAdminJobCreator) libraryRefreshes() []adminjob.LibraryRefreshRequest {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([]adminjob.LibraryRefreshRequest(nil), f.refreshs...)
|
||||
}
|
||||
|
||||
// TestHandleUpdateLibraryLanguageChangeQueuesRefresh covers the trigger half
|
||||
// of issue #211: changing a library's metadata language must enqueue a
|
||||
// library metadata refresh so existing items re-fetch in the new language,
|
||||
// while an update that does not change the language must not enqueue one.
|
||||
func TestHandleUpdateLibraryLanguageChangeQueuesRefresh(t *testing.T) {
|
||||
dsn := os.Getenv("SILO_TEST_DATABASE_URL")
|
||||
if dsn == "" {
|
||||
t.Skip("SILO_TEST_DATABASE_URL is not set")
|
||||
}
|
||||
ctx := context.Background()
|
||||
pool, err := pgxpool.New(ctx, dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("connect db: %v", err)
|
||||
}
|
||||
t.Cleanup(pool.Close)
|
||||
|
||||
var folderID int
|
||||
if err := pool.QueryRow(ctx, `
|
||||
INSERT INTO media_folders (type, name, enabled, metadata_language)
|
||||
VALUES ('movies', 'Lang Change Test', true, 'zh')
|
||||
RETURNING id
|
||||
`).Scan(&folderID); err != nil {
|
||||
t.Fatalf("seed folder: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
_, _ = pool.Exec(ctx, `DELETE FROM media_folders WHERE id = $1`, folderID)
|
||||
})
|
||||
|
||||
jobs := &fakeAdminJobCreator{}
|
||||
h := NewLibraryHandler(catalog.NewFolderRepository(pool), nil, nil, pool, nil)
|
||||
h.JobRepo = jobs
|
||||
|
||||
doUpdate := func(body string) *httptest.ResponseRecorder {
|
||||
req := httptest.NewRequest(http.MethodPut, "/libraries/"+strconv.Itoa(folderID), strings.NewReader(body))
|
||||
chiCtx := chi.NewRouteContext()
|
||||
chiCtx.URLParams.Add("id", strconv.Itoa(folderID))
|
||||
req = req.WithContext(context.WithValue(req.Context(), chi.RouteCtxKey, chiCtx))
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleUpdateLibrary(rec, req)
|
||||
return rec
|
||||
}
|
||||
|
||||
// An update that does not touch the language must not enqueue a refresh.
|
||||
if rec := doUpdate(`{"name":"Lang Change Test Renamed"}`); rec.Code != http.StatusOK {
|
||||
t.Fatalf("rename update status = %d (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if got := jobs.libraryRefreshes(); len(got) != 0 {
|
||||
t.Fatalf("refresh jobs after rename = %d, want 0", len(got))
|
||||
}
|
||||
|
||||
// Changing the metadata language must enqueue a refresh for this library.
|
||||
if rec := doUpdate(`{"metadata_language":"da"}`); rec.Code != http.StatusOK {
|
||||
t.Fatalf("language update status = %d (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
refreshes := jobs.libraryRefreshes()
|
||||
if len(refreshes) != 1 {
|
||||
t.Fatalf("refresh jobs after language change = %d, want 1", len(refreshes))
|
||||
}
|
||||
if refreshes[0].LibraryID != folderID {
|
||||
t.Errorf("refresh library id = %d, want %d", refreshes[0].LibraryID, folderID)
|
||||
}
|
||||
|
||||
// Re-submitting the same language is a no-op update and must not enqueue.
|
||||
if rec := doUpdate(`{"metadata_language":"da"}`); rec.Code != http.StatusOK {
|
||||
t.Fatalf("same-language update status = %d (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if got := jobs.libraryRefreshes(); len(got) != 1 {
|
||||
t.Fatalf("refresh jobs after same-language update = %d, want still 1", len(got))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// TestHandleListUnmatchedItemsExcludesMangaChapters verifies that manga
|
||||
// chapter rows (type='ebook' items linked into a series via manga_chapters)
|
||||
// do not appear in the admin Unmatched queue, while genuinely pending items
|
||||
// still do. Regression test for issue #204.
|
||||
func TestHandleListUnmatchedItemsExcludesMangaChapters(t *testing.T) {
|
||||
dsn := os.Getenv("SILO_TEST_DATABASE_URL")
|
||||
if dsn == "" {
|
||||
t.Skip("set SILO_TEST_DATABASE_URL to run DB-backed unmatched items handler test")
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
pool, err := pgxpool.New(ctx, dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("connect db: %v", err)
|
||||
}
|
||||
t.Cleanup(pool.Close)
|
||||
|
||||
suffix := time.Now().UnixNano()
|
||||
// Embed a unique per-run tag in every seeded title and scope the search
|
||||
// query to it, so concurrent or leftover rows in a shared test database
|
||||
// cannot skew the exact-total assertion below.
|
||||
tag := fmt.Sprintf("issue204-%d", suffix)
|
||||
seriesID := fmt.Sprintf("manga-series-%d", suffix)
|
||||
chapterID := fmt.Sprintf("manga-chapter-%d", suffix)
|
||||
pendingID := fmt.Sprintf("pending-ebook-%d", suffix)
|
||||
t.Cleanup(func() {
|
||||
_, _ = pool.Exec(ctx, `DELETE FROM media_items WHERE content_id = ANY($1)`,
|
||||
[]string{chapterID, seriesID, pendingID})
|
||||
})
|
||||
|
||||
for _, row := range []struct {
|
||||
id, typ, title, status string
|
||||
}{
|
||||
{seriesID, "manga", tag + " Series", "matched"},
|
||||
{chapterID, "ebook", tag + " Series c001", "pending"},
|
||||
{pendingID, "ebook", tag + " Plain Ebook", "pending"},
|
||||
} {
|
||||
if _, err := pool.Exec(ctx, `
|
||||
INSERT INTO media_items (content_id, type, title, status, genres)
|
||||
VALUES ($1, $2, $3, $4, '{}'::text[])
|
||||
`, row.id, row.typ, row.title, row.status); err != nil {
|
||||
t.Fatalf("seed media item %s: %v", row.id, err)
|
||||
}
|
||||
}
|
||||
if _, err := pool.Exec(ctx, `
|
||||
INSERT INTO manga_chapters (chapter_content_id, series_content_id, chapter_index)
|
||||
VALUES ($1, $2, 1)
|
||||
`, chapterID, seriesID); err != nil {
|
||||
t.Fatalf("seed manga chapter link: %v", err)
|
||||
}
|
||||
|
||||
h := NewLibraryHandler(nil, nil, nil, pool, nil)
|
||||
req := httptest.NewRequest(http.MethodGet, "/libraries/unmatched-items?q="+url.QueryEscape(tag), nil)
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleListUnmatchedItems(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
var resp struct {
|
||||
Items []struct {
|
||||
ContentID string `json:"content_id"`
|
||||
} `json:"items"`
|
||||
Total int `json:"total"`
|
||||
}
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("decode response: %v", err)
|
||||
}
|
||||
|
||||
var sawChapter, sawPending bool
|
||||
for _, item := range resp.Items {
|
||||
switch item.ContentID {
|
||||
case chapterID:
|
||||
sawChapter = true
|
||||
case pendingID:
|
||||
sawPending = true
|
||||
}
|
||||
}
|
||||
if !sawPending {
|
||||
t.Errorf("pending ebook %s missing from unmatched list", pendingID)
|
||||
}
|
||||
if sawChapter {
|
||||
t.Errorf("manga chapter %s should be excluded from unmatched list", chapterID)
|
||||
}
|
||||
if resp.Total != 1 {
|
||||
t.Errorf("total = %d, want 1 (chapter must be excluded from the count too)", resp.Total)
|
||||
}
|
||||
}
|
||||
@@ -1566,6 +1566,7 @@ func (h *LibraryCollectionHandler) HandleSyncAdminCollection(w http.ResponseWrit
|
||||
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
||||
return
|
||||
}
|
||||
slog.Error("collection sync failed", "collection_id", collectionID, "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to sync collection")
|
||||
return
|
||||
}
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log/slog"
|
||||
@@ -13,8 +14,27 @@ import (
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/metadata/translation"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
)
|
||||
|
||||
type metadataAIItemAccess interface {
|
||||
GetByID(ctx context.Context, contentID string) (*models.MediaItem, error)
|
||||
EnsureAccessible(ctx context.Context, contentID string, filter catalog.AccessFilter) error
|
||||
}
|
||||
|
||||
type metadataAISeasonLookup interface {
|
||||
GetByID(ctx context.Context, contentID string) (*models.Season, error)
|
||||
}
|
||||
|
||||
type metadataAIEpisodeLookup interface {
|
||||
GetByID(ctx context.Context, contentID string) (*models.Episode, error)
|
||||
}
|
||||
|
||||
type metadataAITarget struct {
|
||||
kind translation.TargetKind
|
||||
accessContentID string
|
||||
}
|
||||
|
||||
// MetadataAIHandler exposes AI translation of catalog descriptions into the
|
||||
// localization tables. The admin routes are mounted under the per-item
|
||||
// metadata curation guard; the on-view route is viewer-facing and enforces
|
||||
@@ -22,7 +42,11 @@ import (
|
||||
type MetadataAIHandler struct {
|
||||
service *translation.Service
|
||||
// ItemAccess authorizes the viewer-facing on-view route; nil disables it.
|
||||
ItemAccess *catalog.ItemRepository
|
||||
ItemAccess metadataAIItemAccess
|
||||
// SeasonLookup and EpisodeLookup let the viewer route resolve non-item
|
||||
// detail pages to the parent series for authorization.
|
||||
SeasonLookup metadataAISeasonLookup
|
||||
EpisodeLookup metadataAIEpisodeLookup
|
||||
}
|
||||
|
||||
// NewMetadataAIHandler creates a handler backed by the given service.
|
||||
@@ -82,7 +106,16 @@ func (h *MetadataAIHandler) HandleTranslateOnView(w http.ResponseWriter, r *http
|
||||
UserID: scope.UserID,
|
||||
ProfileID: scope.ProfileID,
|
||||
}
|
||||
if err := h.ItemAccess.EnsureAccessible(r.Context(), contentID, filter); err != nil {
|
||||
target, err := h.resolveTranslationTarget(r.Context(), contentID)
|
||||
if err != nil {
|
||||
if errors.Is(err, catalog.ErrItemNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Item not found")
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to authorize item")
|
||||
return
|
||||
}
|
||||
if err := h.ItemAccess.EnsureAccessible(r.Context(), target.accessContentID, filter); err != nil {
|
||||
if errors.Is(err, catalog.ErrItemNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Item not found")
|
||||
return
|
||||
@@ -96,7 +129,7 @@ func (h *MetadataAIHandler) HandleTranslateOnView(w http.ResponseWriter, r *http
|
||||
requestedBy = &userID
|
||||
}
|
||||
|
||||
job, err := h.service.RequestOnView(r.Context(), contentID, req.TargetLanguage, requestedBy)
|
||||
job, err := h.service.RequestOnView(r.Context(), target.kind, contentID, req.TargetLanguage, requestedBy)
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, translation.ErrNotConfigured):
|
||||
@@ -115,6 +148,51 @@ func (h *MetadataAIHandler) HandleTranslateOnView(w http.ResponseWriter, r *http
|
||||
writeJSON(w, http.StatusAccepted, map[string]any{"job": job})
|
||||
}
|
||||
|
||||
func (h *MetadataAIHandler) resolveTranslationTarget(ctx context.Context, contentID string) (metadataAITarget, error) {
|
||||
if h.ItemAccess == nil {
|
||||
return metadataAITarget{}, catalog.ErrItemNotFound
|
||||
}
|
||||
|
||||
item, err := h.ItemAccess.GetByID(ctx, contentID)
|
||||
switch {
|
||||
case err == nil:
|
||||
if item == nil {
|
||||
return metadataAITarget{}, catalog.ErrItemNotFound
|
||||
}
|
||||
return metadataAITarget{kind: translation.TargetItem, accessContentID: contentID}, nil
|
||||
case !errors.Is(err, catalog.ErrItemNotFound):
|
||||
return metadataAITarget{}, err
|
||||
}
|
||||
|
||||
if h.SeasonLookup != nil {
|
||||
season, err := h.SeasonLookup.GetByID(ctx, contentID)
|
||||
switch {
|
||||
case err == nil:
|
||||
if season == nil {
|
||||
return metadataAITarget{}, catalog.ErrItemNotFound
|
||||
}
|
||||
return metadataAITarget{kind: translation.TargetSeason, accessContentID: season.SeriesID}, nil
|
||||
case !errors.Is(err, catalog.ErrSeasonNotFound):
|
||||
return metadataAITarget{}, err
|
||||
}
|
||||
}
|
||||
|
||||
if h.EpisodeLookup != nil {
|
||||
episode, err := h.EpisodeLookup.GetByID(ctx, contentID)
|
||||
switch {
|
||||
case err == nil:
|
||||
if episode == nil {
|
||||
return metadataAITarget{}, catalog.ErrItemNotFound
|
||||
}
|
||||
return metadataAITarget{kind: translation.TargetEpisode, accessContentID: episode.SeriesID}, nil
|
||||
case !errors.Is(err, catalog.ErrEpisodeNotFound):
|
||||
return metadataAITarget{}, err
|
||||
}
|
||||
}
|
||||
|
||||
return metadataAITarget{}, catalog.ErrItemNotFound
|
||||
}
|
||||
|
||||
type translateMetadataRequest struct {
|
||||
TargetLanguage string `json:"target_language"`
|
||||
IncludeChildren *bool `json:"include_children"` // default true
|
||||
@@ -134,9 +212,18 @@ func (h *MetadataAIHandler) HandleTranslate(w http.ResponseWriter, r *http.Reque
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "target_language is required")
|
||||
return
|
||||
}
|
||||
includeChildren := true
|
||||
target, err := h.resolveTranslationTarget(r.Context(), contentID)
|
||||
if err != nil {
|
||||
if errors.Is(err, catalog.ErrItemNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Item not found")
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to resolve item")
|
||||
return
|
||||
}
|
||||
includeChildren := target.kind == translation.TargetItem
|
||||
if req.IncludeChildren != nil {
|
||||
includeChildren = *req.IncludeChildren
|
||||
includeChildren = *req.IncludeChildren && target.kind == translation.TargetItem
|
||||
}
|
||||
|
||||
var requestedBy *int
|
||||
@@ -145,7 +232,7 @@ func (h *MetadataAIHandler) HandleTranslate(w http.ResponseWriter, r *http.Reque
|
||||
}
|
||||
|
||||
job, err := h.service.Enqueue(r.Context(), translation.JobRequest{
|
||||
TargetKind: translation.TargetItem,
|
||||
TargetKind: target.kind,
|
||||
ContentID: contentID,
|
||||
TargetLanguage: req.TargetLanguage,
|
||||
IncludeChildren: includeChildren,
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/metadata/translation"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
)
|
||||
|
||||
type fakeMetadataAIItemAccess struct {
|
||||
items map[string]*models.MediaItem
|
||||
ensureErr map[string]error
|
||||
checked []string
|
||||
}
|
||||
|
||||
func (f *fakeMetadataAIItemAccess) GetByID(_ context.Context, contentID string) (*models.MediaItem, error) {
|
||||
if item := f.items[contentID]; item != nil {
|
||||
return item, nil
|
||||
}
|
||||
return nil, catalog.ErrItemNotFound
|
||||
}
|
||||
|
||||
func (f *fakeMetadataAIItemAccess) EnsureAccessible(_ context.Context, contentID string, _ catalog.AccessFilter) error {
|
||||
f.checked = append(f.checked, contentID)
|
||||
return f.ensureErr[contentID]
|
||||
}
|
||||
|
||||
type fakeMetadataAISeasonLookup map[string]*models.Season
|
||||
|
||||
func (f fakeMetadataAISeasonLookup) GetByID(_ context.Context, contentID string) (*models.Season, error) {
|
||||
if season := f[contentID]; season != nil {
|
||||
return season, nil
|
||||
}
|
||||
return nil, catalog.ErrSeasonNotFound
|
||||
}
|
||||
|
||||
type fakeMetadataAIEpisodeLookup map[string]*models.Episode
|
||||
|
||||
func (f fakeMetadataAIEpisodeLookup) GetByID(_ context.Context, contentID string) (*models.Episode, error) {
|
||||
if episode := f[contentID]; episode != nil {
|
||||
return episode, nil
|
||||
}
|
||||
return nil, catalog.ErrEpisodeNotFound
|
||||
}
|
||||
|
||||
func TestMetadataAIResolveOnViewTarget(t *testing.T) {
|
||||
itemAccess := &fakeMetadataAIItemAccess{
|
||||
items: map[string]*models.MediaItem{
|
||||
"movie-1": {ContentID: "movie-1", Type: "movie"},
|
||||
},
|
||||
ensureErr: map[string]error{},
|
||||
}
|
||||
handler := &MetadataAIHandler{
|
||||
ItemAccess: itemAccess,
|
||||
SeasonLookup: fakeMetadataAISeasonLookup{
|
||||
"season-1": {ContentID: "season-1", SeriesID: "series-1"},
|
||||
},
|
||||
EpisodeLookup: fakeMetadataAIEpisodeLookup{
|
||||
"episode-1": {ContentID: "episode-1", SeriesID: "series-1"},
|
||||
},
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
contentID string
|
||||
wantKind translation.TargetKind
|
||||
wantAccess string
|
||||
}{
|
||||
{
|
||||
name: "item",
|
||||
contentID: "movie-1",
|
||||
wantKind: translation.TargetItem,
|
||||
wantAccess: "movie-1",
|
||||
},
|
||||
{
|
||||
name: "season",
|
||||
contentID: "season-1",
|
||||
wantKind: translation.TargetSeason,
|
||||
wantAccess: "series-1",
|
||||
},
|
||||
{
|
||||
name: "episode",
|
||||
contentID: "episode-1",
|
||||
wantKind: translation.TargetEpisode,
|
||||
wantAccess: "series-1",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
itemAccess.checked = nil
|
||||
got, err := handler.resolveTranslationTarget(context.Background(), tc.contentID)
|
||||
if err != nil {
|
||||
t.Fatalf("resolveTranslationTarget: %v", err)
|
||||
}
|
||||
if got.kind != tc.wantKind {
|
||||
t.Fatalf("kind = %s, want %s", got.kind, tc.wantKind)
|
||||
}
|
||||
if got.accessContentID != tc.wantAccess {
|
||||
t.Fatalf("accessContentID = %q, want %q", got.accessContentID, tc.wantAccess)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetadataAIResolveTranslationTargetReportsMissingContent(t *testing.T) {
|
||||
itemAccess := &fakeMetadataAIItemAccess{
|
||||
items: map[string]*models.MediaItem{},
|
||||
ensureErr: map[string]error{},
|
||||
}
|
||||
handler := &MetadataAIHandler{
|
||||
ItemAccess: itemAccess,
|
||||
SeasonLookup: fakeMetadataAISeasonLookup{},
|
||||
EpisodeLookup: fakeMetadataAIEpisodeLookup{},
|
||||
}
|
||||
|
||||
_, err := handler.resolveTranslationTarget(context.Background(), "missing")
|
||||
if !errors.Is(err, catalog.ErrItemNotFound) {
|
||||
t.Fatalf("err = %v, want %v", err, catalog.ErrItemNotFound)
|
||||
}
|
||||
}
|
||||
@@ -43,6 +43,8 @@ type notificationSyncResponse struct {
|
||||
UnreadCount int `json:"unread_count"`
|
||||
}
|
||||
|
||||
type notificationApplePushDisplayResponse = notifications.NotificationDisplay
|
||||
|
||||
type unreadCountResponse struct {
|
||||
Count int `json:"count"`
|
||||
}
|
||||
@@ -150,6 +152,29 @@ func (h *NotificationsHandler) HandleGet(w http.ResponseWriter, r *http.Request)
|
||||
writeJSON(w, http.StatusOK, h.system.PayloadForRow(r.Context(), *row))
|
||||
}
|
||||
|
||||
// HandleApplePushDisplay handles GET /notifications/push/apple/display/{delivery_id}.
|
||||
// It returns only compact display metadata for notification-service-extension
|
||||
// enrichment, scoped to the active profile.
|
||||
func (h *NotificationsHandler) HandleApplePushDisplay(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
profileID := apimw.GetProfileID(r.Context())
|
||||
id := chi.URLParam(r, "delivery_id")
|
||||
if id == "" {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Notification not found")
|
||||
return
|
||||
}
|
||||
row, err := h.system.Deliveries.GetByID(r.Context(), profileID, id)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to load notification")
|
||||
return
|
||||
}
|
||||
if row == nil {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Notification not found")
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, notificationApplePushDisplayResponse(notifications.BuildNotificationDisplay(*row)))
|
||||
}
|
||||
|
||||
// HandleUnreadCount handles GET /notifications/unread-count.
|
||||
func (h *NotificationsHandler) HandleUnreadCount(w http.ResponseWriter, r *http.Request) {
|
||||
profileID := apimw.GetProfileID(r.Context())
|
||||
@@ -318,9 +343,7 @@ type capabilityWebhooks struct {
|
||||
}
|
||||
|
||||
// HandleCapability handles GET /notifications/capability. Clients render
|
||||
// setup UI from this response instead of introspecting admin settings. Push
|
||||
// channels report unavailable until they ship
|
||||
// (docs/superpowers/plans/notifications/02-03).
|
||||
// setup UI from this response instead of introspecting admin settings.
|
||||
func (h *NotificationsHandler) HandleCapability(w http.ResponseWriter, r *http.Request) {
|
||||
webhooks := capabilityWebhooks{Available: false, MaxPerProfile: 0, SupportedTypes: []string{}}
|
||||
if h.system.Webhooks != nil && h.system.Settings.WebhooksEnabled(r.Context()) {
|
||||
@@ -364,9 +387,21 @@ func (h *NotificationsHandler) HandleCapability(w http.ResponseWriter, r *http.R
|
||||
DigestHour: h.system.Settings.DiscordDigestHour(r.Context()),
|
||||
}
|
||||
}
|
||||
applePush := capabilityPush{Available: false, Provider: "off", SupportedModes: []string{"in_app_only"}}
|
||||
// Like web push above, availability requires both the wiring (cipher/store)
|
||||
// and the admin delivery toggle: Available must mean "setup will actually
|
||||
// deliver", not "the server could store a token".
|
||||
if h.system.PushDevices != nil && h.system.PushDevices.Available() &&
|
||||
h.system.Settings.ApplePushDeliveryEnabled(r.Context()) {
|
||||
applePush = capabilityPush{
|
||||
Available: true,
|
||||
Provider: notifications.PushProviderSiloRelay,
|
||||
SupportedModes: []string{notifications.PushModePrivatePush, notifications.PushModeInAppOnly},
|
||||
}
|
||||
}
|
||||
writeJSON(w, http.StatusOK, capabilityResponse{
|
||||
InApp: capabilityInApp{Enabled: h.system.Settings.UIEnabled(r.Context())},
|
||||
ApplePush: capabilityPush{Available: false, Provider: "off", SupportedModes: []string{"in_app_only"}},
|
||||
ApplePush: applePush,
|
||||
AndroidPush: capabilityPush{Available: false, Provider: "off", SupportedModes: []string{"in_app_only"}},
|
||||
WebPush: webPush,
|
||||
Webhooks: webhooks,
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
"github.com/Silo-Server/silo-server/internal/notifications"
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
func TestHandleApplePushDisplayDB(t *testing.T) {
|
||||
dsn := os.Getenv("SILO_TEST_DATABASE_URL")
|
||||
if dsn == "" {
|
||||
t.Skip("set SILO_TEST_DATABASE_URL to run DB-backed notification display handler test")
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
config, err := pgxpool.ParseConfig(dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("parse db config: %v", err)
|
||||
}
|
||||
config.MaxConns = 1
|
||||
pool, err := pgxpool.NewWithConfig(ctx, config)
|
||||
if err != nil {
|
||||
t.Fatalf("connect db: %v", err)
|
||||
}
|
||||
t.Cleanup(pool.Close)
|
||||
|
||||
if _, err := pool.Exec(ctx, `
|
||||
CREATE TEMP TABLE notification_deliveries (
|
||||
id text PRIMARY KEY,
|
||||
release_event_id text,
|
||||
user_id integer NOT NULL,
|
||||
profile_id text NOT NULL,
|
||||
library_id integer,
|
||||
series_id text,
|
||||
episode_id text,
|
||||
type text NOT NULL,
|
||||
reason_flags jsonb NOT NULL,
|
||||
status text NOT NULL DEFAULT 'delivered',
|
||||
read_at timestamptz,
|
||||
delivered_at timestamptz,
|
||||
created_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE TEMP TABLE episodes (
|
||||
content_id text PRIMARY KEY,
|
||||
title text,
|
||||
season_number integer,
|
||||
episode_number integer,
|
||||
overview text
|
||||
);
|
||||
CREATE TEMP TABLE media_items (
|
||||
content_id text PRIMARY KEY,
|
||||
title text,
|
||||
poster_path text,
|
||||
poster_thumbhash text,
|
||||
poster_source_path text,
|
||||
type text,
|
||||
year integer,
|
||||
overview text,
|
||||
genres text[],
|
||||
content_rating text,
|
||||
rating_imdb double precision,
|
||||
rating_tmdb double precision,
|
||||
imdb_id text,
|
||||
tmdb_id text,
|
||||
tvdb_id text
|
||||
);
|
||||
INSERT INTO media_items (content_id, title, type, genres)
|
||||
VALUES ('series-1', 'Severance', 'series', ARRAY[]::text[]);
|
||||
INSERT INTO episodes (content_id, title, season_number, episode_number)
|
||||
VALUES ('episode-1', 'Hello, Ms. Cobel', 2, 1);
|
||||
INSERT INTO notification_deliveries (
|
||||
id, release_event_id, user_id, profile_id, library_id, series_id, episode_id,
|
||||
type, reason_flags, status
|
||||
) VALUES (
|
||||
'delivery-1', 'event-1', 42, 'profile-1', 7, 'series-1', 'episode-1',
|
||||
'episode.available', '{"favorite":true}', 'delivered'
|
||||
);
|
||||
`); err != nil {
|
||||
t.Fatalf("seed temp tables: %v", err)
|
||||
}
|
||||
|
||||
handler := NewNotificationsHandler(¬ifications.System{
|
||||
Deliveries: notifications.NewDeliveryRepository(pool),
|
||||
}, nil)
|
||||
router := chi.NewRouter()
|
||||
router.Get("/notifications/push/apple/display/{delivery_id}", handler.HandleApplePushDisplay)
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/notifications/push/apple/display/delivery-1", nil)
|
||||
req = req.WithContext(apimw.SetProfileID(req.Context(), "profile-1"))
|
||||
rr := httptest.NewRecorder()
|
||||
router.ServeHTTP(rr, req)
|
||||
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body = %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
if got := rr.Header().Get("Cache-Control"); got != "no-store" {
|
||||
t.Fatalf("Cache-Control = %q", got)
|
||||
}
|
||||
var response notifications.NotificationDisplay
|
||||
if err := json.NewDecoder(rr.Body).Decode(&response); err != nil {
|
||||
t.Fatalf("decode response: %v", err)
|
||||
}
|
||||
if response.DeliveryID != "delivery-1" ||
|
||||
response.Title != "The latest episode of Severance S02E01 just dropped!" ||
|
||||
response.Body != "Hello, Ms. Cobel" ||
|
||||
response.ThreadID != "series:series-1" ||
|
||||
response.Category != "episode_available" ||
|
||||
response.URL != "/item/episode-1" {
|
||||
t.Fatalf("response = %+v", response)
|
||||
}
|
||||
|
||||
req = httptest.NewRequest(http.MethodGet, "/notifications/push/apple/display/delivery-1", nil)
|
||||
req = req.WithContext(apimw.SetProfileID(req.Context(), "other-profile"))
|
||||
rr = httptest.NewRecorder()
|
||||
router.ServeHTTP(rr, req)
|
||||
if rr.Code != http.StatusNotFound {
|
||||
t.Fatalf("cross-profile status = %d, body = %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
if got := rr.Header().Get("Cache-Control"); got != "no-store" {
|
||||
t.Fatalf("cross-profile Cache-Control = %q", got)
|
||||
}
|
||||
if strings.Contains(rr.Body.String(), "Severance") {
|
||||
t.Fatalf("cross-profile response leaked display metadata: %s", rr.Body.String())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
"github.com/Silo-Server/silo-server/internal/notifications"
|
||||
)
|
||||
|
||||
type applePushRegisterRequest struct {
|
||||
DeviceID string `json:"device_id"`
|
||||
APNsToken string `json:"apns_token"`
|
||||
APNsEnvironment string `json:"apns_environment"`
|
||||
APNsTopic string `json:"apns_topic"`
|
||||
PushMode string `json:"push_mode"`
|
||||
}
|
||||
|
||||
type applePushRegisterResponse struct {
|
||||
ID string `json:"id"`
|
||||
ServerDeviceID string `json:"server_device_id"`
|
||||
Enabled bool `json:"enabled"`
|
||||
PushMode string `json:"push_mode"`
|
||||
}
|
||||
|
||||
func (h *NotificationsHandler) pushDevices() *notifications.PushDeviceService {
|
||||
if h == nil || h.system == nil {
|
||||
return nil
|
||||
}
|
||||
return h.system.PushDevices
|
||||
}
|
||||
|
||||
// HandleRegisterApplePushDevice handles POST /devices/push/apple.
|
||||
func (h *NotificationsHandler) HandleRegisterApplePushDevice(w http.ResponseWriter, r *http.Request) {
|
||||
service := h.pushDevices()
|
||||
if service == nil || !service.Available() {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Apple push registration is not available")
|
||||
return
|
||||
}
|
||||
|
||||
var req applePushRegisterRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Invalid request body")
|
||||
return
|
||||
}
|
||||
|
||||
device, err := service.RegisterApple(r.Context(), apimw.GetUserID(r.Context()), apimw.GetProfileID(r.Context()), notifications.ApplePushRegistrationInput{
|
||||
DeviceID: req.DeviceID,
|
||||
APNsToken: req.APNsToken,
|
||||
APNsEnvironment: req.APNsEnvironment,
|
||||
APNsTopic: req.APNsTopic,
|
||||
PushMode: req.PushMode,
|
||||
})
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, notifications.ErrPushDeviceInvalid):
|
||||
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
||||
case errors.Is(err, notifications.ErrPushDeviceUnsupported):
|
||||
writeError(w, http.StatusUnprocessableEntity, "unsupported_push_device", err.Error())
|
||||
case errors.Is(err, notifications.ErrPushDeviceUnavailable):
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Apple push registration is not available")
|
||||
default:
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to register Apple push device")
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, applePushRegisterResponse{
|
||||
ID: device.ID,
|
||||
ServerDeviceID: device.ServerDeviceID,
|
||||
Enabled: device.Enabled,
|
||||
PushMode: device.PushMode,
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,152 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
"github.com/Silo-Server/silo-server/internal/auth"
|
||||
"github.com/Silo-Server/silo-server/internal/notifications"
|
||||
"github.com/Silo-Server/silo-server/internal/secret"
|
||||
)
|
||||
|
||||
type handlerPushStore struct {
|
||||
got notifications.ApplePushDeviceRegistration
|
||||
calls int
|
||||
device *notifications.PushDevice
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *handlerPushStore) UpsertApple(ctx context.Context, registration notifications.ApplePushDeviceRegistration, cipher *secret.Cipher) (*notifications.PushDevice, error) {
|
||||
f.calls++
|
||||
f.got = registration
|
||||
if f.err != nil {
|
||||
return nil, f.err
|
||||
}
|
||||
if f.device != nil {
|
||||
return f.device, nil
|
||||
}
|
||||
return ¬ifications.PushDevice{
|
||||
ID: "push-row",
|
||||
ServerDeviceID: "server-device",
|
||||
Enabled: true,
|
||||
PushMode: registration.PushMode,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func handlerPushCipher(t *testing.T) *secret.Cipher {
|
||||
t.Helper()
|
||||
cipher, err := secret.New([]byte("01234567890123456789012345678901"))
|
||||
if err != nil {
|
||||
t.Fatalf("new cipher: %v", err)
|
||||
}
|
||||
return cipher
|
||||
}
|
||||
|
||||
func newApplePushRequest(body string) *http.Request {
|
||||
req := httptest.NewRequest(http.MethodPost, "/api/v1/devices/push/apple", strings.NewReader(body))
|
||||
ctx := apimw.SetClaims(req.Context(), &auth.Claims{UserID: 42, Role: "user", TokenType: auth.TokenTypeAccess})
|
||||
ctx = apimw.SetProfileID(ctx, "profile-1")
|
||||
return req.WithContext(ctx)
|
||||
}
|
||||
|
||||
func TestHandleRegisterApplePushDevice(t *testing.T) {
|
||||
store := &handlerPushStore{}
|
||||
handler := NewNotificationsHandler(¬ifications.System{
|
||||
PushDevices: notifications.NewPushDeviceService(store, handlerPushCipher(t)),
|
||||
}, nil)
|
||||
|
||||
body := `{
|
||||
"device_id":"local-device",
|
||||
"apns_token":"` + strings.Repeat("a", 64) + `",
|
||||
"apns_environment":"production",
|
||||
"apns_topic":"org.siloserver.silo",
|
||||
"push_mode":"private_push"
|
||||
}`
|
||||
rr := httptest.NewRecorder()
|
||||
handler.HandleRegisterApplePushDevice(rr, newApplePushRequest(body))
|
||||
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body = %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
var response applePushRegisterResponse
|
||||
if err := json.NewDecoder(rr.Body).Decode(&response); err != nil {
|
||||
t.Fatalf("decode response: %v", err)
|
||||
}
|
||||
if response.ID != "push-row" || response.ServerDeviceID != "server-device" || !response.Enabled || response.PushMode != notifications.PushModePrivatePush {
|
||||
t.Fatalf("unexpected response: %+v", response)
|
||||
}
|
||||
if store.calls != 1 {
|
||||
t.Fatalf("store calls = %d, want 1", store.calls)
|
||||
}
|
||||
if store.got.UserID != 42 || store.got.ProfileID != "profile-1" || store.got.DeviceID != "local-device" {
|
||||
t.Fatalf("unexpected stored registration: %+v", store.got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleRegisterApplePushDeviceErrors(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
system *notifications.System
|
||||
body string
|
||||
wantStatus int
|
||||
}{
|
||||
{
|
||||
name: "service unavailable",
|
||||
system: ¬ifications.System{},
|
||||
body: `{}`,
|
||||
wantStatus: http.StatusServiceUnavailable,
|
||||
},
|
||||
{
|
||||
name: "invalid json",
|
||||
system: ¬ifications.System{
|
||||
PushDevices: notifications.NewPushDeviceService(&handlerPushStore{}, handlerPushCipher(t)),
|
||||
},
|
||||
body: `{`,
|
||||
wantStatus: http.StatusBadRequest,
|
||||
},
|
||||
{
|
||||
name: "invalid field",
|
||||
system: ¬ifications.System{
|
||||
PushDevices: notifications.NewPushDeviceService(&handlerPushStore{}, handlerPushCipher(t)),
|
||||
},
|
||||
body: `{
|
||||
"device_id":"local-device",
|
||||
"apns_token":"abcd",
|
||||
"apns_environment":"production",
|
||||
"apns_topic":"org.siloserver.silo",
|
||||
"push_mode":"private_push"
|
||||
}`,
|
||||
wantStatus: http.StatusBadRequest,
|
||||
},
|
||||
{
|
||||
name: "unsupported topic",
|
||||
system: ¬ifications.System{
|
||||
PushDevices: notifications.NewPushDeviceService(&handlerPushStore{}, handlerPushCipher(t)),
|
||||
},
|
||||
body: `{
|
||||
"device_id":"local-device",
|
||||
"apns_token":"` + strings.Repeat("a", 64) + `",
|
||||
"apns_environment":"production",
|
||||
"apns_topic":"com.example.app",
|
||||
"push_mode":"private_push"
|
||||
}`,
|
||||
wantStatus: http.StatusUnprocessableEntity,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
handler := NewNotificationsHandler(tt.system, nil)
|
||||
rr := httptest.NewRecorder()
|
||||
handler.HandleRegisterApplePushDevice(rr, newApplePushRequest(tt.body))
|
||||
if rr.Code != tt.wantStatus {
|
||||
t.Fatalf("status = %d, want %d, body = %s", rr.Code, tt.wantStatus, rr.Body.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
+585
-192
File diff suppressed because it is too large
Load Diff
@@ -13,7 +13,11 @@ type playbackCommandRecord struct {
|
||||
Name playback.CommandName
|
||||
}
|
||||
|
||||
func (h *PlaybackHandler) stopPlaybackSession(ctx context.Context, session *playback.Session) error {
|
||||
// stopPlaybackSession stops a session. userInitiated must be true only for a
|
||||
// genuine user/admin stop (the recipe card is then deleted); false for system
|
||||
// teardown such as an ffmpeg-exit cleanup, where the card is kept so the client
|
||||
// can reconstruct.
|
||||
func (h *PlaybackHandler) stopPlaybackSession(ctx context.Context, session *playback.Session, userInitiated bool) error {
|
||||
if h == nil || session == nil || session.ID == "" {
|
||||
return playback.ErrSessionNotFound
|
||||
}
|
||||
@@ -21,11 +25,11 @@ func (h *PlaybackHandler) stopPlaybackSession(ctx context.Context, session *play
|
||||
if err := h.sessionMgr.StopSession(session.ID); err != nil {
|
||||
return err
|
||||
}
|
||||
h.finalizeSessionStop(ctx, session, true, "stop")
|
||||
h.finalizeSessionStop(ctx, session, true, "stop", userInitiated)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (h *PlaybackHandler) stopPlaybackSessionByID(ctx context.Context, sessionID string) error {
|
||||
func (h *PlaybackHandler) stopPlaybackSessionByID(ctx context.Context, sessionID string, userInitiated bool) error {
|
||||
if h == nil || sessionID == "" {
|
||||
return playback.ErrSessionNotFound
|
||||
}
|
||||
@@ -33,7 +37,7 @@ func (h *PlaybackHandler) stopPlaybackSessionByID(ctx context.Context, sessionID
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return h.stopPlaybackSession(ctx, session)
|
||||
return h.stopPlaybackSession(ctx, session, userInitiated)
|
||||
}
|
||||
|
||||
func (h *PlaybackHandler) abortPlaybackSession(ctx context.Context, session *playback.Session) error {
|
||||
|
||||
@@ -22,6 +22,7 @@ import (
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
"github.com/Silo-Server/silo-server/internal/nodepool"
|
||||
"github.com/Silo-Server/silo-server/internal/playback"
|
||||
"github.com/Silo-Server/silo-server/internal/streamtoken"
|
||||
"github.com/Silo-Server/silo-server/internal/transcodenode"
|
||||
"github.com/Silo-Server/silo-server/internal/userdb"
|
||||
"github.com/Silo-Server/silo-server/internal/userstore"
|
||||
@@ -1162,6 +1163,244 @@ func TestHandleStartTranscode_PreservesRecomputedBaseMethodAfterFallback(t *test
|
||||
}
|
||||
}
|
||||
|
||||
// newRemoteAudioSwitchSession builds a session that is already an offloaded
|
||||
// (remote) transcode with a populated target recipe, mirroring the post-start
|
||||
// state HandleChangeAudioTrack sees for an offloaded transcode.
|
||||
func newRemoteAudioSwitchSession(t *testing.T, sessionMgr *playback.SessionManager, file *models.MediaFile, nodeURL string) string {
|
||||
t.Helper()
|
||||
session, err := sessionMgr.StartSession(1, "profile-1", file.ID, playback.PlayTranscode, true)
|
||||
if err != nil {
|
||||
t.Fatalf("StartSession: %v", err)
|
||||
}
|
||||
if err := sessionMgr.UpdateStreamState(session.ID, playback.SessionStreamState{
|
||||
PlayMethod: playback.PlayTranscode,
|
||||
BasePlayMethod: playback.PlayTranscode,
|
||||
AudioTrackIndex: 0,
|
||||
TranscodeAudio: true,
|
||||
TargetResolution: "720p",
|
||||
TargetVideoCodec: "h264",
|
||||
TargetAudioCodec: "aac",
|
||||
TargetBitrateKbps: 2000,
|
||||
TranscodeHWAccel: "none",
|
||||
}); err != nil {
|
||||
t.Fatalf("UpdateStreamState: %v", err)
|
||||
}
|
||||
if err := sessionMgr.SetTranscodeNodeURL(session.ID, nodeURL); err != nil {
|
||||
t.Fatalf("SetTranscodeNodeURL: %v", err)
|
||||
}
|
||||
return session.ID
|
||||
}
|
||||
|
||||
// TestHandleChangeAudioTrack_RemoteTranscodeRestartsNodeAndMintsFullRecipe
|
||||
// verifies BUG A + BUG B: an audio switch on an offloaded transcode POSTs a
|
||||
// fresh /transcode/start to the node carrying the NEW AudioTrackIndex (so the
|
||||
// node's ffmpeg actually switches tracks), and the returned proxy URL carries a
|
||||
// recipe-COMPLETE stream token (not identity-only) so a later node restart can
|
||||
// reconstruct with the switched audio.
|
||||
func TestHandleChangeAudioTrack_RemoteTranscodeRestartsNodeAndMintsFullRecipe(t *testing.T) {
|
||||
sessionMgr := playback.NewSessionManager(0, 0)
|
||||
file := &models.MediaFile{
|
||||
ID: 42,
|
||||
ContentID: "movie-1",
|
||||
FilePath: writePlaybackTestMediaFile(t, "movie.mkv"),
|
||||
Resolution: "1080p",
|
||||
CodecVideo: "h264",
|
||||
CodecAudio: "aac",
|
||||
Container: "mkv",
|
||||
Bitrate: 8000,
|
||||
Duration: 3600,
|
||||
AudioTracks: []models.AudioTrack{
|
||||
{Codec: "aac", Default: true},
|
||||
{Codec: "ac3"},
|
||||
},
|
||||
}
|
||||
|
||||
var remoteStartRequests int
|
||||
var remoteStartReq transcodenode.TranscodeStartRequest
|
||||
remote := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodPost || r.URL.Path != "/transcode/start" {
|
||||
t.Errorf("unexpected remote request %s %s", r.Method, r.URL.Path)
|
||||
http.Error(w, "unexpected", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
remoteStartRequests++
|
||||
if err := json.NewDecoder(r.Body).Decode(&remoteStartReq); err != nil {
|
||||
t.Errorf("decode remote start request: %v", err)
|
||||
}
|
||||
w.WriteHeader(http.StatusAccepted)
|
||||
_ = json.NewEncoder(w).Encode(transcodenode.TranscodeStartResponse{
|
||||
SessionID: remoteStartReq.SessionID,
|
||||
Status: "accepted",
|
||||
HWAccel: "none",
|
||||
})
|
||||
}))
|
||||
defer remote.Close()
|
||||
|
||||
handler := NewPlaybackHandler(sessionMgr, testPlaybackFileResolver{file: file})
|
||||
handler.ItemAccess = allowAllPlaybackItemAccess{}
|
||||
handler.JWTSecret = "test-secret"
|
||||
|
||||
group := "rack-a"
|
||||
proxies := nodepool.NewProxyPool()
|
||||
proxies.SetNodes([]*nodepool.Node{{
|
||||
ID: 1, Name: "proxy-1", Type: nodepool.NodeTypeProxy, URL: "http://proxy-1",
|
||||
Enabled: true, Healthy: true, Group: &group,
|
||||
}})
|
||||
transcodes := nodepool.NewTranscodePool()
|
||||
transcodes.SetNodes([]*nodepool.Node{{
|
||||
ID: 2, Name: "transcode-1", Type: nodepool.NodeTypeTranscode, URL: remote.URL,
|
||||
Enabled: true, Healthy: true, Group: &group,
|
||||
}})
|
||||
handler.NodePlanner = nodepool.NewPlanner(proxies, transcodes)
|
||||
|
||||
sessionID := newRemoteAudioSwitchSession(t, sessionMgr, file, remote.URL)
|
||||
|
||||
changeReq := httptest.NewRequest(
|
||||
http.MethodPatch,
|
||||
"/api/v1/playback/"+sessionID+"/audio",
|
||||
strings.NewReader(`{"audio_track_index":1,"position":120}`),
|
||||
)
|
||||
changeReq = changeReq.WithContext(newAuthorizedPlaybackContext())
|
||||
changeReq = withPlaybackRouteParam(changeReq, "session_id", sessionID)
|
||||
|
||||
rr := httptest.NewRecorder()
|
||||
handler.HandleChangeAudioTrack(rr, changeReq)
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("change status = %d, body = %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
|
||||
// BUG A: the node must have been told to restart on the new track.
|
||||
if remoteStartRequests != 1 {
|
||||
t.Fatalf("remote /transcode/start calls = %d, want 1", remoteStartRequests)
|
||||
}
|
||||
if remoteStartReq.AudioTrackIndex != 1 {
|
||||
t.Fatalf("remote AudioTrackIndex = %d, want 1", remoteStartReq.AudioTrackIndex)
|
||||
}
|
||||
if remoteStartReq.SessionID != sessionID {
|
||||
t.Fatalf("remote SessionID = %q, want %q", remoteStartReq.SessionID, sessionID)
|
||||
}
|
||||
// Seek 120s with the node-default 2s segments => start segment 60.
|
||||
if remoteStartReq.StartSegmentNumber != 60 {
|
||||
t.Fatalf("remote StartSegmentNumber = %d, want 60", remoteStartReq.StartSegmentNumber)
|
||||
}
|
||||
if remoteStartReq.TargetResolution != "720p" || remoteStartReq.TargetCodecVideo != "h264" {
|
||||
t.Fatalf("remote target recipe = %q/%q, want 720p/h264", remoteStartReq.TargetResolution, remoteStartReq.TargetCodecVideo)
|
||||
}
|
||||
|
||||
var resp changeAudioResponse
|
||||
if err := json.NewDecoder(rr.Body).Decode(&resp); err != nil {
|
||||
t.Fatalf("decode change response: %v", err)
|
||||
}
|
||||
|
||||
// BUG B: the returned proxy URL token must be recipe-COMPLETE.
|
||||
prefix := "http://proxy-1/stream/transcode/"
|
||||
suffix := "/master.m3u8"
|
||||
if !strings.HasPrefix(resp.StreamURL, prefix) || !strings.HasSuffix(resp.StreamURL, suffix) {
|
||||
t.Fatalf("stream URL = %q, want proxy transcode manifest URL", resp.StreamURL)
|
||||
}
|
||||
token := strings.TrimSuffix(strings.TrimPrefix(resp.StreamURL, prefix), suffix)
|
||||
claims, err := streamtoken.Verify(token, handler.JWTSecret)
|
||||
if err != nil {
|
||||
t.Fatalf("verify stream token: %v", err)
|
||||
}
|
||||
if claims.AudioTrackIndex != 1 {
|
||||
t.Fatalf("token AudioTrackIndex = %d, want 1", claims.AudioTrackIndex)
|
||||
}
|
||||
// Identity-only claims would leave these zero; a full recipe carries them.
|
||||
if claims.TargetCodec != "h264" {
|
||||
t.Fatalf("token TargetCodec (video) = %q, want h264 (recipe-complete)", claims.TargetCodec)
|
||||
}
|
||||
if claims.TargetCodecAudio != "aac" {
|
||||
t.Fatalf("token TargetCodecAudio = %q, want aac (recipe-complete)", claims.TargetCodecAudio)
|
||||
}
|
||||
if claims.TargetRes != "720p" {
|
||||
t.Fatalf("token TargetRes = %q, want 720p (recipe-complete)", claims.TargetRes)
|
||||
}
|
||||
if claims.TargetBitrateKbps != 2000 {
|
||||
t.Fatalf("token TargetBitrateKbps = %d, want 2000 (recipe-complete)", claims.TargetBitrateKbps)
|
||||
}
|
||||
if claims.SeekSeconds != 120 {
|
||||
t.Fatalf("token SeekSeconds = %v, want 120", claims.SeekSeconds)
|
||||
}
|
||||
if claims.StartSegmentNumber != 60 {
|
||||
t.Fatalf("token StartSegmentNumber = %d, want 60", claims.StartSegmentNumber)
|
||||
}
|
||||
// SegmentDuration must be > 0: the node treats a zero/omitted value as an
|
||||
// incomplete token and falls back to a recipe store the native path never
|
||||
// populates, which would 404 on a node restart. This guards that regression.
|
||||
if claims.SegmentDuration != playback.DefaultSegmentDuration {
|
||||
t.Fatalf("token SegmentDuration = %d, want %d (recipe-complete)", claims.SegmentDuration, playback.DefaultSegmentDuration)
|
||||
}
|
||||
if claims.MediaPath != file.FilePath {
|
||||
t.Fatalf("token MediaPath = %q, want %q", claims.MediaPath, file.FilePath)
|
||||
}
|
||||
if claims.TranscodeNode != remote.URL {
|
||||
t.Fatalf("token TranscodeNode = %q, want %q", claims.TranscodeNode, remote.URL)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleChangeAudioTrack_RemoteTranscodeNodeUnreachableSurfacesError
|
||||
// verifies the chosen error posture: when the transcode node cannot be reached
|
||||
// for the restart, the handler surfaces a 502 rather than returning a 200 that
|
||||
// clients (silo-android / silo-apple) would trust as "audio switched".
|
||||
func TestHandleChangeAudioTrack_RemoteTranscodeNodeUnreachableSurfacesError(t *testing.T) {
|
||||
sessionMgr := playback.NewSessionManager(0, 0)
|
||||
file := &models.MediaFile{
|
||||
ID: 42,
|
||||
ContentID: "movie-1",
|
||||
FilePath: writePlaybackTestMediaFile(t, "movie.mkv"),
|
||||
Resolution: "1080p",
|
||||
CodecVideo: "h264",
|
||||
CodecAudio: "aac",
|
||||
Container: "mkv",
|
||||
Bitrate: 8000,
|
||||
Duration: 3600,
|
||||
AudioTracks: []models.AudioTrack{
|
||||
{Codec: "aac", Default: true},
|
||||
{Codec: "ac3"},
|
||||
},
|
||||
}
|
||||
|
||||
// A node URL that nothing is listening on => connection refused.
|
||||
deadNodeURL := "http://127.0.0.1:1"
|
||||
|
||||
handler := NewPlaybackHandler(sessionMgr, testPlaybackFileResolver{file: file})
|
||||
handler.ItemAccess = allowAllPlaybackItemAccess{}
|
||||
handler.JWTSecret = "test-secret"
|
||||
|
||||
group := "rack-a"
|
||||
proxies := nodepool.NewProxyPool()
|
||||
proxies.SetNodes([]*nodepool.Node{{
|
||||
ID: 1, Name: "proxy-1", Type: nodepool.NodeTypeProxy, URL: "http://proxy-1",
|
||||
Enabled: true, Healthy: true, Group: &group,
|
||||
}})
|
||||
transcodes := nodepool.NewTranscodePool()
|
||||
transcodes.SetNodes([]*nodepool.Node{{
|
||||
ID: 2, Name: "transcode-1", Type: nodepool.NodeTypeTranscode, URL: deadNodeURL,
|
||||
Enabled: true, Healthy: true, Group: &group,
|
||||
}})
|
||||
handler.NodePlanner = nodepool.NewPlanner(proxies, transcodes)
|
||||
|
||||
sessionID := newRemoteAudioSwitchSession(t, sessionMgr, file, deadNodeURL)
|
||||
|
||||
changeReq := httptest.NewRequest(
|
||||
http.MethodPatch,
|
||||
"/api/v1/playback/"+sessionID+"/audio",
|
||||
strings.NewReader(`{"audio_track_index":1,"position":0}`),
|
||||
)
|
||||
changeReq = changeReq.WithContext(newAuthorizedPlaybackContext())
|
||||
changeReq = withPlaybackRouteParam(changeReq, "session_id", sessionID)
|
||||
|
||||
rr := httptest.NewRecorder()
|
||||
handler.HandleChangeAudioTrack(rr, changeReq)
|
||||
|
||||
// Chosen posture: surface the failure (mirror HandleStartTranscode) so a 200
|
||||
// never lies to the client about an audio switch that didn't happen.
|
||||
if rr.Code != http.StatusBadGateway {
|
||||
t.Fatalf("change status = %d, want %d, body = %s", rr.Code, http.StatusBadGateway, rr.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleStartTranscode_LocalPathPropagatesSelectedAudioTrack(t *testing.T) {
|
||||
sessionMgr := playback.NewSessionManager(0, 0)
|
||||
filePath := writePlaybackTestMediaFile(t, "movie.mkv")
|
||||
@@ -1213,7 +1452,7 @@ func TestHandleStartTranscode_LocalPathPropagatesSelectedAudioTrack(t *testing.T
|
||||
t.Fatalf("transcode status = %d, body = %s", transcodeRR.Code, transcodeRR.Body.String())
|
||||
}
|
||||
|
||||
transcodeSession := handler.getTranscodeSession(startResp.SessionID)
|
||||
transcodeSession := handler.tm.GetTranscodeSession(startResp.SessionID)
|
||||
if transcodeSession == nil {
|
||||
t.Fatal("expected local transcode session")
|
||||
}
|
||||
@@ -1264,7 +1503,7 @@ func TestHandleStartTranscode_MPEG2SeekedCopyRemainsCopyVideo(t *testing.T) {
|
||||
t.Fatalf("transcode status = %d, body = %s", transcodeRR.Code, transcodeRR.Body.String())
|
||||
}
|
||||
|
||||
transcodeSession := handler.getTranscodeSession(session.ID)
|
||||
transcodeSession := handler.tm.GetTranscodeSession(session.ID)
|
||||
if transcodeSession == nil {
|
||||
t.Fatal("expected local transcode session")
|
||||
}
|
||||
|
||||
@@ -28,6 +28,12 @@ type ProfileHandler struct {
|
||||
ProfileTokens *access.ProfileTokenService
|
||||
AvatarStore profileAvatarStore
|
||||
AvatarTTL time.Duration
|
||||
// DeviceLibraryPurger removes a deleted profile's device rows (and, via
|
||||
// cascade, its managed downloads and subscriptions). Profiles may live
|
||||
// outside Postgres, so no FK cascade covers these shared tables.
|
||||
DeviceLibraryPurger interface {
|
||||
PurgeProfileDevices(ctx context.Context, userID int, profileID string) error
|
||||
}
|
||||
}
|
||||
|
||||
// NewProfileHandler creates a new ProfileHandler.
|
||||
@@ -589,6 +595,13 @@ func (h *ProfileHandler) HandleDeleteProfile(w http.ResponseWriter, r *http.Requ
|
||||
slog.Warn("profile avatar cleanup failed after delete", "user_id", userID, "profile_id", profileID, "error", cleanupErr)
|
||||
}
|
||||
}
|
||||
if h.DeviceLibraryPurger != nil {
|
||||
purgeCtx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), 30*time.Second)
|
||||
defer cancel()
|
||||
if purgeErr := h.DeviceLibraryPurger.PurgeProfileDevices(purgeCtx, userID, profileID); purgeErr != nil {
|
||||
slog.Warn("profile device-library purge failed after delete", "user_id", userID, "profile_id", profileID, "error", purgeErr)
|
||||
}
|
||||
}
|
||||
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
@@ -3,8 +3,10 @@ package handlers
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/access"
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
@@ -12,6 +14,34 @@ import (
|
||||
"github.com/Silo-Server/silo-server/internal/userstore"
|
||||
)
|
||||
|
||||
// progressClockSkew bounds how far ahead of server time a client-supplied
|
||||
// progress event time may sit before it is clamped to "now".
|
||||
const progressClockSkew = 2 * time.Minute
|
||||
|
||||
// parseClientEventTime parses an RFC3339 client event time. Malformed values
|
||||
// are an error the caller must reject: treating them as "now" would let a
|
||||
// stale offline event win LWW as a fresh server-time write.
|
||||
func parseClientEventTime(s string) (time.Time, error) {
|
||||
t, err := time.Parse(time.RFC3339, s)
|
||||
if err != nil {
|
||||
return time.Time{}, err
|
||||
}
|
||||
return t.UTC(), nil
|
||||
}
|
||||
|
||||
// clampEventAt bounds a client event time to at most now+skew: a value past the
|
||||
// window is clamped to now, so a skewed or malicious clock can at most claim
|
||||
// "now" for its own profile and never lock in a far-future LWW win (invariant 1).
|
||||
func clampEventAt(client, now time.Time) time.Time {
|
||||
if client.IsZero() {
|
||||
return now
|
||||
}
|
||||
if client.After(now.Add(progressClockSkew)) {
|
||||
return now
|
||||
}
|
||||
return client
|
||||
}
|
||||
|
||||
// ProgressLibraryLookup resolves which progress items belong to a library.
|
||||
type ProgressLibraryLookup interface {
|
||||
GetItemsInFolder(ctx context.Context, contentIDs []string, folderID int) (map[string]bool, error)
|
||||
@@ -57,6 +87,8 @@ type progressEntryResponse struct {
|
||||
|
||||
type progressListResponse struct {
|
||||
Progress []progressEntryResponse `json:"progress"`
|
||||
// NextCursor is the opaque server token to resume a ?since= delta from.
|
||||
NextCursor string `json:"next_cursor,omitempty"`
|
||||
}
|
||||
|
||||
type syncProgressItem struct {
|
||||
@@ -64,6 +96,9 @@ type syncProgressItem struct {
|
||||
Position float64 `json:"position"`
|
||||
Duration float64 `json:"duration"`
|
||||
ForceOverwrite bool `json:"force_overwrite"`
|
||||
// UpdatedAt is the client EVENT time (RFC3339) for an offline-queued item.
|
||||
// The server clamps it to now+skew and uses it only as the LWW key.
|
||||
UpdatedAt *string `json:"updated_at,omitempty"`
|
||||
}
|
||||
|
||||
type syncProgressRequest struct {
|
||||
@@ -94,6 +129,7 @@ func (h *ProgressHandler) HandleListProgress(w http.ResponseWriter, r *http.Requ
|
||||
}
|
||||
|
||||
status := r.URL.Query().Get("status")
|
||||
since := r.URL.Query().Get("since")
|
||||
limit, offset := parsePagination(r)
|
||||
libraryID, err := parseLibraryIDParam(r)
|
||||
if err != nil {
|
||||
@@ -101,7 +137,16 @@ func (h *ProgressHandler) HandleListProgress(w http.ResponseWriter, r *http.Requ
|
||||
return
|
||||
}
|
||||
|
||||
entries, err := store.ListProgress(r.Context(), profileID, status, limit, offset)
|
||||
// A ?since= cursor switches to server-ordered delta delivery (rows changed
|
||||
// elsewhere since the cursor), immune to client clock skew. Absent since →
|
||||
// today's status/pagination listing.
|
||||
var entries []userstore.WatchProgress
|
||||
var nextCursor string
|
||||
if since != "" {
|
||||
entries, nextCursor, err = store.ListProgressSince(r.Context(), profileID, since)
|
||||
} else {
|
||||
entries, err = store.ListProgress(r.Context(), profileID, status, limit, offset)
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to list progress")
|
||||
return
|
||||
@@ -138,7 +183,8 @@ func (h *ProgressHandler) HandleListProgress(w http.ResponseWriter, r *http.Requ
|
||||
}
|
||||
|
||||
resp := progressListResponse{
|
||||
Progress: make([]progressEntryResponse, 0, len(entries)),
|
||||
Progress: make([]progressEntryResponse, 0, len(entries)),
|
||||
NextCursor: nextCursor,
|
||||
}
|
||||
for _, e := range entries {
|
||||
resp.Progress = append(resp.Progress, progressEntryResponse{
|
||||
@@ -280,9 +326,32 @@ func (h *ProgressHandler) HandleSyncProgress(w http.ResponseWriter, r *http.Requ
|
||||
}
|
||||
|
||||
var updateErr error
|
||||
if item.ForceOverwrite {
|
||||
switch {
|
||||
case item.UpdatedAt != nil:
|
||||
// Offline-queued event: clamp the client event time and merge
|
||||
// last-write-wins on the bounded event_at. synced_seq (the cursor) is
|
||||
// stamped server-side; completion still comes from the threshold logic,
|
||||
// never the timestamp alone.
|
||||
client, parseErr := parseClientEventTime(*item.UpdatedAt)
|
||||
if parseErr != nil {
|
||||
result.Status = "error"
|
||||
result.Error = "updated_at must be RFC3339"
|
||||
results = append(results, result)
|
||||
continue
|
||||
}
|
||||
now := time.Now()
|
||||
eventAt := clampEventAt(client, now)
|
||||
if !client.IsZero() && client.After(now.Add(progressClockSkew)) {
|
||||
slog.Warn("clamped future-dated progress event time",
|
||||
"profile_id", profileID, "media_item_id", item.MediaItemID)
|
||||
}
|
||||
pos, completed, skip := userstore.ResolveProgressState(item.Position, item.Duration, thresholds)
|
||||
if !skip {
|
||||
_, updateErr = store.SetProgressIfNewer(r.Context(), profileID, item.MediaItemID, pos, item.Duration, completed, eventAt)
|
||||
}
|
||||
case item.ForceOverwrite:
|
||||
updateErr = store.SetProgress(r.Context(), profileID, item.MediaItemID, item.Position, item.Duration, thresholds)
|
||||
} else {
|
||||
default:
|
||||
updateErr = store.UpdateProgress(r.Context(), profileID, item.MediaItemID, item.Position, item.Duration, thresholds)
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/userstore"
|
||||
)
|
||||
|
||||
func TestClampEventAt(t *testing.T) {
|
||||
now := time.Date(2026, 6, 19, 12, 0, 0, 0, time.UTC)
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
client time.Time
|
||||
want time.Time
|
||||
}{
|
||||
{"zero defaults to now", time.Time{}, now},
|
||||
{"far future clamps to now", now.Add(48 * time.Hour), now},
|
||||
{"just past skew clamps to now", now.Add(progressClockSkew + time.Minute), now},
|
||||
{"within skew is kept", now.Add(time.Minute), now.Add(time.Minute)},
|
||||
{"past is kept", now.Add(-3 * time.Hour), now.Add(-3 * time.Hour)},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := clampEventAt(tc.client, now); !got.Equal(tc.want) {
|
||||
t.Fatalf("clampEventAt(%v) = %v, want %v", tc.client, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestOfflineProgressState(t *testing.T) {
|
||||
thresholds := userstore.ProgressThresholds{WatchedPct: 90, MinResumePct: 5}
|
||||
|
||||
// Below the min-resume floor → skipped.
|
||||
if _, _, skip := userstore.ResolveProgressState(10, 1000, thresholds); !skip {
|
||||
t.Fatal("tiny progress should be skipped")
|
||||
}
|
||||
// Above the watched threshold → completed latch, position reset.
|
||||
pos, completed, skip := userstore.ResolveProgressState(950, 1000, thresholds)
|
||||
if skip || !completed || pos != 0 {
|
||||
t.Fatalf("watched item = (pos=%v completed=%v skip=%v), want (0 true false)", pos, completed, skip)
|
||||
}
|
||||
// Normal mid-progress → kept as-is.
|
||||
pos, completed, skip = userstore.ResolveProgressState(500, 1000, thresholds)
|
||||
if skip || completed || pos != 500 {
|
||||
t.Fatalf("mid progress = (pos=%v completed=%v skip=%v), want (500 false false)", pos, completed, skip)
|
||||
}
|
||||
}
|
||||
@@ -182,7 +182,7 @@ func (h *PlaybackHandler) handleRealtimeClientMessage(sessionID string, data []b
|
||||
}
|
||||
switch record.Name {
|
||||
case playback.CommandStop, playback.CommandTerminate:
|
||||
err := h.stopPlaybackSessionByID(context.Background(), sessionID)
|
||||
err := h.stopPlaybackSessionByID(context.Background(), sessionID, true)
|
||||
if err != nil && !errors.Is(err, playback.ErrSessionNotFound) {
|
||||
slog.Error("failed to stop playback after realtime completion", "session", sessionID, "playback_session_id", sessionID, "error", err)
|
||||
}
|
||||
|
||||
@@ -32,6 +32,15 @@ type StreamHandler struct {
|
||||
EventsHub *evt.Hub
|
||||
AdminStore PlaybackAdminStore
|
||||
SessionSyncer PlaybackSessionSyncer
|
||||
// TM is the shared transcode/reconstruct manager (same instance as the
|
||||
// PlaybackHandler's). It lets a direct/remux stream rebuild its playback
|
||||
// Session from the recipe card after a server restart instead of 404-ing.
|
||||
// May be nil (tests / minimal setups) — reconstruct is then simply off.
|
||||
TM *playback.TranscodeManager
|
||||
// JWTSecret verifies the stream token carried on the serve URL (?st=), which
|
||||
// is the reconstruction descriptor for direct/remux after a restart. Empty
|
||||
// disables token-based reconstruct (tests / minimal setups).
|
||||
JWTSecret string
|
||||
// PlaybackConfig returns the current playback config; read it through
|
||||
// ffmpegPath(). May be nil (tests).
|
||||
PlaybackConfig func() config.PlaybackConfig
|
||||
@@ -54,6 +63,10 @@ func NewStreamHandler(sessionMgr SessionManagerInterface, fileResolver FilePathR
|
||||
return &StreamHandler{
|
||||
sessionMgr: sessionMgr,
|
||||
fileResolver: fileResolver,
|
||||
// A bare manager (no recipe store) behaves as "no reconstruct" — plain
|
||||
// GetSession + ownership — so HandleStream has a single code path. The
|
||||
// router overwrites this with the shared manager to enable reconstruct.
|
||||
TM: playback.NewTranscodeManager(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -75,14 +88,22 @@ func (h *StreamHandler) HandleStream(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
setPlaybackSessionLogContext(r, sessionID)
|
||||
|
||||
session, err := h.sessionMgr.GetSession(sessionID)
|
||||
if err != nil {
|
||||
// Look up the session, reconstructing it from the recipe card on a not-found
|
||||
// miss (e.g. after a server restart) so a direct/remux stream resumes instead
|
||||
// of 404-ing. The client re-supplies its position (HTTP Range for direct, the
|
||||
// ?seek= query for remux), so no runtime beyond the Session needs rebuilding.
|
||||
// Without a token (or signing secret) reconstruct is off, collapsing to a
|
||||
// plain GetSession + ownership check.
|
||||
card := streamCardFromToken(r.URL.Query().Get(streamTokenParam), sessionID, h.JWTSecret)
|
||||
session, status := h.TM.LoadOrReconstructSession(r.Context(), h.sessionMgr.GetSession, sessionID, userID, card)
|
||||
switch status {
|
||||
case playback.SessionMissing:
|
||||
writePlaybackSessionNotFound(w)
|
||||
return
|
||||
}
|
||||
|
||||
// Verify session ownership.
|
||||
if session.UserID != userID {
|
||||
case playback.SessionLoadFailed:
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to load playback session")
|
||||
return
|
||||
case playback.SessionForbidden:
|
||||
writeError(w, http.StatusForbidden, "forbidden", "Session belongs to another user")
|
||||
return
|
||||
}
|
||||
@@ -240,37 +261,49 @@ func (h *StreamHandler) HandleSubtitle(w http.ResponseWriter, r *http.Request) {
|
||||
// Check downloaded subtitles (from S3).
|
||||
if h.SubtitleRepo != nil && h.S3Client != nil {
|
||||
downloaded, err := h.SubtitleRepo.ListDownloadedSubtitles(r.Context(), file.ID)
|
||||
if err == nil {
|
||||
downloadedIndex := embeddedIndex - len(file.SubtitleTracks)
|
||||
if downloadedIndex >= 0 && downloadedIndex < len(downloaded) {
|
||||
dl := downloaded[downloadedIndex]
|
||||
data, err := h.S3Client.GetObject(r.Context(), h.S3Bucket, dl.S3Key)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, "s3_error", "Failed to load subtitle from storage")
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
// A DB failure here must not masquerade as "track not found":
|
||||
// surface it as an internal error (with a server-side signal)
|
||||
// so the real failure is diagnosable instead of looking like an
|
||||
// intermittent 404 to the client.
|
||||
slog.ErrorContext(r.Context(), "list downloaded subtitles failed",
|
||||
"file_id", file.ID,
|
||||
"track", trackIndex,
|
||||
"error", err,
|
||||
)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to list downloaded subtitles")
|
||||
return
|
||||
}
|
||||
|
||||
// Serve ASS/SSA downloaded subtitles as raw data.
|
||||
if playback.IsASS(string(dl.Format)) {
|
||||
playback.ServeSubtitle(w, data, "ass")
|
||||
return
|
||||
}
|
||||
|
||||
// If the subtitle is already VTT, serve directly.
|
||||
if dl.Format == subtitles.FormatVTT {
|
||||
playback.ServeSubtitle(w, data, "vtt")
|
||||
return
|
||||
}
|
||||
|
||||
// Convert to VTT using the playback conversion pipeline.
|
||||
vttData, err := playback.ConvertToVTT(data, string(dl.Format))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "convert_error", "Failed to convert subtitle")
|
||||
return
|
||||
}
|
||||
playback.ServeSubtitle(w, vttData, "vtt")
|
||||
downloadedIndex := embeddedIndex - len(file.SubtitleTracks)
|
||||
if downloadedIndex >= 0 && downloadedIndex < len(downloaded) {
|
||||
dl := downloaded[downloadedIndex]
|
||||
data, err := h.S3Client.GetObject(r.Context(), h.S3Bucket, dl.S3Key)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, "s3_error", "Failed to load subtitle from storage")
|
||||
return
|
||||
}
|
||||
|
||||
// Serve ASS/SSA downloaded subtitles as raw data.
|
||||
if playback.IsASS(string(dl.Format)) {
|
||||
playback.ServeSubtitle(w, data, "ass")
|
||||
return
|
||||
}
|
||||
|
||||
// If the subtitle is already VTT, serve directly.
|
||||
if dl.Format == subtitles.FormatVTT {
|
||||
playback.ServeSubtitle(w, data, "vtt")
|
||||
return
|
||||
}
|
||||
|
||||
// Convert to VTT using the playback conversion pipeline.
|
||||
vttData, err := playback.ConvertToVTT(data, string(dl.Format))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "convert_error", "Failed to convert subtitle")
|
||||
return
|
||||
}
|
||||
playback.ServeSubtitle(w, vttData, "vtt")
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -2,12 +2,15 @@ package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
"github.com/Silo-Server/silo-server/internal/playback"
|
||||
)
|
||||
@@ -121,6 +124,55 @@ func TestHandleStream_KeepsSessionWhenLookupFailsForNonMissingReason(t *testing.
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleSubtitle_ListDownloadedSubtitlesErrorReturns500 pins the fix for
|
||||
// issue #248: a failure listing downloaded subtitles must surface as a 500 with
|
||||
// an "internal_error" code, not be swallowed and reported to the client as a
|
||||
// generic "Subtitle track not found" 404 (which made a real backing-store
|
||||
// failure look like an intermittent client-side subtitle bug).
|
||||
func TestHandleSubtitle_ListDownloadedSubtitlesErrorReturns500(t *testing.T) {
|
||||
// No external or embedded tracks, so track index 0 falls through to the
|
||||
// downloaded-subtitle branch that queries the repository.
|
||||
file := &models.MediaFile{
|
||||
ID: 42,
|
||||
ContentID: "movie-1",
|
||||
FilePath: "/tmp/movie.mkv",
|
||||
Duration: 3600,
|
||||
}
|
||||
baseMgr := playback.NewSessionManager(0, 0)
|
||||
session, err := baseMgr.StartSession(1, "profile-1", 42, playback.PlayDirect, false)
|
||||
if err != nil {
|
||||
t.Fatalf("StartSession: %v", err)
|
||||
}
|
||||
|
||||
handler := NewStreamHandler(baseMgr, testPlaybackFileResolver{file: file})
|
||||
handler.SubtitleRepo = &handlerMockSubtitleRepo{listErr: errors.New("db unavailable")}
|
||||
handler.S3Client = newMockS3ClientForHandler()
|
||||
handler.S3Bucket = "test-bucket"
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/api/v1/stream/"+session.ID+"/subtitles/0.vtt", nil)
|
||||
req = req.WithContext(newAuthorizedPlaybackContext())
|
||||
routeCtx := chi.NewRouteContext()
|
||||
routeCtx.URLParams.Add("session_id", session.ID)
|
||||
routeCtx.URLParams.Add("track", "0.vtt")
|
||||
req = req.WithContext(context.WithValue(req.Context(), chi.RouteCtxKey, routeCtx))
|
||||
|
||||
rr := httptest.NewRecorder()
|
||||
handler.HandleSubtitle(rr, req)
|
||||
|
||||
if rr.Code != http.StatusInternalServerError {
|
||||
t.Fatalf("status = %d, body = %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
var body struct {
|
||||
Error string `json:"error"`
|
||||
}
|
||||
if err := json.Unmarshal(rr.Body.Bytes(), &body); err != nil {
|
||||
t.Fatalf("decode error body: %v (body = %s)", err, rr.Body.String())
|
||||
}
|
||||
if body.Error != "internal_error" {
|
||||
t.Fatalf("error code = %q, want %q (body = %s)", body.Error, "internal_error", rr.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleTransportStartFailure_KeepsSessionForNonMissingError(t *testing.T) {
|
||||
filePath := writePlaybackTestMediaFile(t, "movie.mkv")
|
||||
file := &models.MediaFile{
|
||||
|
||||
@@ -239,6 +239,9 @@ type handlerMockSubtitleRepo struct {
|
||||
subtitles map[int]*subtitles.DownloadedSubtitle
|
||||
nextID int
|
||||
byKey map[string]*subtitles.DownloadedSubtitle
|
||||
// listErr, when set, is returned by ListDownloadedSubtitles to simulate a
|
||||
// backing-store failure.
|
||||
listErr error
|
||||
}
|
||||
|
||||
func newMockSubtitleRepoForHandler() *handlerMockSubtitleRepo {
|
||||
@@ -265,6 +268,9 @@ func (m *handlerMockSubtitleRepo) GetDownloadedSubtitle(_ context.Context, id in
|
||||
}
|
||||
|
||||
func (m *handlerMockSubtitleRepo) ListDownloadedSubtitles(context.Context, int) ([]subtitles.DownloadedSubtitle, error) {
|
||||
if m.listErr != nil {
|
||||
return nil, m.listErr
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"io"
|
||||
"net/http"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/access"
|
||||
@@ -110,9 +111,16 @@ type watchTogetherPingMessage struct {
|
||||
ClientSentAt string `json:"client_sent_at"`
|
||||
}
|
||||
|
||||
// watchTogetherRoomConn serializes every write to the underlying gorilla
|
||||
// connection. gorilla/websocket does not support concurrent writers, and room
|
||||
// broadcasts arrive from other members' goroutines, so all writes — including
|
||||
// pong/error replies from the read loop — must go through this wrapper.
|
||||
type watchTogetherRoomConn struct {
|
||||
conn *websocket.Conn
|
||||
writeMu sync.Mutex
|
||||
// pingSentAtNano is the send time of the most recent protocol-level ping,
|
||||
// used to measure round-trip latency when the pong arrives.
|
||||
pingSentAtNano atomic.Int64
|
||||
}
|
||||
|
||||
func (c *watchTogetherRoomConn) WriteJSON(v any) error {
|
||||
@@ -130,9 +138,35 @@ func (c *watchTogetherRoomConn) Close() error {
|
||||
func (c *watchTogetherRoomConn) WritePing() error {
|
||||
c.writeMu.Lock()
|
||||
defer c.writeMu.Unlock()
|
||||
c.pingSentAtNano.Store(time.Now().UnixNano())
|
||||
return writeWebSocketControl(c.conn, websocket.PingMessage, nil)
|
||||
}
|
||||
|
||||
// TakePingSentAt returns and clears the send time of the last unanswered
|
||||
// protocol-level ping.
|
||||
func (c *watchTogetherRoomConn) TakePingSentAt() time.Time {
|
||||
nano := c.pingSentAtNano.Swap(0)
|
||||
if nano == 0 {
|
||||
return time.Time{}
|
||||
}
|
||||
return time.Unix(0, nano)
|
||||
}
|
||||
|
||||
func (c *watchTogetherRoomConn) WriteError(code, message string) {
|
||||
_ = c.WriteJSON(map[string]string{
|
||||
"type": "error",
|
||||
"code": code,
|
||||
"message": message,
|
||||
})
|
||||
}
|
||||
|
||||
func (c *watchTogetherRoomConn) writeRoomClosed(reason string) {
|
||||
_ = c.WriteJSON(map[string]string{
|
||||
"type": "room_closed",
|
||||
"reason": reason,
|
||||
})
|
||||
}
|
||||
|
||||
func NewWatchTogetherHandler(
|
||||
service *watchtogether.Service,
|
||||
scopeResolver WatchTogetherScopeResolver,
|
||||
@@ -653,19 +687,33 @@ func (h *WatchTogetherHandler) HandleRoomWebSocket(w http.ResponseWriter, r *htt
|
||||
|
||||
reg, snapshot, err := h.Service.Connect(ctx, roomID, claims.UserID, profileID, realtimeConn)
|
||||
if err != nil {
|
||||
// Terminal failures use room_closed so clients stop reconnecting
|
||||
// instead of retrying a room that will never come back.
|
||||
if errors.Is(err, watchtogether.ErrRoomNotFound) {
|
||||
writeWebSocketError(conn, "not_found", "Room not found")
|
||||
realtimeConn.writeRoomClosed("not_found")
|
||||
} else if errors.Is(err, watchtogether.ErrRoomClosed) {
|
||||
writeWebSocketError(conn, "gone", "Room is no longer active")
|
||||
realtimeConn.writeRoomClosed("ended")
|
||||
} else {
|
||||
writeWebSocketError(conn, "internal_error", "Failed to connect room socket")
|
||||
realtimeConn.WriteError("internal_error", "Failed to connect room socket")
|
||||
}
|
||||
return
|
||||
}
|
||||
defer h.Service.Disconnect(reg, false)
|
||||
|
||||
configureWebSocket(conn)
|
||||
// Measure round-trip latency from protocol-level ping/pong on the server
|
||||
// clock; client-reported timestamps are subject to clock skew and cannot
|
||||
// be trusted for command scheduling.
|
||||
conn.SetPongHandler(func(string) error {
|
||||
_ = conn.SetReadDeadline(time.Now().Add(wsPingInterval + wsPongTimeout))
|
||||
if sentAt := realtimeConn.TakePingSentAt(); !sentAt.IsZero() {
|
||||
_ = h.Service.HandlePingForConnection(ctx, reg, claims.UserID, profileID, time.Since(sentAt).Milliseconds())
|
||||
}
|
||||
return nil
|
||||
})
|
||||
startWebSocketPingLoop(ctx, realtimeConn.WritePing)
|
||||
// Prime an RTT sample right away instead of waiting for the first tick.
|
||||
_ = realtimeConn.WritePing()
|
||||
|
||||
if err := realtimeConn.WriteJSON(map[string]any{
|
||||
"type": "snapshot",
|
||||
@@ -680,15 +728,15 @@ func (h *WatchTogetherHandler) HandleRoomWebSocket(w http.ResponseWriter, r *htt
|
||||
return
|
||||
}
|
||||
|
||||
if err := h.handleRoomClientMessage(ctx, conn, reg, claims.UserID, profileID, data); err != nil {
|
||||
writeWebSocketError(conn, "bad_request", err.Error())
|
||||
if err := h.handleRoomClientMessage(ctx, realtimeConn, reg, claims.UserID, profileID, data); err != nil {
|
||||
realtimeConn.WriteError("bad_request", err.Error())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (h *WatchTogetherHandler) handleRoomClientMessage(
|
||||
ctx context.Context,
|
||||
conn *websocket.Conn,
|
||||
rc *watchTogetherRoomConn,
|
||||
reg *watchtogether.Registration,
|
||||
userID int,
|
||||
profileID string,
|
||||
@@ -768,16 +816,11 @@ func (h *WatchTogetherHandler) handleRoomClientMessage(
|
||||
if err := json.Unmarshal(data, &msg); err != nil {
|
||||
return err
|
||||
}
|
||||
clientSentAt, err := time.Parse(time.RFC3339Nano, msg.ClientSentAt)
|
||||
if err != nil {
|
||||
return errors.New("client_sent_at must be RFC3339Nano")
|
||||
}
|
||||
// The echoed timestamps only serve the client's clock-offset estimate;
|
||||
// latency for command scheduling is measured server-side from
|
||||
// protocol-level ping/pong.
|
||||
now := time.Now().UTC()
|
||||
pingMS := now.Sub(clientSentAt.UTC()).Milliseconds()
|
||||
if pingMS > 0 {
|
||||
_ = h.Service.HandlePingForConnection(ctx, reg, userID, profileID, pingMS)
|
||||
}
|
||||
return writeWebSocketJSON(conn, map[string]string{
|
||||
return rc.WriteJSON(map[string]string{
|
||||
"type": "pong",
|
||||
"client_sent_at": msg.ClientSentAt,
|
||||
"server_received_at": now.Format(time.RFC3339Nano),
|
||||
|
||||
+80
-17
@@ -32,7 +32,7 @@ import (
|
||||
"github.com/Silo-Server/silo-server/internal/catalogseed"
|
||||
"github.com/Silo-Server/silo-server/internal/clientip"
|
||||
"github.com/Silo-Server/silo-server/internal/config"
|
||||
"github.com/Silo-Server/silo-server/internal/download"
|
||||
"github.com/Silo-Server/silo-server/internal/downloads"
|
||||
evt "github.com/Silo-Server/silo-server/internal/events"
|
||||
"github.com/Silo-Server/silo-server/internal/historyimport"
|
||||
"github.com/Silo-Server/silo-server/internal/intromarkers"
|
||||
@@ -132,8 +132,9 @@ type Dependencies struct {
|
||||
ActivityLogRepo *activitylog.Repo
|
||||
OpsLogRepo *opslog.Repo
|
||||
FFmpegLogSink playback.FFmpegLogSink
|
||||
RedisClient *redis.Client // for session listing (may be nil)
|
||||
TaskManager *taskmanager.TaskManager // task manager (may be nil)
|
||||
RedisClient *redis.Client // for session listing (may be nil)
|
||||
TaskManager *taskmanager.TaskManager // task manager (may be nil)
|
||||
ArtifactManager *downloads.ArtifactManager // download prepare-to-file pipeline (may be nil)
|
||||
AdminJobCancelRegistry *adminjob.CancelRegistry
|
||||
IntroRepository *intromarkers.Repository
|
||||
IntroAnalyzer *intromarkers.Analyzer
|
||||
@@ -143,6 +144,7 @@ type Dependencies struct {
|
||||
MarkerContributionStore *markers.ContributionStore
|
||||
MarkerContributionService *markers.ContributionService
|
||||
WatchProviderService handlers.WatchProviderService
|
||||
WatchCompletionObserver watchstate.CompletionObserver
|
||||
PluginService *plugins.Service
|
||||
PluginHTTPProxy *plugins.HTTPProxy
|
||||
PluginUserConfig *plugins.UserConfigStore
|
||||
@@ -448,14 +450,10 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
ebookAnnotationStore = handlers.NewPGEbookReaderAnnotationStore(deps.DB)
|
||||
browseRepo := catalog.NewBrowseRepository(deps.DB)
|
||||
itemRepo = catalog.NewItemRepository(deps.DB)
|
||||
catalogSearchSettings, err := catalog.LoadCatalogSearchSettings(context.Background(), settingsRepo)
|
||||
if err != nil {
|
||||
slog.Warn("catalog search: failed to load settings; using postgres", "err", err)
|
||||
catalogSearchSettings = catalog.DefaultCatalogSearchSettings()
|
||||
}
|
||||
searchIndexEvents := catalog.NewSearchIndexEventRepository(deps.DB)
|
||||
catalogSearchService = catalog.NewCatalogSearchServiceFromSettings(
|
||||
catalogSearchSettings,
|
||||
catalogSearchService = catalog.NewCatalogSearchService(
|
||||
context.Background(),
|
||||
settingsRepo,
|
||||
itemRepo,
|
||||
searchIndexEvents,
|
||||
deps.CatalogSearchVectorizer,
|
||||
@@ -523,6 +521,9 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
if dispatcher, ok := deps.WatchProviderService.(handlers.LocalWatchEventDispatcher); ok {
|
||||
itemsHandler.SetLocalWatchEventDispatcher(dispatcher)
|
||||
}
|
||||
if deps.WatchCompletionObserver != nil {
|
||||
itemsHandler.SetCompletionObserver(deps.WatchCompletionObserver)
|
||||
}
|
||||
if ebookProgressStore != nil {
|
||||
itemsHandler.SetEbookReaderProgressStore(ebookProgressStore)
|
||||
}
|
||||
@@ -638,8 +639,8 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
personalDataHandler.SetEpisodeRepo(episodeRepo)
|
||||
personalDataHandler.SetSeasonRepo(seasonRepo)
|
||||
personalDataHandler.EventsHub = deps.EventsHub
|
||||
if dispatcher, ok := deps.WatchProviderService.(handlers.LocalFavoriteEventDispatcher); ok {
|
||||
personalDataHandler.SetLocalFavoriteEventDispatcher(dispatcher)
|
||||
if dispatcher, ok := deps.WatchProviderService.(handlers.LocalListEventDispatcher); ok {
|
||||
personalDataHandler.SetLocalListEventDispatcher(dispatcher)
|
||||
}
|
||||
progressHandler = handlers.NewProgressHandler(deps.UserStoreProvider)
|
||||
progressHandler.EventsHub = deps.EventsHub
|
||||
@@ -734,6 +735,7 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
playbackHandler.StoreProvider = deps.UserStoreProvider
|
||||
}
|
||||
playbackHandler.StableIdentityResolver = watchstate.NewStableIdentityResolver(itemRepo, episodeRepo, providerIDRepo)
|
||||
playbackHandler.CompletionObserver = deps.WatchCompletionObserver
|
||||
if scrobbler, ok := deps.WatchProviderService.(handlers.PlaybackWatchScrobbler); ok {
|
||||
playbackHandler.WatchScrobbler = scrobbler
|
||||
}
|
||||
@@ -746,6 +748,13 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
playbackHandler.SessionSyncer = deps.SessionSyncer
|
||||
}
|
||||
if streamHandler != nil {
|
||||
// Share the playback handler's transcode/reconstruct manager so a
|
||||
// direct/remux stream can rebuild its session from the token recipe
|
||||
// after a restart (same manager, same SessionManager).
|
||||
streamHandler.TM = playbackHandler.TranscodeManager()
|
||||
if deps.Config != nil {
|
||||
streamHandler.JWTSecret = deps.Config.Auth.JWTSecret
|
||||
}
|
||||
streamHandler.AdminStore = playbackAdminStore
|
||||
streamHandler.EventsHub = deps.EventsHub
|
||||
streamHandler.SessionSyncer = deps.SessionSyncer
|
||||
@@ -814,9 +823,9 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
watchtogether.NewRepository(deps.DB),
|
||||
deps.SessionMgr,
|
||||
deps.FileRepo,
|
||||
playbackHandler.CommandDispatcher,
|
||||
watchtogether.NewCatalogSelectionResolver(detailSvc),
|
||||
watchtogether.NewSuggestionRepository(deps.DB),
|
||||
watchtogether.NewProfileNameResolver(deps.UserStoreProvider),
|
||||
),
|
||||
viewerResolver,
|
||||
roomTokenService,
|
||||
@@ -1348,18 +1357,18 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
// Build download handler.
|
||||
var downloadHandler *handlers.DownloadHandler
|
||||
if deps.DB != nil && deps.FileRepo != nil && deps.Config != nil {
|
||||
downloadRepo := download.NewRepository(deps.DB)
|
||||
downloadBandwidth := download.NewBandwidthManager(
|
||||
downloadRepo := downloads.NewRepository(deps.DB)
|
||||
downloadBandwidth := downloads.NewBandwidthManager(
|
||||
deps.Config.Download.ServerBandwidthBPS,
|
||||
deps.Config.Download.UserBandwidthBPS,
|
||||
)
|
||||
downloadLimiter := download.NewQuantityLimiter(
|
||||
downloadLimiter := downloads.NewQuantityLimiter(
|
||||
downloadRepo,
|
||||
deps.Config.Download.MaxConcurrentPerUser,
|
||||
deps.Config.Download.MaxPerPeriod,
|
||||
deps.Config.Download.PeriodDuration,
|
||||
)
|
||||
downloadSvc := download.NewService(
|
||||
downloadSvc := downloads.NewService(
|
||||
downloadRepo,
|
||||
downloadBandwidth,
|
||||
downloadLimiter,
|
||||
@@ -1371,7 +1380,31 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
settingsRepo,
|
||||
&deps.Config.Download,
|
||||
)
|
||||
if detailSvc != nil {
|
||||
// Offline manifest + artwork/subtitle proxies (Phase 2). subtitleManager
|
||||
// may be nil when subtitles are unconfigured; pass a nil interface so the
|
||||
// downloaded-subtitle path reports unavailable instead of panicking.
|
||||
var subtitleSource downloads.SubtitleSource
|
||||
if subtitleManager != nil {
|
||||
subtitleSource = subtitleManager
|
||||
}
|
||||
downloadSvc.SetOfflineDeps(detailSvc, subtitleSource, nil)
|
||||
}
|
||||
if deps.ArtifactManager != nil {
|
||||
// Prepare-to-file pipeline (Phase 3): remux/transcode-to-single-file.
|
||||
downloadSvc.SetArtifactManager(deps.ArtifactManager)
|
||||
}
|
||||
// Series monitoring (auto-download subscriptions). Client-pull only:
|
||||
// devices sync on app open / background refresh; there is no server
|
||||
// background worker.
|
||||
downloadSvc.SetSubscriptions(downloads.NewSubscriptionRepository(deps.DB))
|
||||
downloadHandler = handlers.NewDownloadHandler(downloadSvc)
|
||||
if profileHandler != nil {
|
||||
// Profiles may live outside Postgres (sqlite userdb backend), so
|
||||
// deleting one cannot FK-cascade the shared user_devices table;
|
||||
// purge the device library (and its downloads) in-app instead.
|
||||
profileHandler.DeviceLibraryPurger = downloadRepo
|
||||
}
|
||||
} else {
|
||||
downloadHandler = handlers.NewDownloadHandler(nil)
|
||||
}
|
||||
@@ -1625,6 +1658,7 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
}
|
||||
notificationsHandler := handlers.NewNotificationsHandler(deps.Notifications, deps.EventsHub)
|
||||
r.With(apimw.RequireProfile).Post("/events/ws-ticket", notificationsHandler.HandleMintWSTicket)
|
||||
r.With(apimw.RequireProfile).Post("/devices/push/apple", notificationsHandler.HandleRegisterApplePushDevice)
|
||||
// Discord DM channel: the linked identity and mode hang off
|
||||
// the login account, not a profile, so these stay outside
|
||||
// the RequireProfile subrouter below (static paths coexist
|
||||
@@ -1643,6 +1677,7 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
r.Get("/capability", notificationsHandler.HandleCapability)
|
||||
r.Get("/preferences", notificationsHandler.HandleGetPreferences)
|
||||
r.Put("/preferences", notificationsHandler.HandleUpdatePreferences)
|
||||
r.Get("/push/apple/display/{delivery_id}", notificationsHandler.HandleApplePushDisplay)
|
||||
r.Get("/email-preferences", notificationsHandler.HandleGetEmailPreferences)
|
||||
r.Put("/email-preferences", notificationsHandler.HandleUpdateEmailPreferences)
|
||||
r.Put("/email-preferences/address", notificationsHandler.HandleRequestEmailAddress)
|
||||
@@ -2046,6 +2081,8 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
r.Get("/metadata/ai/status", metadataAIHandler.HandleStatus)
|
||||
if itemRepo != nil {
|
||||
metadataAIHandler.ItemAccess = itemRepo
|
||||
metadataAIHandler.SeasonLookup = seasonRepo
|
||||
metadataAIHandler.EpisodeLookup = episodeRepo
|
||||
r.Post("/items/{id}/translate-description", metadataAIHandler.HandleTranslateOnView)
|
||||
}
|
||||
} else {
|
||||
@@ -2148,10 +2185,27 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
|
||||
// Download routes.
|
||||
r.Route("/downloads", func(r chi.Router) {
|
||||
r.Get("/capability", downloadHandler.HandleCapability)
|
||||
r.Post("/", downloadHandler.HandleCreateDownload)
|
||||
r.Get("/", downloadHandler.HandleListDownloads)
|
||||
// Series monitoring (auto-download) subscriptions.
|
||||
r.Post("/subscriptions", downloadHandler.HandleCreateSubscription)
|
||||
r.Post("/subscriptions/sync", downloadHandler.HandleSyncSubscriptions)
|
||||
r.Get("/subscriptions", downloadHandler.HandleListSubscriptions)
|
||||
r.Get("/subscriptions/{id}", downloadHandler.HandleGetSubscription)
|
||||
r.Patch("/subscriptions/{id}", downloadHandler.HandlePatchSubscription)
|
||||
r.Delete("/subscriptions/{id}", downloadHandler.HandleDeleteSubscription)
|
||||
r.Get("/batches/{batch_id}/manifests", downloadHandler.HandleBatchManifests)
|
||||
r.Patch("/{id}", downloadHandler.HandlePatchDownload)
|
||||
r.Delete("/{id}", downloadHandler.HandleDeleteDownload)
|
||||
// GET+HEAD: background download stacks probe with HEAD
|
||||
// before issuing ranged GETs; http.ServeContent handles
|
||||
// HEAD natively.
|
||||
r.Get("/{id}/file", downloadHandler.HandleDownloadFile)
|
||||
r.Head("/{id}/file", downloadHandler.HandleDownloadFile)
|
||||
r.Get("/{id}/manifest", downloadHandler.HandleManifest)
|
||||
r.Get("/{id}/artwork/{kind}", downloadHandler.HandleArtwork)
|
||||
r.Get("/{id}/subtitles/{ref}", downloadHandler.HandleSubtitle)
|
||||
})
|
||||
r.Get("/direct-download", downloadHandler.HandleDirectDownload)
|
||||
r.Head("/direct-download", downloadHandler.HandleDirectDownload)
|
||||
@@ -2300,6 +2354,15 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
if discordNotificationsHandler != nil {
|
||||
r.Post("/notifications/discord/test", discordNotificationsHandler.HandleAdminTest)
|
||||
}
|
||||
if deps.Notifications != nil || settingsRepo != nil {
|
||||
applePushHandler := handlers.NewAdminApplePushHandler(deps.Notifications, settingsRepo)
|
||||
if deps.Notifications != nil {
|
||||
r.Post("/notifications/push/apple/test", applePushHandler.HandleTest)
|
||||
}
|
||||
if settingsRepo != nil {
|
||||
r.Post("/notifications/push/relay/register", applePushHandler.HandleRegisterRelay)
|
||||
}
|
||||
}
|
||||
if deps.Notifications != nil && deps.Notifications.ServerChannels != nil {
|
||||
serverChannelsHandler := handlers.NewAdminServerChannelsHandler(deps.Notifications)
|
||||
r.Route("/notifications/server-channels", func(r chi.Router) {
|
||||
|
||||
@@ -172,6 +172,9 @@ func (h *Handler) handlePublicTrack(w http.ResponseWriter, r *http.Request) {
|
||||
http.Error(w, "session expired", http.StatusGone)
|
||||
return
|
||||
}
|
||||
if h.beginNativePlaybackTransport(sid) {
|
||||
defer h.endNativePlaybackTransport(sid)
|
||||
}
|
||||
|
||||
access, err := h.accessFilterForAuth(r.Context(), ctxAuth{UserID: sess.UserID, ProfileID: sess.ProfileID})
|
||||
if err != nil {
|
||||
|
||||
@@ -258,6 +258,14 @@ type Dependencies struct {
|
||||
// SocketIO is the Socket.io server mounted at /abs/socket.io/. May be nil;
|
||||
// the route is only registered when a non-nil value is supplied.
|
||||
SocketIO SocketIOServer
|
||||
// NativeSessions mirrors ABS playback into Silo's native playback session
|
||||
// manager so shared live-session views, limits, and stale-session cleanup
|
||||
// see Audiobookshelf-compatible clients. May be nil; ABS playback still
|
||||
// functions, but admin live-session visibility is unavailable.
|
||||
NativeSessions PlaybackSessionManager
|
||||
// NativeSessionSyncer flushes native session-manager state into the shared
|
||||
// admin live-session table after ABS play/sync/close events.
|
||||
NativeSessionSyncer PlaybackSessionSyncer
|
||||
// CoverResolver translates a raw silo poster path (e.g.
|
||||
// "local/audiobooks/.../original.webp") into a fully-qualified URL
|
||||
// the ABS client can fetch. Optional; when nil, /api/items/{id}/cover
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
package abs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/clientip"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
"github.com/Silo-Server/silo-server/internal/playback"
|
||||
)
|
||||
|
||||
// PlaybackSessionManager is the native Silo playback-session surface ABS needs
|
||||
// to make compatibility playback visible to live admin/session monitoring.
|
||||
type PlaybackSessionManager interface {
|
||||
StartSessionWithFilesContext(ctx context.Context, userID int, profileID string, effectiveFileID int, requestedFileID int, method playback.PlayMethod, transcodeAudio bool) (*playback.Session, error)
|
||||
UpdateProgress(sessionID string, position float64, isPaused bool) error
|
||||
UpdateStreamState(sessionID string, state playback.SessionStreamState) error
|
||||
BeginTransport(sessionID string) error
|
||||
EndTransport(sessionID string) error
|
||||
SetProgressPersistenceDisabled(sessionID string, disabled bool) error
|
||||
StopSession(sessionID string) error
|
||||
}
|
||||
|
||||
// PlaybackSessionSyncer flushes the local native-session snapshot into the
|
||||
// shared admin live-session table.
|
||||
type PlaybackSessionSyncer interface {
|
||||
SyncNow(ctx context.Context) error
|
||||
}
|
||||
|
||||
func (h *Handler) startNativePlaybackSession(
|
||||
r *http.Request,
|
||||
a ctxAuth,
|
||||
files []*models.MediaFile,
|
||||
startPosition float64,
|
||||
) (*playback.Session, error) {
|
||||
if h == nil || h.deps.NativeSessions == nil || len(files) == 0 {
|
||||
return nil, nil
|
||||
}
|
||||
userID, err := strconv.Atoi(a.UserID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("invalid ABS user id %q: %w", a.UserID, err)
|
||||
}
|
||||
|
||||
file := files[0]
|
||||
sessionCtx := playback.WithClientInfo(r.Context(), absPlaybackClientInfoFromRequest(r))
|
||||
session, err := h.deps.NativeSessions.StartSessionWithFilesContext(
|
||||
sessionCtx,
|
||||
userID,
|
||||
a.ProfileID,
|
||||
file.ID,
|
||||
file.ID,
|
||||
playback.PlayDirect,
|
||||
false,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if err := h.deps.NativeSessions.SetProgressPersistenceDisabled(session.ID, true); err != nil {
|
||||
slog.Warn("abs play: disable native progress persistence failed",
|
||||
"session_id", session.ID, "error", err)
|
||||
} else {
|
||||
session.DisableProgressPersistence = true
|
||||
}
|
||||
|
||||
if startPosition > 0 {
|
||||
if err := h.deps.NativeSessions.UpdateProgress(session.ID, startPosition, false); err != nil {
|
||||
slog.Warn("abs play: seed native session progress failed",
|
||||
"session_id", session.ID, "position", startPosition, "error", err)
|
||||
} else {
|
||||
session.Position = startPosition
|
||||
session.IsPaused = false
|
||||
}
|
||||
}
|
||||
|
||||
streamBitrateKbps := 0
|
||||
if file.Bitrate > 0 {
|
||||
streamBitrateKbps = file.Bitrate
|
||||
}
|
||||
if err := h.deps.NativeSessions.UpdateStreamState(session.ID, playback.SessionStreamState{
|
||||
PlayMethod: playback.PlayDirect,
|
||||
BasePlayMethod: playback.PlayDirect,
|
||||
ClientIP: requestClientIP(r),
|
||||
ClientName: session.ClientName,
|
||||
ClientVersion: session.ClientVersion,
|
||||
ClientUserAgent: session.ClientUserAgent,
|
||||
StreamBitrateKbps: streamBitrateKbps,
|
||||
}); err != nil {
|
||||
slog.Warn("abs play: update native session stream state failed",
|
||||
"session_id", session.ID, "error", err)
|
||||
} else {
|
||||
session.PlayMethod = playback.PlayDirect
|
||||
session.BasePlayMethod = playback.PlayDirect
|
||||
session.ClientIP = requestClientIP(r)
|
||||
session.StreamBitrateKbps = streamBitrateKbps
|
||||
}
|
||||
|
||||
h.syncNativeSessionsNow(r.Context(), "abs_start")
|
||||
return session, nil
|
||||
}
|
||||
|
||||
func (h *Handler) updateNativePlaybackProgress(ctx context.Context, sessionID string, position float64) {
|
||||
if h == nil || h.deps.NativeSessions == nil || sessionID == "" {
|
||||
return
|
||||
}
|
||||
if err := h.deps.NativeSessions.UpdateProgress(sessionID, position, false); err != nil {
|
||||
if !errors.Is(err, playback.ErrSessionNotFound) {
|
||||
slog.Warn("abs session sync: update native session progress failed",
|
||||
"session_id", sessionID, "position", position, "error", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
h.syncNativeSessionsNow(ctx, "abs_progress")
|
||||
}
|
||||
|
||||
func (h *Handler) stopNativePlaybackSession(ctx context.Context, sessionID string) {
|
||||
if h == nil || h.deps.NativeSessions == nil || sessionID == "" {
|
||||
return
|
||||
}
|
||||
if err := h.deps.NativeSessions.StopSession(sessionID); err != nil {
|
||||
if !errors.Is(err, playback.ErrSessionNotFound) {
|
||||
slog.Warn("abs session close: stop native session failed",
|
||||
"session_id", sessionID, "error", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
h.syncNativeSessionsNow(ctx, "abs_close")
|
||||
}
|
||||
|
||||
func (h *Handler) beginNativePlaybackTransport(sessionID string) bool {
|
||||
if h == nil || h.deps.NativeSessions == nil || sessionID == "" {
|
||||
return false
|
||||
}
|
||||
if err := h.deps.NativeSessions.BeginTransport(sessionID); err != nil {
|
||||
if !errors.Is(err, playback.ErrSessionNotFound) {
|
||||
slog.Warn("abs public track: begin native transport failed",
|
||||
"session_id", sessionID, "error", err)
|
||||
}
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (h *Handler) endNativePlaybackTransport(sessionID string) {
|
||||
if h == nil || h.deps.NativeSessions == nil || sessionID == "" {
|
||||
return
|
||||
}
|
||||
if err := h.deps.NativeSessions.EndTransport(sessionID); err != nil && !errors.Is(err, playback.ErrSessionNotFound) {
|
||||
slog.Warn("abs public track: end native transport failed",
|
||||
"session_id", sessionID, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *Handler) syncNativeSessionsNow(ctx context.Context, reason string) {
|
||||
if h == nil || h.deps.NativeSessionSyncer == nil {
|
||||
return
|
||||
}
|
||||
if err := h.deps.NativeSessionSyncer.SyncNow(ctx); err != nil {
|
||||
slog.Error("abs: failed to sync native playback sessions", "reason", reason, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func writeNativePlaybackStartError(w http.ResponseWriter, err error) {
|
||||
switch {
|
||||
case errors.Is(err, playback.ErrTooManyStreams):
|
||||
http.Error(w, "too many concurrent streams", http.StatusTooManyRequests)
|
||||
case errors.Is(err, playback.ErrTooManyTranscodes):
|
||||
http.Error(w, "too many concurrent transcodes", http.StatusTooManyRequests)
|
||||
default:
|
||||
slog.Error("abs play: start native playback session failed", "error", err)
|
||||
http.Error(w, "failed to start playback session", http.StatusInternalServerError)
|
||||
}
|
||||
}
|
||||
|
||||
func absPlaybackClientInfoFromRequest(r *http.Request) playback.ClientInfo {
|
||||
if r == nil {
|
||||
return playback.ClientInfo{Name: "Audiobookshelf"}
|
||||
}
|
||||
name := firstHeaderValue(r,
|
||||
"X-Silo-Client",
|
||||
"X-Client-Name",
|
||||
"X-Device-Name",
|
||||
"X-Emby-Client",
|
||||
)
|
||||
if name == "" {
|
||||
name = "Audiobookshelf"
|
||||
}
|
||||
return playback.ClientInfo{
|
||||
Name: name,
|
||||
Version: firstHeaderValue(r,
|
||||
"X-Silo-Client-Version",
|
||||
"X-Client-Version",
|
||||
"X-Emby-Client-Version",
|
||||
),
|
||||
UserAgent: r.UserAgent(),
|
||||
}
|
||||
}
|
||||
|
||||
func firstHeaderValue(r *http.Request, names ...string) string {
|
||||
for _, name := range names {
|
||||
if value := strings.TrimSpace(r.Header.Get(name)); value != "" {
|
||||
return value
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func requestClientIP(r *http.Request) string {
|
||||
if r == nil {
|
||||
return ""
|
||||
}
|
||||
if ip := clientip.FromContext(r.Context()); ip != "" {
|
||||
return ip
|
||||
}
|
||||
host, _, err := net.SplitHostPort(r.RemoteAddr)
|
||||
if err == nil {
|
||||
return host
|
||||
}
|
||||
return strings.TrimSpace(r.RemoteAddr)
|
||||
}
|
||||
@@ -0,0 +1,229 @@
|
||||
package abs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
"github.com/Silo-Server/silo-server/internal/playback"
|
||||
)
|
||||
|
||||
type playStartMediaStore struct {
|
||||
noopMediaStore
|
||||
item *models.MediaItem
|
||||
files []*models.MediaFile
|
||||
}
|
||||
|
||||
func (s *playStartMediaStore) GetAudiobookByID(_ context.Context, id string, _ catalog.AccessFilter) (*models.MediaItem, error) {
|
||||
if s.item != nil && s.item.ContentID == id {
|
||||
return s.item, nil
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (s *playStartMediaStore) GetMediaFiles(_ context.Context, contentID string, _ catalog.AccessFilter) ([]*models.MediaFile, error) {
|
||||
if s.item != nil && s.item.ContentID == contentID {
|
||||
return s.files, nil
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
type recordingPlaybackSessionSyncer struct {
|
||||
calls int
|
||||
}
|
||||
|
||||
func (s *recordingPlaybackSessionSyncer) SyncNow(context.Context) error {
|
||||
s.calls++
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestHandlePlayStartCreatesNativePlaybackSession(t *testing.T) {
|
||||
now := time.Now()
|
||||
media := &playStartMediaStore{
|
||||
item: &models.MediaItem{
|
||||
ContentID: "book-1",
|
||||
Type: "audiobook",
|
||||
Title: "Native Session Book",
|
||||
UpdatedAt: now,
|
||||
AddedAt: &now,
|
||||
},
|
||||
files: []*models.MediaFile{{
|
||||
ID: 42,
|
||||
ContentID: "book-1",
|
||||
FilePath: "/tmp/book.mp3",
|
||||
FileSize: 1024,
|
||||
Duration: 3600,
|
||||
Bitrate: 128,
|
||||
CodecAudio: "mp3",
|
||||
}},
|
||||
}
|
||||
absSessions := &fakePlaybackSessionStore{}
|
||||
nativeSessions := playback.NewSessionManager(0, 0)
|
||||
syncer := &recordingPlaybackSessionSyncer{}
|
||||
progress := &fakeProgressStore{row: &ProgressRow{
|
||||
UserID: "1",
|
||||
ProfileID: "profile-1",
|
||||
ContentID: "book-1",
|
||||
CurrentSeconds: 123.5,
|
||||
DurationSeconds: 3600,
|
||||
UpdatedAt: now,
|
||||
}}
|
||||
h := New(Dependencies{
|
||||
MediaStore: media,
|
||||
ProgressStore: progress,
|
||||
PlaybackSessionStore: absSessions,
|
||||
NativeSessions: nativeSessions,
|
||||
NativeSessionSyncer: syncer,
|
||||
})
|
||||
|
||||
rec := dispatchABSWithParams(
|
||||
http.MethodPost,
|
||||
"/api/items/book-1/play",
|
||||
map[string]string{"libraryItemId": "book-1"},
|
||||
nil,
|
||||
"1",
|
||||
"profile-1",
|
||||
h.handlePlayStart,
|
||||
)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
|
||||
var body map[string]any
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
|
||||
t.Fatalf("decode response: %v", err)
|
||||
}
|
||||
sessionID, _ := body["id"].(string)
|
||||
if sessionID == "" {
|
||||
t.Fatalf("response id is empty: %#v", body["id"])
|
||||
}
|
||||
native, err := nativeSessions.GetSession(sessionID)
|
||||
if err != nil {
|
||||
t.Fatalf("native session %q missing: %v", sessionID, err)
|
||||
}
|
||||
if native.MediaFileID != 42 || native.RequestedMediaFileID != 42 {
|
||||
t.Fatalf("native file ids = (%d, %d), want (42, 42)", native.MediaFileID, native.RequestedMediaFileID)
|
||||
}
|
||||
if !native.DisableProgressPersistence {
|
||||
t.Fatalf("native session should disable progress persistence")
|
||||
}
|
||||
if native.Position != 123.5 {
|
||||
t.Fatalf("native position = %v, want 123.5", native.Position)
|
||||
}
|
||||
if syncer.calls == 0 {
|
||||
t.Fatalf("native session syncer was not called")
|
||||
}
|
||||
absSession, err := absSessions.GetPlaybackSession(context.Background(), sessionID)
|
||||
if err != nil {
|
||||
t.Fatalf("ABS session %q missing: %v", sessionID, err)
|
||||
}
|
||||
if absSession.CurrentPositionSeconds != 123.5 {
|
||||
t.Fatalf("ABS session position = %v, want 123.5", absSession.CurrentPositionSeconds)
|
||||
}
|
||||
tracks, _ := body["audioTracks"].([]any)
|
||||
if len(tracks) != 1 {
|
||||
t.Fatalf("audioTracks length = %d, want 1", len(tracks))
|
||||
}
|
||||
track, _ := tracks[0].(map[string]any)
|
||||
if got, _ := track["contentUrl"].(string); got == "" || !strings.Contains(got, "/abs/public/session/"+sessionID+"/track/1") {
|
||||
t.Fatalf("contentUrl = %q, want session-scoped URL", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleSessionSyncUpdatesNativePlaybackSession(t *testing.T) {
|
||||
media := &playStartMediaStore{
|
||||
item: &models.MediaItem{ContentID: "book-1", Type: "audiobook", Title: "Book", UpdatedAt: time.Now()},
|
||||
}
|
||||
absSessions := &fakePlaybackSessionStore{}
|
||||
nativeSessions := playback.NewSessionManager(0, 0)
|
||||
native, err := nativeSessions.StartSessionWithFilesContext(context.Background(), 1, "profile-1", 42, 42, playback.PlayDirect, false)
|
||||
if err != nil {
|
||||
t.Fatalf("start native session: %v", err)
|
||||
}
|
||||
_ = absSessions.InsertPlaybackSession(context.Background(), ABSPlaybackSession{
|
||||
ID: native.ID,
|
||||
UserID: "1",
|
||||
ProfileID: "profile-1",
|
||||
ContentID: "book-1",
|
||||
})
|
||||
syncer := &recordingPlaybackSessionSyncer{}
|
||||
h := New(Dependencies{
|
||||
MediaStore: media,
|
||||
ProgressStore: &fakeProgressStore{},
|
||||
PlaybackSessionStore: absSessions,
|
||||
NativeSessions: nativeSessions,
|
||||
NativeSessionSyncer: syncer,
|
||||
})
|
||||
|
||||
rec := dispatchABSWithParams(
|
||||
http.MethodPatch,
|
||||
"/api/session/"+native.ID,
|
||||
map[string]string{"sid": native.ID},
|
||||
[]byte(`{"currentTime":55.25,"timeListening":10}`),
|
||||
"1",
|
||||
"profile-1",
|
||||
h.handleSessionSync,
|
||||
)
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
updated, err := nativeSessions.GetSession(native.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("native session missing: %v", err)
|
||||
}
|
||||
if updated.Position != 55.25 {
|
||||
t.Fatalf("native position = %v, want 55.25", updated.Position)
|
||||
}
|
||||
if updated.IsPaused {
|
||||
t.Fatalf("native session should be marked playing")
|
||||
}
|
||||
if syncer.calls == 0 {
|
||||
t.Fatalf("native session syncer was not called")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleSessionCloseStopsNativePlaybackSession(t *testing.T) {
|
||||
absSessions := &fakePlaybackSessionStore{}
|
||||
nativeSessions := playback.NewSessionManager(0, 0)
|
||||
native, err := nativeSessions.StartSessionWithFilesContext(context.Background(), 1, "profile-1", 42, 42, playback.PlayDirect, false)
|
||||
if err != nil {
|
||||
t.Fatalf("start native session: %v", err)
|
||||
}
|
||||
_ = absSessions.InsertPlaybackSession(context.Background(), ABSPlaybackSession{
|
||||
ID: native.ID,
|
||||
UserID: "1",
|
||||
ProfileID: "profile-1",
|
||||
ContentID: "book-1",
|
||||
})
|
||||
syncer := &recordingPlaybackSessionSyncer{}
|
||||
h := New(Dependencies{
|
||||
MediaStore: noopMediaStore{},
|
||||
PlaybackSessionStore: absSessions,
|
||||
NativeSessions: nativeSessions,
|
||||
NativeSessionSyncer: syncer,
|
||||
})
|
||||
|
||||
rec := dispatchABSWithParams(
|
||||
http.MethodPost,
|
||||
"/api/session/"+native.ID+"/close",
|
||||
map[string]string{"sid": native.ID},
|
||||
nil,
|
||||
"1",
|
||||
"profile-1",
|
||||
h.handleSessionClose,
|
||||
)
|
||||
if rec.Code != http.StatusNoContent {
|
||||
t.Fatalf("status = %d, want 204; body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if _, err := nativeSessions.GetSession(native.ID); err == nil {
|
||||
t.Fatalf("native session still exists after close")
|
||||
}
|
||||
if syncer.calls == 0 {
|
||||
t.Fatalf("native session syncer was not called")
|
||||
}
|
||||
}
|
||||
@@ -59,21 +59,45 @@ func (h *Handler) handlePlayStart(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
|
||||
// currentTime seeds the audio element's initial position so cross-device
|
||||
// resume works. Lookup is best-effort: any error returns position 0,
|
||||
// which is always correct for a first listen.
|
||||
//
|
||||
// Note: we deliberately do NOT emit a "progress" (0.0-1.0) field on the
|
||||
// play session response. The canonical continuum-plugin handler omits it
|
||||
// too — "progress" belongs to /me/progress responses, not playbackSession.
|
||||
// The spec's Phase 0 row mentioning "currentTime AND progress fields" was
|
||||
// over-specified; matching the canonical wire shape is the load-bearing
|
||||
// requirement.
|
||||
var currentTime float64
|
||||
currentTime, err = resolveResumeTime(r.Context(), h.deps.ProgressStore, a.UserID, a.ProfileID, contentID)
|
||||
if err != nil {
|
||||
slog.Debug("play: progress lookup failed", "user", a.UserID, "item", contentID, "err", err)
|
||||
// currentTime is already 0 on error path; safe to continue.
|
||||
}
|
||||
|
||||
// The {ino} parameter used by handleFileStream is a 0-based file index, but
|
||||
// we want iOS clients to resolve it via the stable MD5 derivation. Emit inos
|
||||
// that handleFileStream can reverse without a database lookup.
|
||||
baseURL := h.absBaseURL(r)
|
||||
sessionID := ulid.Make().String()
|
||||
if nativeSession, err := h.startNativePlaybackSession(r, a, files, currentTime); err != nil {
|
||||
writeNativePlaybackStartError(w, err)
|
||||
return
|
||||
} else if nativeSession != nil {
|
||||
sessionID = nativeSession.ID
|
||||
}
|
||||
|
||||
// Persist the session row so subsequent PATCH /session/{sid} heartbeats
|
||||
// and POST /session/{sid}/close can find it. Without this, the session
|
||||
// ID is returned to the client but every sync/close lookup 404s.
|
||||
if h.deps.PlaybackSessionStore != nil {
|
||||
sess := ABSPlaybackSession{
|
||||
ID: sessionID,
|
||||
UserID: a.UserID,
|
||||
ProfileID: a.ProfileID,
|
||||
ContentID: contentID,
|
||||
ID: sessionID,
|
||||
UserID: a.UserID,
|
||||
ProfileID: a.ProfileID,
|
||||
ContentID: contentID,
|
||||
CurrentPositionSeconds: currentTime,
|
||||
}
|
||||
if len(files) > 0 {
|
||||
fid := files[0].ID
|
||||
@@ -111,23 +135,6 @@ func (h *Handler) handlePlayStart(w http.ResponseWriter, r *http.Request) {
|
||||
dateStr := now.UTC().Format("2006-01-02")
|
||||
dayOfWeek := now.UTC().Weekday().String()
|
||||
|
||||
// currentTime seeds the audio element's initial position so cross-device
|
||||
// resume works. Lookup is best-effort: any error returns position 0,
|
||||
// which is always correct for a first listen.
|
||||
//
|
||||
// Note: we deliberately do NOT emit a "progress" (0.0-1.0) field on the
|
||||
// play session response. The canonical continuum-plugin handler omits it
|
||||
// too — "progress" belongs to /me/progress responses, not playbackSession.
|
||||
// The spec's Phase 0 row mentioning "currentTime AND progress fields" was
|
||||
// over-specified; matching the canonical wire shape is the load-bearing
|
||||
// requirement.
|
||||
var currentTime float64
|
||||
currentTime, err = resolveResumeTime(r.Context(), h.deps.ProgressStore, a.UserID, a.ProfileID, contentID)
|
||||
if err != nil {
|
||||
slog.Debug("play: progress lookup failed", "user", a.UserID, "item", contentID, "err", err)
|
||||
// currentTime is already 0 on error path; safe to continue.
|
||||
}
|
||||
|
||||
playbackSession := map[string]any{
|
||||
"id": sessionID,
|
||||
"userId": a.UserID,
|
||||
|
||||
@@ -376,6 +376,7 @@ func (h *Handler) handleSessionSync(w http.ResponseWriter, r *http.Request) {
|
||||
"session_id", sid, "content_id", sess.ContentID, "error", err)
|
||||
}
|
||||
}
|
||||
h.updateNativePlaybackProgress(r.Context(), sid, p.CurrentTime)
|
||||
|
||||
// Realtime push to other connected clients.
|
||||
h.publish(a.UserID, "user_item_progress_updated", map[string]any{
|
||||
@@ -421,6 +422,7 @@ func (h *Handler) handleSessionClose(w http.ResponseWriter, r *http.Request) {
|
||||
http.Error(w, err.Error(), http.StatusInternalServerError)
|
||||
return
|
||||
}
|
||||
h.stopNativePlaybackSession(r.Context(), sid)
|
||||
|
||||
h.publish(a.UserID, "user_session_closed", map[string]any{
|
||||
"id": sid,
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"github.com/Silo-Server/silo-server/internal/audiobooks/abs"
|
||||
"github.com/Silo-Server/silo-server/internal/audiobooks/abssocket"
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/playback"
|
||||
"github.com/Silo-Server/silo-server/internal/recommendations"
|
||||
"github.com/Silo-Server/silo-server/internal/scanner"
|
||||
|
||||
@@ -54,7 +55,9 @@ type ABSHandlerDeps struct {
|
||||
// Detail resolves audiobook poster S3 paths into fully-qualified URLs
|
||||
// that ABS clients can fetch. Optional; when nil, /api/items/{id}/cover
|
||||
// 404s rather than redirecting to an unreachable storage path.
|
||||
Detail *catalog.DetailService
|
||||
Detail *catalog.DetailService
|
||||
SessionMgr *playback.SessionManager
|
||||
SessionSyncer abs.PlaybackSessionSyncer
|
||||
}
|
||||
|
||||
// absAuthAdapter is the narrow slice of internal/auth that BuildABSHandler
|
||||
@@ -158,6 +161,8 @@ func (s *Service) BuildABSHandler(deps ABSHandlerDeps) *abs.Handler {
|
||||
SmartCollectionStore: smartCollectionStore,
|
||||
RSSFeedStore: rssFeedStore,
|
||||
SocketIO: socketServer,
|
||||
NativeSessions: deps.SessionMgr,
|
||||
NativeSessionSyncer: deps.SessionSyncer,
|
||||
CoverResolver: func(ctx context.Context, path, variant string) string {
|
||||
if deps.Detail == nil {
|
||||
return ""
|
||||
|
||||
@@ -124,6 +124,157 @@ func (r *BrowseRepository) browse(ctx context.Context, filters BrowseFilters, in
|
||||
}, nil
|
||||
}
|
||||
|
||||
// BrowseRecentlyAddedAcrossLibraries serves a recently_added browse spanning
|
||||
// multiple libraries by running the proven single-library index path once per
|
||||
// library and merging the already-sorted results in memory. Each per-library
|
||||
// query filters on exactly one media_folder_id, so it takes the
|
||||
// singleLibraryNoDedup fast path and walks idx_item_libraries_folder_seen_content
|
||||
// as a top-N index scan (~1ms). N small walks replace one multi-library
|
||||
// MIN(first_seen_at) + GROUP BY scan of the whole catalog (HashAggregate over
|
||||
// ~180k groups + top-N heapsort, ~2s with a wide projection and cold cache).
|
||||
//
|
||||
// It only serves offset 0 — the /Items/Latest hot path. A per-library top-Limit
|
||||
// fetch cannot satisfy a deep global offset, so callers must route offset>0
|
||||
// pages to BrowsePage; passing a positive Offset here is an error rather than a
|
||||
// silently-wrong page.
|
||||
//
|
||||
// Cross-library dedup is performed app-side: a content_id present in more than
|
||||
// one library keeps the EARLIEST first_seen_at, preserving the
|
||||
// MIN(first_seen_at) "added at" semantics of the multi-library GROUP BY path.
|
||||
// The reported Total is a lower bound (exact when the page is the whole result),
|
||||
// chosen so HasMore remains correct for offset 0.
|
||||
//
|
||||
// Known limitation vs the exact GROUP BY query, bounded to content_ids that
|
||||
// live in 2+ libraries: such a dup is only deduped against the slices actually
|
||||
// fetched, so if its true MIN(first_seen_at) library does not rank it in that
|
||||
// library's top-Limit, the merge sees only a later first_seen_at and may rank it
|
||||
// too recent (or drop it if it is outside top-Limit in every library holding it).
|
||||
// This is acceptable for an approximate "recently added" rail; do not reuse this
|
||||
// path where exact MIN ranking across libraries is required.
|
||||
func (r *BrowseRepository) BrowseRecentlyAddedAcrossLibraries(ctx context.Context, base BrowseFilters, libraryIDs []int) (*BrowseResult, error) {
|
||||
if base.Offset > 0 {
|
||||
return nil, fmt.Errorf("browse recently added across libraries: offset %d unsupported, use BrowsePage", base.Offset)
|
||||
}
|
||||
limit := base.Limit
|
||||
if limit <= 0 {
|
||||
limit = 20
|
||||
}
|
||||
// Each per-library subquery is clamped to MaxLimit by BrowsePage, but the
|
||||
// merged page is trimmed to this `limit`; clamp it too so the final result
|
||||
// honors the same BrowseFilters.MaxLimit contract as the single-library path.
|
||||
if base.MaxLimit > 0 && limit > base.MaxLimit {
|
||||
limit = base.MaxLimit
|
||||
}
|
||||
|
||||
perLibrary := make([][]*models.MediaItem, 0, len(libraryIDs))
|
||||
anyLibraryHasMore := false
|
||||
for _, id := range libraryIDs {
|
||||
sub := base
|
||||
sub.LibraryID = id
|
||||
// Clearing the multi-library scopes is what selects the
|
||||
// singleLibraryNoDedup fast path in buildBrowsePlan.
|
||||
sub.LibraryIDs = nil
|
||||
sub.DisabledLibraryIDs = nil
|
||||
sub.Offset = 0
|
||||
sub.Limit = limit
|
||||
res, err := r.BrowsePage(ctx, sub, false)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("browse recently added in library %d: %w", id, err)
|
||||
}
|
||||
if res == nil {
|
||||
continue
|
||||
}
|
||||
perLibrary = append(perLibrary, res.Items)
|
||||
if res.HasMore {
|
||||
anyLibraryHasMore = true
|
||||
}
|
||||
}
|
||||
|
||||
merged := mergeRecentlyAddedItems(perLibrary)
|
||||
// More results exist when the merged set already overflows the page, or when
|
||||
// any single library still has rows behind its own top-Limit window (which
|
||||
// implies that library alone holds more than `limit` items).
|
||||
hasMore := anyLibraryHasMore || len(merged) > limit
|
||||
total := len(merged)
|
||||
if hasMore && total <= limit {
|
||||
// A library has rows beyond the page but the merged set fit exactly;
|
||||
// nudge the total so offset-0 HasMore stays true for total-driven callers.
|
||||
total = limit + 1
|
||||
}
|
||||
if len(merged) > limit {
|
||||
merged = merged[:limit]
|
||||
}
|
||||
|
||||
return &BrowseResult{Items: merged, Total: total, HasMore: hasMore}, nil
|
||||
}
|
||||
|
||||
// mergeRecentlyAddedItems merges per-library recently_added result slices — each
|
||||
// already sorted by first_seen_at DESC, content_id ASC — into a single globally
|
||||
// ordered slice. Duplicate content_ids are collapsed to one entry whose AddedAt
|
||||
// is the EARLIEST across libraries, mirroring MIN(first_seen_at) from the
|
||||
// multi-library GROUP BY path. The result is sorted by AddedAt DESC, content_id
|
||||
// ASC and is NOT truncated; callers trim to the page size and derive HasMore.
|
||||
func mergeRecentlyAddedItems(perLibrary [][]*models.MediaItem) []*models.MediaItem {
|
||||
byID := make(map[string]*models.MediaItem)
|
||||
order := make([]*models.MediaItem, 0)
|
||||
for _, items := range perLibrary {
|
||||
for _, it := range items {
|
||||
if it == nil {
|
||||
continue
|
||||
}
|
||||
existing, ok := byID[it.ContentID]
|
||||
if !ok {
|
||||
byID[it.ContentID] = it
|
||||
order = append(order, it)
|
||||
continue
|
||||
}
|
||||
// Keep the earliest first_seen_at to preserve MIN semantics.
|
||||
if addedAtBefore(it.AddedAt, existing.AddedAt) {
|
||||
existing.AddedAt = it.AddedAt
|
||||
}
|
||||
}
|
||||
}
|
||||
// Re-sort after dedup because collapsing to MIN(first_seen_at) can move an
|
||||
// item earlier than its position in any single per-library slice.
|
||||
slices.SortStableFunc(order, compareRecentlyAdded)
|
||||
return order
|
||||
}
|
||||
|
||||
// compareRecentlyAdded orders items by AddedAt DESC, then content_id ASC,
|
||||
// matching "ORDER BY mil.first_seen_at DESC, mi.content_id ASC". A nil AddedAt
|
||||
// sorts last (treated as the oldest).
|
||||
func compareRecentlyAdded(a, b *models.MediaItem) int {
|
||||
at, bt := a.AddedAt, b.AddedAt
|
||||
switch {
|
||||
case at == nil && bt == nil:
|
||||
// fall through to the content_id tie-break
|
||||
case at == nil:
|
||||
return 1
|
||||
case bt == nil:
|
||||
return -1
|
||||
default:
|
||||
if at.After(*bt) {
|
||||
return -1
|
||||
}
|
||||
if at.Before(*bt) {
|
||||
return 1
|
||||
}
|
||||
}
|
||||
return strings.Compare(a.ContentID, b.ContentID)
|
||||
}
|
||||
|
||||
// addedAtBefore reports whether a is strictly earlier than b, treating nil as
|
||||
// the oldest possible value so a real timestamp always wins the MIN.
|
||||
func addedAtBefore(a, b *time.Time) bool {
|
||||
if a == nil {
|
||||
return b != nil
|
||||
}
|
||||
if b == nil {
|
||||
return false
|
||||
}
|
||||
return a.Before(*b)
|
||||
}
|
||||
|
||||
// browseQueryPlan captures the rendered SQL fragments and bound args for a
|
||||
// browse data query. It is produced by buildBrowsePlan and consumed both by
|
||||
// browse() (to execute) and by tests (to inspect the emitted SQL without a
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
)
|
||||
|
||||
func tsPtr(t time.Time) *time.Time { return &t }
|
||||
|
||||
func contentIDs(items []*models.MediaItem) []string {
|
||||
ids := make([]string, len(items))
|
||||
for i, it := range items {
|
||||
ids[i] = it.ContentID
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
func TestMergeRecentlyAddedItems_OrdersByAddedAtDescThenContentID(t *testing.T) {
|
||||
base := time.Date(2026, 6, 1, 0, 0, 0, 0, time.UTC)
|
||||
// Two already-sorted per-library slices (added_at DESC, content_id ASC).
|
||||
libA := []*models.MediaItem{
|
||||
{ContentID: "a3", AddedAt: tsPtr(base.Add(3 * time.Hour))},
|
||||
{ContentID: "a1", AddedAt: tsPtr(base.Add(1 * time.Hour))},
|
||||
}
|
||||
libB := []*models.MediaItem{
|
||||
{ContentID: "b4", AddedAt: tsPtr(base.Add(4 * time.Hour))},
|
||||
{ContentID: "b2", AddedAt: tsPtr(base.Add(2 * time.Hour))},
|
||||
}
|
||||
|
||||
got := contentIDs(mergeRecentlyAddedItems([][]*models.MediaItem{libA, libB}))
|
||||
want := []string{"b4", "a3", "b2", "a1"}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("merged length = %d, want %d (%v)", len(got), len(want), got)
|
||||
}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
t.Fatalf("merged order = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeRecentlyAddedItems_TieBreaksByContentIDAsc(t *testing.T) {
|
||||
at := time.Date(2026, 6, 1, 12, 0, 0, 0, time.UTC)
|
||||
libA := []*models.MediaItem{{ContentID: "zeta", AddedAt: tsPtr(at)}}
|
||||
libB := []*models.MediaItem{{ContentID: "alpha", AddedAt: tsPtr(at)}}
|
||||
|
||||
got := contentIDs(mergeRecentlyAddedItems([][]*models.MediaItem{libA, libB}))
|
||||
want := []string{"alpha", "zeta"}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
t.Fatalf("tie-break order = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeRecentlyAddedItems_DedupsKeepingEarliestAddedAt(t *testing.T) {
|
||||
early := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
|
||||
late := time.Date(2026, 6, 1, 0, 0, 0, 0, time.UTC)
|
||||
other := time.Date(2026, 3, 1, 0, 0, 0, 0, time.UTC)
|
||||
|
||||
// "dup" appears in both libraries with different first_seen_at. The merge
|
||||
// must collapse it to one entry carrying the EARLIEST timestamp (MIN
|
||||
// semantics), which also moves it after "other" in DESC order.
|
||||
libA := []*models.MediaItem{
|
||||
{ContentID: "dup", AddedAt: tsPtr(late)},
|
||||
}
|
||||
libB := []*models.MediaItem{
|
||||
{ContentID: "other", AddedAt: tsPtr(other)},
|
||||
{ContentID: "dup", AddedAt: tsPtr(early)},
|
||||
}
|
||||
|
||||
merged := mergeRecentlyAddedItems([][]*models.MediaItem{libA, libB})
|
||||
if got := len(merged); got != 2 {
|
||||
t.Fatalf("merged length = %d, want 2 (dup must collapse): %v", got, contentIDs(merged))
|
||||
}
|
||||
// other (March) should now precede dup (January).
|
||||
want := []string{"other", "dup"}
|
||||
for i := range want {
|
||||
if merged[i].ContentID != want[i] {
|
||||
t.Fatalf("dedup order = %v, want %v", contentIDs(merged), want)
|
||||
}
|
||||
}
|
||||
var dup *models.MediaItem
|
||||
for _, it := range merged {
|
||||
if it.ContentID == "dup" {
|
||||
dup = it
|
||||
}
|
||||
}
|
||||
if dup.AddedAt == nil || !dup.AddedAt.Equal(early) {
|
||||
t.Fatalf("dup AddedAt = %v, want earliest %v", dup.AddedAt, early)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeRecentlyAddedItems_NilAddedAtSortsLast(t *testing.T) {
|
||||
at := time.Date(2026, 6, 1, 0, 0, 0, 0, time.UTC)
|
||||
libA := []*models.MediaItem{{ContentID: "nil1", AddedAt: nil}}
|
||||
libB := []*models.MediaItem{{ContentID: "dated", AddedAt: tsPtr(at)}}
|
||||
|
||||
got := contentIDs(mergeRecentlyAddedItems([][]*models.MediaItem{libA, libB}))
|
||||
want := []string{"dated", "nil1"}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
t.Fatalf("nil-handling order = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,166 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
)
|
||||
|
||||
func newRestampTestPool(t *testing.T) *pgxpool.Pool {
|
||||
t.Helper()
|
||||
dsn := os.Getenv("SILO_TEST_DATABASE_URL")
|
||||
if dsn == "" {
|
||||
t.Skip("SILO_TEST_DATABASE_URL is not set")
|
||||
}
|
||||
pool, err := pgxpool.New(context.Background(), dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("connect test database: %v", err)
|
||||
}
|
||||
t.Cleanup(pool.Close)
|
||||
return pool
|
||||
}
|
||||
|
||||
// TestItemUpsertRestampsDefaultMetadataLanguage covers the persistence half of
|
||||
// issue #211: when a refresh adopts a new canonical language, the upsert must
|
||||
// accept the incoming non-empty default_metadata_language instead of pinning
|
||||
// to the stamp written at first match. An empty incoming value must still
|
||||
// keep the existing stamp (scanner skeleton writes send the zero value).
|
||||
func TestItemUpsertRestampsDefaultMetadataLanguage(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
pool := newRestampTestPool(t)
|
||||
repo := NewItemRepository(pool)
|
||||
|
||||
contentID := fmt.Sprintf("restamp-item-%d", time.Now().UnixNano())
|
||||
t.Cleanup(func() {
|
||||
_, _ = pool.Exec(ctx, `DELETE FROM media_items WHERE content_id = $1`, contentID)
|
||||
})
|
||||
|
||||
base := &models.MediaItem{
|
||||
ContentID: contentID,
|
||||
Type: "movie",
|
||||
Title: "Gammel Titel",
|
||||
Status: "matched",
|
||||
DefaultMetadataLanguage: "zh",
|
||||
Studios: []string{},
|
||||
Networks: []string{},
|
||||
Countries: []string{},
|
||||
Genres: []string{},
|
||||
}
|
||||
if err := repo.Upsert(ctx, base); err != nil {
|
||||
t.Fatalf("initial upsert: %v", err)
|
||||
}
|
||||
|
||||
restamped := *base
|
||||
restamped.Title = "Ny Titel"
|
||||
restamped.DefaultMetadataLanguage = "da"
|
||||
if err := repo.Upsert(ctx, &restamped); err != nil {
|
||||
t.Fatalf("restamp upsert: %v", err)
|
||||
}
|
||||
got, err := repo.GetByID(ctx, contentID)
|
||||
if err != nil {
|
||||
t.Fatalf("get item: %v", err)
|
||||
}
|
||||
if got.DefaultMetadataLanguage != "da" {
|
||||
t.Errorf("default_metadata_language = %q, want da (incoming non-empty stamp must win)", got.DefaultMetadataLanguage)
|
||||
}
|
||||
|
||||
emptied := restamped
|
||||
emptied.DefaultMetadataLanguage = ""
|
||||
if err := repo.Upsert(ctx, &emptied); err != nil {
|
||||
t.Fatalf("empty-language upsert: %v", err)
|
||||
}
|
||||
got, err = repo.GetByID(ctx, contentID)
|
||||
if err != nil {
|
||||
t.Fatalf("get item after empty upsert: %v", err)
|
||||
}
|
||||
if got.DefaultMetadataLanguage != "da" {
|
||||
t.Errorf("default_metadata_language = %q, want da kept when incoming is empty", got.DefaultMetadataLanguage)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSeasonAndEpisodeUpsertRestampDefaultMetadataLanguage verifies the same
|
||||
// restamp-on-upsert semantics for the season and episode tables, which carry
|
||||
// their own default_metadata_language pins.
|
||||
func TestSeasonAndEpisodeUpsertRestampDefaultMetadataLanguage(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
pool := newRestampTestPool(t)
|
||||
|
||||
suffix := time.Now().UnixNano()
|
||||
seriesID := fmt.Sprintf("restamp-series-%d", suffix)
|
||||
seasonID := fmt.Sprintf("restamp-season-%d", suffix)
|
||||
episodeID := fmt.Sprintf("restamp-episode-%d", suffix)
|
||||
t.Cleanup(func() {
|
||||
// Seasons and episodes cascade from the series row (FK ON DELETE
|
||||
// CASCADE), so a single delete cleans up everything seeded here.
|
||||
_, _ = pool.Exec(ctx, `DELETE FROM media_items WHERE content_id = $1`, seriesID)
|
||||
})
|
||||
|
||||
itemRepo := NewItemRepository(pool)
|
||||
if err := itemRepo.Upsert(ctx, &models.MediaItem{
|
||||
ContentID: seriesID,
|
||||
Type: "series",
|
||||
Title: "Restamp Series",
|
||||
Status: "matched",
|
||||
DefaultMetadataLanguage: "zh",
|
||||
Studios: []string{},
|
||||
Networks: []string{},
|
||||
Countries: []string{},
|
||||
Genres: []string{},
|
||||
}); err != nil {
|
||||
t.Fatalf("seed series: %v", err)
|
||||
}
|
||||
|
||||
seasonRepo := NewSeasonRepository(pool)
|
||||
season := &models.Season{
|
||||
ContentID: seasonID,
|
||||
SeriesID: seriesID,
|
||||
SeasonNumber: 1,
|
||||
Title: "Sæson 1",
|
||||
DefaultMetadataLanguage: "zh",
|
||||
}
|
||||
if err := seasonRepo.Upsert(ctx, season); err != nil {
|
||||
t.Fatalf("seed season: %v", err)
|
||||
}
|
||||
season.DefaultMetadataLanguage = "da"
|
||||
if err := seasonRepo.Upsert(ctx, season); err != nil {
|
||||
t.Fatalf("restamp season: %v", err)
|
||||
}
|
||||
var seasonLang string
|
||||
if err := pool.QueryRow(ctx, `SELECT default_metadata_language FROM seasons WHERE content_id = $1`, seasonID).Scan(&seasonLang); err != nil {
|
||||
t.Fatalf("read season language: %v", err)
|
||||
}
|
||||
if seasonLang != "da" {
|
||||
t.Errorf("season default_metadata_language = %q, want da", seasonLang)
|
||||
}
|
||||
|
||||
episodeRepo := NewEpisodeRepository(pool)
|
||||
episode := &models.Episode{
|
||||
ContentID: episodeID,
|
||||
SeriesID: seriesID,
|
||||
SeasonID: seasonID,
|
||||
SeasonNumber: 1,
|
||||
EpisodeNumber: 1,
|
||||
Title: "Afsnit 1",
|
||||
DefaultMetadataLanguage: "zh",
|
||||
}
|
||||
if err := episodeRepo.Upsert(ctx, episode); err != nil {
|
||||
t.Fatalf("seed episode: %v", err)
|
||||
}
|
||||
episode.DefaultMetadataLanguage = "da"
|
||||
if err := episodeRepo.Upsert(ctx, episode); err != nil {
|
||||
t.Fatalf("restamp episode: %v", err)
|
||||
}
|
||||
var episodeLang string
|
||||
if err := pool.QueryRow(ctx, `SELECT default_metadata_language FROM episodes WHERE content_id = $1`, episodeID).Scan(&episodeLang); err != nil {
|
||||
t.Fatalf("read episode language: %v", err)
|
||||
}
|
||||
if episodeLang != "da" {
|
||||
t.Errorf("episode default_metadata_language = %q, want da", episodeLang)
|
||||
}
|
||||
}
|
||||
+326
-52
@@ -677,6 +677,55 @@ func (s *DetailService) PendingTranslationLanguage(ctx context.Context, item *mo
|
||||
return ""
|
||||
}
|
||||
loc, err := s.itemLocRepo.Get(ctx, item.ContentID, language)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return pendingTranslationLanguageWith(item, language, loc)
|
||||
}
|
||||
|
||||
// pendingTranslationLanguageWith is the shared core of PendingTranslationLanguage:
|
||||
// it decides the pending-translation language for an item given a pre-resolved
|
||||
// presentation language and the item's localization row (nil when none exists).
|
||||
// The batch detail path reuses it after a single bulk localization lookup so it
|
||||
// yields an identical result to the per-item method.
|
||||
func pendingTranslationLanguageWith(item *models.MediaItem, language string, loc *models.MediaItemLocalization) string {
|
||||
if item == nil || strings.TrimSpace(item.Overview) == "" || language == "" || sameMetadataLanguage(item.DefaultMetadataLanguage, language) {
|
||||
return ""
|
||||
}
|
||||
if loc != nil && loc.Overview != "" {
|
||||
return ""
|
||||
}
|
||||
return language
|
||||
}
|
||||
|
||||
// PendingSeasonTranslationLanguage is the season-row equivalent of
|
||||
// PendingTranslationLanguage.
|
||||
func (s *DetailService) PendingSeasonTranslationLanguage(ctx context.Context, season *models.Season, filter AccessFilter) string {
|
||||
if season == nil || strings.TrimSpace(season.Overview) == "" || s.seasonLocRepo == nil {
|
||||
return ""
|
||||
}
|
||||
language, err := s.resolvePresentationLanguage(ctx, filter)
|
||||
if err != nil || language == "" || sameMetadataLanguage(season.DefaultMetadataLanguage, language) {
|
||||
return ""
|
||||
}
|
||||
loc, err := s.seasonLocRepo.Get(ctx, season.ContentID, language)
|
||||
if err != nil || (loc != nil && loc.Overview != "") {
|
||||
return ""
|
||||
}
|
||||
return language
|
||||
}
|
||||
|
||||
// PendingEpisodeTranslationLanguage is the episode-row equivalent of
|
||||
// PendingTranslationLanguage.
|
||||
func (s *DetailService) PendingEpisodeTranslationLanguage(ctx context.Context, episode *models.Episode, filter AccessFilter) string {
|
||||
if episode == nil || strings.TrimSpace(episode.Overview) == "" || s.episodeLocRepo == nil {
|
||||
return ""
|
||||
}
|
||||
language, err := s.resolvePresentationLanguage(ctx, filter)
|
||||
if err != nil || language == "" || sameMetadataLanguage(episode.DefaultMetadataLanguage, language) {
|
||||
return ""
|
||||
}
|
||||
loc, err := s.episodeLocRepo.Get(ctx, episode.ContentID, language)
|
||||
if err != nil || (loc != nil && loc.Overview != "") {
|
||||
return ""
|
||||
}
|
||||
@@ -710,10 +759,22 @@ func (s *DetailService) LocalizeItemModel(ctx context.Context, item *models.Medi
|
||||
return cloneMediaItem(item), err
|
||||
}
|
||||
loc, err := s.itemLocRepo.Get(ctx, item.ContentID, language)
|
||||
if err != nil || loc == nil {
|
||||
if err != nil {
|
||||
return cloneMediaItem(item), err
|
||||
}
|
||||
return applyItemLocalization(item, loc), nil
|
||||
return s.localizeItemModelWith(item, language, loc), nil
|
||||
}
|
||||
|
||||
// localizeItemModelWith applies a pre-resolved localization to item, returning a
|
||||
// clone when there is nothing to localize (no language, base language already
|
||||
// matches, no localization repo, or no localization row). Shared core of
|
||||
// LocalizeItemModel so the batch detail path produces identical models after a
|
||||
// single bulk localization lookup.
|
||||
func (s *DetailService) localizeItemModelWith(item *models.MediaItem, language string, loc *models.MediaItemLocalization) *models.MediaItem {
|
||||
if language == "" || sameMetadataLanguage(item.DefaultMetadataLanguage, language) || s.itemLocRepo == nil || loc == nil {
|
||||
return cloneMediaItem(item)
|
||||
}
|
||||
return applyItemLocalization(item, loc)
|
||||
}
|
||||
|
||||
// LocalizeItemModels applies presentation-language localization to a batch of
|
||||
@@ -842,7 +903,7 @@ func (s *DetailService) GetItemDetail(ctx context.Context, contentID string, fil
|
||||
if err := s.validatePresentationItemAccess(ctx, filter, contentID); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return s.buildMediaItemDetail(ctx, item, contentID, filter)
|
||||
return s.buildMediaItemDetail(ctx, item, contentID, filter, nil)
|
||||
case !errors.Is(err, ErrItemNotFound):
|
||||
return nil, err
|
||||
}
|
||||
@@ -971,6 +1032,164 @@ func (s *DetailService) GetEpisodeDetailsForSeries(
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// fileVersionBatchFetcher is the optional batch form of FileVersionFetcher.
|
||||
// When the configured fetcher implements it, GetItemDetailsByIDs loads every
|
||||
// page item's media files in one query instead of one per item. Fetchers that
|
||||
// don't implement it (e.g. some test fakes) transparently fall back to the
|
||||
// per-item GetByContentID path inside buildMediaItemDetail.
|
||||
type fileVersionBatchFetcher interface {
|
||||
ListByContentIDs(ctx context.Context, contentIDs []string) (map[string][]*models.MediaFile, error)
|
||||
}
|
||||
|
||||
// GetItemDetailsByIDs returns full item details for the given content IDs,
|
||||
// batching the shared per-item lookups that GetItemDetail otherwise issues
|
||||
// one-at-a-time: row load (GetByIDs), access checks (EnsureAccessibleIDs +
|
||||
// presentation-library membership), localization, credits, media files, and
|
||||
// work summaries. The detail produced for each id is byte-for-byte identical to
|
||||
// calling GetItemDetail(id): the same builder (buildMediaItemDetail) assembles
|
||||
// it, only fed pre-resolved inputs via itemDetailPrefetch. Any input building
|
||||
// block missing a batch form falls back to its inline per-item lookup, so the
|
||||
// output never drifts from the unbatched path.
|
||||
//
|
||||
// IDs that are not item-typed media (e.g. episodes or seasons — handled by
|
||||
// GetEpisodeDetailsForSeries / GetItemDetail respectively), are access-filtered,
|
||||
// or fail to build are simply absent from the result map, mirroring
|
||||
// GetItemDetail returning an error so callers fall back to list-level rendering.
|
||||
func (s *DetailService) GetItemDetailsByIDs(ctx context.Context, contentIDs []string, filter AccessFilter) (map[string]*ItemDetail, error) {
|
||||
result := make(map[string]*ItemDetail, len(contentIDs))
|
||||
if len(contentIDs) == 0 {
|
||||
return result, nil
|
||||
}
|
||||
|
||||
items, err := s.itemRepo.GetByIDs(ctx, contentIDs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(items) == 0 {
|
||||
return result, nil
|
||||
}
|
||||
|
||||
foundIDs := make([]string, 0, len(items))
|
||||
for _, item := range items {
|
||||
foundIDs = append(foundIDs, item.ContentID)
|
||||
}
|
||||
|
||||
// Access checks, batched to match GetItemDetail's per-item EnsureAccessible
|
||||
// and validatePresentationItemAccess exactly.
|
||||
accessible, err := s.itemRepo.EnsureAccessibleIDs(ctx, foundIDs, filter)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
presentationOK := func(string) bool { return true }
|
||||
if filter.PresentationLibraryID != nil {
|
||||
membership, err := s.itemRepo.GetItemsInLibrary(ctx, foundIDs, *filter.PresentationLibraryID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
presentationOK = func(id string) bool { return membership[id] }
|
||||
}
|
||||
|
||||
visible := make([]*models.MediaItem, 0, len(items))
|
||||
visibleIDs := make([]string, 0, len(items))
|
||||
nonSeriesIDs := make([]string, 0, len(items))
|
||||
for _, item := range items {
|
||||
if !accessible[item.ContentID] || !presentationOK(item.ContentID) {
|
||||
continue
|
||||
}
|
||||
visible = append(visible, item)
|
||||
visibleIDs = append(visibleIDs, item.ContentID)
|
||||
if item.Type != "series" {
|
||||
nonSeriesIDs = append(nonSeriesIDs, item.ContentID)
|
||||
}
|
||||
}
|
||||
if len(visible) == 0 {
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// Localization: resolve the presentation language once, then one bulk
|
||||
// localization lookup. A resolution failure surfaces the same wrapped error
|
||||
// GetItemDetail would produce from buildMediaItemDetail, so the caller
|
||||
// degrades to the per-item path.
|
||||
language, err := s.resolvePresentationLanguage(ctx, filter)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("localizing item detail: %w", err)
|
||||
}
|
||||
var locByID map[string]*models.MediaItemLocalization
|
||||
if language != "" && s.itemLocRepo != nil {
|
||||
locByID, err = s.itemLocRepo.GetByContentIDs(ctx, visibleIDs, language)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("localizing item detail: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Credits for the whole page in one query.
|
||||
var creditsByID map[string][]models.ItemPerson
|
||||
if s.personRepo != nil {
|
||||
creditsByID, err = s.personRepo.ListForItems(ctx, visibleIDs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
// Media files (playable versions) for non-series items in one query, when the
|
||||
// fetcher supports batching.
|
||||
var filesByID map[string][]*models.MediaFile
|
||||
batchFetcher, haveFileBatch := s.fileFetcher.(fileVersionBatchFetcher)
|
||||
if haveFileBatch && len(nonSeriesIDs) > 0 {
|
||||
filesByID, err = batchFetcher.ListByContentIDs(ctx, nonSeriesIDs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
// Work summaries for the whole page in one query, when the provider supports
|
||||
// batching (ListSummariesForContentIDs is the documented batch equivalent of
|
||||
// GetSummaryForContentID).
|
||||
var workSummaries map[string]*WorkSummary
|
||||
batchSummary, haveWorkSummaryBatch := s.workSummary.(WorkSummaryBatchProvider)
|
||||
if haveWorkSummaryBatch {
|
||||
workSummaries, err = batchSummary.ListSummariesForContentIDs(ctx, visibleIDs, filter)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
for _, item := range visible {
|
||||
id := item.ContentID
|
||||
loc := locByID[id]
|
||||
pending := ""
|
||||
if s.itemLocRepo != nil {
|
||||
pending = pendingTranslationLanguageWith(item, language, loc)
|
||||
}
|
||||
pf := &itemDetailPrefetch{
|
||||
haveLocalization: true,
|
||||
pendingTranslation: pending,
|
||||
localizedItem: s.localizeItemModelWith(item, language, loc),
|
||||
haveCredits: s.personRepo != nil,
|
||||
}
|
||||
if s.personRepo != nil {
|
||||
pf.castCredits, pf.crewCredits = splitCastCrew(s.personCredits(ctx, creditsByID[id]))
|
||||
}
|
||||
if item.Type != "series" && haveFileBatch {
|
||||
pf.haveFiles = true
|
||||
pf.files = filesByID[id]
|
||||
}
|
||||
if haveWorkSummaryBatch {
|
||||
pf.haveWorkSummary = true
|
||||
pf.workSummary = workSummaries[id]
|
||||
}
|
||||
detail, err := s.buildMediaItemDetail(ctx, item, id, filter, pf)
|
||||
if err != nil {
|
||||
// Skip rather than fail the batch; the caller falls back to list
|
||||
// rendering for any id missing from the result, matching the
|
||||
// per-item path where one item's build error never breaks the page.
|
||||
continue
|
||||
}
|
||||
result[id] = detail
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// fetchCredits returns cast and crew credits for the given content ID.
|
||||
func (s *DetailService) fetchCredits(ctx context.Context, contentID string) ([]CastCredit, []CrewCredit) {
|
||||
if s.personRepo == nil {
|
||||
@@ -984,14 +1203,45 @@ func (s *DetailService) fetchCredits(ctx context.Context, contentID string) ([]C
|
||||
return splitCastCrew(credits)
|
||||
}
|
||||
|
||||
func (s *DetailService) buildMediaItemDetail(ctx context.Context, item *models.MediaItem, contentID string, filter AccessFilter) (*ItemDetail, error) {
|
||||
pendingTranslation := s.PendingTranslationLanguage(ctx, item, filter)
|
||||
localizedItem, err := s.LocalizeItemModel(ctx, item, filter)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("localizing item detail: %w", err)
|
||||
// itemDetailPrefetch carries per-item lookups that GetItemDetailsByIDs resolves
|
||||
// once for a whole page so buildMediaItemDetail can skip the equivalent per-item
|
||||
// queries. Each piece is opt-in via its "have" flag: when unset, the builder
|
||||
// falls back to its normal inline lookup, so the result is byte-for-byte
|
||||
// identical to the unbatched GetItemDetail path regardless of which pieces were
|
||||
// prefetched. A nil prefetch reproduces the original per-item behavior exactly.
|
||||
type itemDetailPrefetch struct {
|
||||
haveLocalization bool
|
||||
pendingTranslation string
|
||||
localizedItem *models.MediaItem
|
||||
haveCredits bool
|
||||
castCredits []CastCredit
|
||||
crewCredits []CrewCredit
|
||||
haveFiles bool
|
||||
files []*models.MediaFile
|
||||
haveWorkSummary bool
|
||||
workSummary *WorkSummary
|
||||
}
|
||||
|
||||
func (s *DetailService) buildMediaItemDetail(ctx context.Context, item *models.MediaItem, contentID string, filter AccessFilter, pf *itemDetailPrefetch) (*ItemDetail, error) {
|
||||
var pendingTranslation string
|
||||
if pf != nil && pf.haveLocalization {
|
||||
pendingTranslation = pf.pendingTranslation
|
||||
item = pf.localizedItem
|
||||
} else {
|
||||
pendingTranslation = s.PendingTranslationLanguage(ctx, item, filter)
|
||||
localizedItem, err := s.LocalizeItemModel(ctx, item, filter)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("localizing item detail: %w", err)
|
||||
}
|
||||
item = localizedItem
|
||||
}
|
||||
var castCredits []CastCredit
|
||||
var crewCredits []CrewCredit
|
||||
if pf != nil && pf.haveCredits {
|
||||
castCredits, crewCredits = pf.castCredits, pf.crewCredits
|
||||
} else {
|
||||
castCredits, crewCredits = s.fetchCredits(ctx, contentID)
|
||||
}
|
||||
item = localizedItem
|
||||
castCredits, crewCredits := s.fetchCredits(ctx, contentID)
|
||||
detail := &ItemDetail{
|
||||
ContentID: item.ContentID,
|
||||
Type: item.Type,
|
||||
@@ -1042,9 +1292,15 @@ func (s *DetailService) buildMediaItemDetail(ctx context.Context, item *models.M
|
||||
// For series, each episode file shares the series content_id, so
|
||||
// GetByContentID would return every episode — not alternate encodings.
|
||||
if item.Type != "series" {
|
||||
files, err := s.fileFetcher.GetByContentID(ctx, contentID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("fetching file versions: %w", err)
|
||||
var files []*models.MediaFile
|
||||
if pf != nil && pf.haveFiles {
|
||||
files = pf.files
|
||||
} else {
|
||||
fetched, err := s.fileFetcher.GetByContentID(ctx, contentID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("fetching file versions: %w", err)
|
||||
}
|
||||
files = fetched
|
||||
}
|
||||
|
||||
files = FilterMediaFilesByAccess(files, filter)
|
||||
@@ -1115,7 +1371,11 @@ func (s *DetailService) buildMediaItemDetail(ctx context.Context, item *models.M
|
||||
}
|
||||
}
|
||||
|
||||
applyWorkSummary(ctx, detail, s.workSummary, filter)
|
||||
if pf != nil && pf.haveWorkSummary {
|
||||
applyWorkSummaryValue(detail, pf.workSummary)
|
||||
} else {
|
||||
applyWorkSummary(ctx, detail, s.workSummary, filter)
|
||||
}
|
||||
return detail, nil
|
||||
}
|
||||
|
||||
@@ -1124,7 +1384,17 @@ func applyWorkSummary(ctx context.Context, detail *ItemDetail, provider WorkSumm
|
||||
return
|
||||
}
|
||||
summary, err := provider.GetSummaryForContentID(ctx, detail.ContentID, filter)
|
||||
if err != nil || summary == nil {
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
applyWorkSummaryValue(detail, summary)
|
||||
}
|
||||
|
||||
// applyWorkSummaryValue copies a resolved work summary onto the detail. Shared by
|
||||
// the per-item (applyWorkSummary) and batched (GetItemDetailsByIDs) paths so a
|
||||
// summary fetched in bulk applies identically. A nil summary is a no-op.
|
||||
func applyWorkSummaryValue(detail *ItemDetail, summary *WorkSummary) {
|
||||
if detail == nil || summary == nil {
|
||||
return
|
||||
}
|
||||
detail.WorkID = summary.WorkID
|
||||
@@ -1755,6 +2025,7 @@ func clearSentinel(s string) string {
|
||||
}
|
||||
|
||||
func (s *DetailService) buildSeasonDetail(ctx context.Context, season *models.Season, filter AccessFilter) (*ItemDetail, error) {
|
||||
pendingTranslation := s.PendingSeasonTranslationLanguage(ctx, season, filter)
|
||||
localizedSeason, err := s.LocalizeSeasonModel(ctx, season, filter)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("localizing season detail: %w", err)
|
||||
@@ -1790,22 +2061,23 @@ func (s *DetailService) buildSeasonDetail(ctx context.Context, season *models.Se
|
||||
seasonNumber := season.SeasonNumber
|
||||
castCredits, crewCredits := s.fetchCredits(ctx, season.SeriesID)
|
||||
detail := &ItemDetail{
|
||||
ContentID: season.ContentID,
|
||||
Type: "season",
|
||||
Title: title,
|
||||
Overview: season.Overview,
|
||||
PosterThumbhash: season.PosterThumbhash,
|
||||
BackdropThumbhash: series.BackdropThumbhash,
|
||||
SeriesID: season.SeriesID,
|
||||
SeriesTitle: series.Title,
|
||||
SeasonNumber: &seasonNumber,
|
||||
EpisodeCount: &episodeCount,
|
||||
IsSpecials: season.SeasonNumber == 0,
|
||||
Cast: castCredits,
|
||||
Crew: crewCredits,
|
||||
Versions: []FileVersion{},
|
||||
PlaybackVariants: []PlaybackVariant{},
|
||||
Subtitles: []SubtitleInfo{},
|
||||
ContentID: season.ContentID,
|
||||
Type: "season",
|
||||
Title: title,
|
||||
Overview: season.Overview,
|
||||
PendingTranslationLanguage: pendingTranslation,
|
||||
PosterThumbhash: season.PosterThumbhash,
|
||||
BackdropThumbhash: series.BackdropThumbhash,
|
||||
SeriesID: season.SeriesID,
|
||||
SeriesTitle: series.Title,
|
||||
SeasonNumber: &seasonNumber,
|
||||
EpisodeCount: &episodeCount,
|
||||
IsSpecials: season.SeasonNumber == 0,
|
||||
Cast: castCredits,
|
||||
Crew: crewCredits,
|
||||
Versions: []FileVersion{},
|
||||
PlaybackVariants: []PlaybackVariant{},
|
||||
Subtitles: []SubtitleInfo{},
|
||||
}
|
||||
if season.AirDate != nil {
|
||||
airDate := season.AirDate.Format("2006-01-02")
|
||||
@@ -1818,6 +2090,7 @@ func (s *DetailService) buildSeasonDetail(ctx context.Context, season *models.Se
|
||||
}
|
||||
|
||||
func (s *DetailService) buildEpisodeDetail(ctx context.Context, episode *models.Episode, seriesCtx *seriesDetailContext, filter AccessFilter) (*ItemDetail, error) {
|
||||
pendingTranslation := s.PendingEpisodeTranslationLanguage(ctx, episode, filter)
|
||||
localizedEpisode, err := s.LocalizeEpisodeModel(ctx, episode, filter)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("localizing episode detail: %w", err)
|
||||
@@ -1828,27 +2101,28 @@ func (s *DetailService) buildEpisodeDetail(ctx context.Context, episode *models.
|
||||
seasonNumber := episode.SeasonNumber
|
||||
episodeNumber := episode.EpisodeNumber
|
||||
detail := &ItemDetail{
|
||||
ContentID: episode.ContentID,
|
||||
Type: "episode",
|
||||
Title: episode.Title,
|
||||
Overview: episode.Overview,
|
||||
Runtime: episode.Runtime,
|
||||
RatingIMDB: episode.RatingIMDB,
|
||||
RatingTMDB: episode.RatingTMDB,
|
||||
ImdbID: episode.ImdbID,
|
||||
TmdbID: episode.TmdbID,
|
||||
TvdbID: episode.TvdbID,
|
||||
PosterThumbhash: episode.StillThumbhash,
|
||||
BackdropThumbhash: series.BackdropThumbhash,
|
||||
SeriesID: episode.SeriesID,
|
||||
SeriesTitle: series.Title,
|
||||
SeasonNumber: &seasonNumber,
|
||||
EpisodeNumber: &episodeNumber,
|
||||
Cast: seriesCtx.castCredits,
|
||||
Crew: seriesCtx.crewCredits,
|
||||
Versions: []FileVersion{},
|
||||
PlaybackVariants: []PlaybackVariant{},
|
||||
Subtitles: []SubtitleInfo{},
|
||||
ContentID: episode.ContentID,
|
||||
Type: "episode",
|
||||
Title: episode.Title,
|
||||
Overview: episode.Overview,
|
||||
PendingTranslationLanguage: pendingTranslation,
|
||||
Runtime: episode.Runtime,
|
||||
RatingIMDB: episode.RatingIMDB,
|
||||
RatingTMDB: episode.RatingTMDB,
|
||||
ImdbID: episode.ImdbID,
|
||||
TmdbID: episode.TmdbID,
|
||||
TvdbID: episode.TvdbID,
|
||||
PosterThumbhash: episode.StillThumbhash,
|
||||
BackdropThumbhash: series.BackdropThumbhash,
|
||||
SeriesID: episode.SeriesID,
|
||||
SeriesTitle: series.Title,
|
||||
SeasonNumber: &seasonNumber,
|
||||
EpisodeNumber: &episodeNumber,
|
||||
Cast: seriesCtx.castCredits,
|
||||
Crew: seriesCtx.crewCredits,
|
||||
Versions: []FileVersion{},
|
||||
PlaybackVariants: []PlaybackVariant{},
|
||||
Subtitles: []SubtitleInfo{},
|
||||
}
|
||||
if episode.AirDate != nil {
|
||||
airDate := episode.AirDate.Format("2006-01-02")
|
||||
|
||||
@@ -0,0 +1,281 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"reflect"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
)
|
||||
|
||||
// batchEquivFileFetcher is a FileVersionFetcher (and the optional batch
|
||||
// fileVersionBatchFetcher) backed by one in-memory map. Serving both the
|
||||
// per-item GetByContentID and the batched ListByContentIDs from the same
|
||||
// backing data is exactly how the real scanner.FileRepository behaves: the two
|
||||
// methods return the same files for a given content ID. This lets the
|
||||
// equivalence test prove that GetItemDetailsByIDs routes prefetched files into
|
||||
// buildMediaItemDetail identically to the per-item GetItemDetail path, without
|
||||
// pulling the scanner package in (which would create an import cycle).
|
||||
type batchEquivFileFetcher struct {
|
||||
files map[string][]*models.MediaFile
|
||||
}
|
||||
|
||||
func (f *batchEquivFileFetcher) GetByContentID(_ context.Context, contentID string) ([]*models.MediaFile, error) {
|
||||
return f.files[contentID], nil
|
||||
}
|
||||
|
||||
func (f *batchEquivFileFetcher) GetByEpisodeID(_ context.Context, _ string) ([]*models.MediaFile, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (f *batchEquivFileFetcher) ListByContentIDs(_ context.Context, contentIDs []string) (map[string][]*models.MediaFile, error) {
|
||||
out := make(map[string][]*models.MediaFile, len(contentIDs))
|
||||
for _, id := range contentIDs {
|
||||
if files, ok := f.files[id]; ok {
|
||||
out[id] = files
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// batchEquivWorkSummaryProvider implements both the per-item
|
||||
// WorkSummaryProvider and the batched WorkSummaryBatchProvider from one backing
|
||||
// map, the same contract literaryworks.Repository satisfies in production. As
|
||||
// with files, this proves the batch path's work-summary routing matches the
|
||||
// per-item path without importing literaryworks (import cycle).
|
||||
type batchEquivWorkSummaryProvider struct {
|
||||
summaries map[string]*WorkSummary
|
||||
}
|
||||
|
||||
func (p *batchEquivWorkSummaryProvider) GetSummaryForContentID(_ context.Context, contentID string, _ AccessFilter) (*WorkSummary, error) {
|
||||
return p.summaries[contentID], nil
|
||||
}
|
||||
|
||||
func (p *batchEquivWorkSummaryProvider) ListSummariesForContentIDs(_ context.Context, contentIDs []string, _ AccessFilter) (map[string]*WorkSummary, error) {
|
||||
out := make(map[string]*WorkSummary, len(contentIDs))
|
||||
for _, id := range contentIDs {
|
||||
if s, ok := p.summaries[id]; ok {
|
||||
out[id] = s
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func newBatchEquivTestPool(t *testing.T) *pgxpool.Pool {
|
||||
t.Helper()
|
||||
dsn := os.Getenv("SILO_TEST_DATABASE_URL")
|
||||
if dsn == "" {
|
||||
t.Skip("SILO_TEST_DATABASE_URL is not set")
|
||||
}
|
||||
ctx := context.Background()
|
||||
pool, err := pgxpool.New(ctx, dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("connect test database: %v", err)
|
||||
}
|
||||
t.Cleanup(pool.Close)
|
||||
var tableName *string
|
||||
if err := pool.QueryRow(ctx, `SELECT to_regclass('public.media_items')::text`).Scan(&tableName); err != nil {
|
||||
t.Fatalf("check media_items table: %v", err)
|
||||
}
|
||||
if tableName == nil || *tableName == "" {
|
||||
t.Skip("test database has not applied base schema")
|
||||
}
|
||||
return pool
|
||||
}
|
||||
|
||||
func batchEquivExec(t *testing.T, pool *pgxpool.Pool, sql string, args ...any) {
|
||||
t.Helper()
|
||||
if _, err := pool.Exec(context.Background(), sql, args...); err != nil {
|
||||
t.Fatalf("seed (%s): %v", sql, err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestGetItemDetailsByIDs_MatchesGetItemDetail exercises the REAL
|
||||
// DetailService.GetItemDetailsByIDs against the REAL per-item GetItemDetail over
|
||||
// shared in-memory/real repositories, asserting per-id deep equality of the
|
||||
// returned *ItemDetail.
|
||||
//
|
||||
// Unlike the jellycompat handler test (which stubs the content service so both
|
||||
// paths return the same canned *ItemDetail), this drives the genuinely-batched
|
||||
// repository code: ItemRepository.GetByIDs vs GetByID, EnsureAccessibleIDs vs
|
||||
// EnsureAccessible, MediaItemLocalizationRepository.GetByContentIDs vs Get, and
|
||||
// PersonRepository.ListForItems vs ListForItem all run against a real Postgres.
|
||||
// Media files and work summaries route through interface fakes that serve the
|
||||
// batch and per-item method shapes from one backing map (mirroring the real
|
||||
// FileRepository / literaryworks.Repository), so the file/work-summary
|
||||
// prefetch wiring is compared too.
|
||||
//
|
||||
// Equivalence axes covered, with a mixed movie+movie+series dataset:
|
||||
// - access checks (EnsureAccessibleIDs vs EnsureAccessible), incl. an
|
||||
// access-filtered R-rated item asserted ABSENT from the batch map and
|
||||
// rejected by per-item GetItemDetail;
|
||||
// - localization: a movie with an fr localization row (localized title/
|
||||
// overview path) alongside a movie that has none (nil short-circuit) and
|
||||
// a pending-translation-language case;
|
||||
// - credits ordering (ListForItems vs ListForItem) on the first movie;
|
||||
// - media files (ListByContentIDs vs GetByContentID) on the first movie;
|
||||
// - work summary (ListSummariesForContentIDs vs GetSummaryForContentID) on
|
||||
// the series.
|
||||
func TestGetItemDetailsByIDs_MatchesGetItemDetail(t *testing.T) {
|
||||
pool := newBatchEquivTestPool(t)
|
||||
ctx := context.Background()
|
||||
|
||||
suffix := time.Now().UnixNano()
|
||||
movieA := fmt.Sprintf("batch-equiv-movie-a-%d", suffix)
|
||||
movieB := fmt.Sprintf("batch-equiv-movie-b-%d", suffix)
|
||||
series := fmt.Sprintf("batch-equiv-series-%d", suffix)
|
||||
movieR := fmt.Sprintf("batch-equiv-movie-r-%d", suffix)
|
||||
personActor1 := suffix
|
||||
personActor2 := suffix + 1
|
||||
personDirector := suffix + 2
|
||||
|
||||
t.Cleanup(func() {
|
||||
ids := []string{movieA, movieB, series, movieR}
|
||||
batchEquivExec(t, pool, `DELETE FROM item_people WHERE content_id = ANY($1)`, ids)
|
||||
batchEquivExec(t, pool, `DELETE FROM people WHERE id = ANY($1)`, []int64{personActor1, personActor2, personDirector})
|
||||
batchEquivExec(t, pool, `DELETE FROM media_item_localizations WHERE content_id = ANY($1)`, ids)
|
||||
batchEquivExec(t, pool, `DELETE FROM media_items WHERE content_id = ANY($1)`, ids)
|
||||
})
|
||||
|
||||
insertItem := func(contentID, mediaType, title, overview, rating string) {
|
||||
batchEquivExec(t, pool, `
|
||||
INSERT INTO media_items (content_id, type, title, genres, overview, content_rating, default_metadata_language)
|
||||
VALUES ($1, $2, $3, '{}'::text[], $4, $5, 'en')
|
||||
`, contentID, mediaType, title, overview, rating)
|
||||
}
|
||||
// movieA: localized into fr, credited, has files. content_rating PG -> visible.
|
||||
insertItem(movieA, "movie", "Movie A", "Movie A overview (en)", "PG")
|
||||
// movieB: overview present, default lang en, NO fr localization row -> the
|
||||
// pending-translation-language path must fire and match between paths.
|
||||
insertItem(movieB, "movie", "Movie B", "Movie B overview (en)", "PG")
|
||||
// series: has a work summary, no playable files (series skip the file path).
|
||||
insertItem(series, "series", "Series One", "Series overview (en)", "PG")
|
||||
// movieR: R-rated -> filtered out by MaxContentRating=PG-13 on BOTH paths.
|
||||
insertItem(movieR, "movie", "Movie R", "Movie R overview (en)", "R")
|
||||
|
||||
// fr localization for movieA only.
|
||||
batchEquivExec(t, pool, `
|
||||
INSERT INTO media_item_localizations (
|
||||
content_id, language, title, sort_title, overview, tagline,
|
||||
poster_path, poster_source_path, poster_thumbhash,
|
||||
backdrop_path, backdrop_source_path, backdrop_thumbhash,
|
||||
logo_path, logo_source_path, overview_source, tagline_source
|
||||
)
|
||||
VALUES ($1, 'fr', 'Film A', '', 'Synopsis du film A (fr)', '',
|
||||
'', '', '', '', '', '', '', '', 'manual', 'manual')
|
||||
`, movieA)
|
||||
|
||||
// Credits on movieA: two actors (cast) + one director (crew). Seeded with
|
||||
// sort_order so ListForItems / ListForItem ordering parity is observable.
|
||||
insertPerson := func(id int64, name string) {
|
||||
batchEquivExec(t, pool, `INSERT INTO people (id, name) VALUES ($1, $2)`, id, name)
|
||||
}
|
||||
insertPerson(personActor1, "Lead Actor")
|
||||
insertPerson(personActor2, "Supporting Actor")
|
||||
insertPerson(personDirector, "The Director")
|
||||
insertCredit := func(id, personID int64, kind models.PersonKind, character string, order int) {
|
||||
batchEquivExec(t, pool, `
|
||||
INSERT INTO item_people (id, person_id, content_id, kind, character, sort_order)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)
|
||||
`, id, personID, movieA, int16(kind), character, order)
|
||||
}
|
||||
insertCredit(personActor1, personActor1, models.PersonKindActor, "Hero", 0)
|
||||
insertCredit(personActor2, personActor2, models.PersonKindActor, "Sidekick", 1)
|
||||
insertCredit(personDirector, personDirector, models.PersonKindDirector, "", 0)
|
||||
|
||||
fileFetcher := &batchEquivFileFetcher{files: map[string][]*models.MediaFile{
|
||||
movieA: {
|
||||
{ID: 9001, ContentID: movieA, FilePath: "/media/movie-a-1080p.mkv", Container: "mkv", Resolution: "1080p", Duration: 6000, FileSize: 5_000_000},
|
||||
{ID: 9002, ContentID: movieA, FilePath: "/media/movie-a-720p.mkv", Container: "mkv", Resolution: "720p", Duration: 6000, FileSize: 2_000_000},
|
||||
},
|
||||
}}
|
||||
workSummaries := &batchEquivWorkSummaryProvider{summaries: map[string]*WorkSummary{
|
||||
series: {
|
||||
WorkID: "work-series-1",
|
||||
Title: "Series One Work",
|
||||
Formats: []WorkFormatSummary{
|
||||
{Type: "series", ContentID: series, LibraryID: 3},
|
||||
},
|
||||
},
|
||||
}}
|
||||
|
||||
newService := func() *DetailService {
|
||||
svc := NewDetailService(
|
||||
NewItemRepository(pool),
|
||||
NewEpisodeRepository(pool),
|
||||
NewSeasonRepository(pool),
|
||||
NewPersonRepository(pool),
|
||||
fileFetcher,
|
||||
)
|
||||
svc.SetWorkSummaryProvider(workSummaries)
|
||||
return svc
|
||||
}
|
||||
|
||||
// Single shared access filter for BOTH paths: PG-13 ceiling (so PG items are
|
||||
// visible, the R item is not) and an explicit fr presentation language (so
|
||||
// the localization path actually runs rather than short-circuiting).
|
||||
filter := AccessFilter{
|
||||
PresentationLanguage: "fr",
|
||||
MaxContentRating: "PG-13",
|
||||
UserID: 1,
|
||||
ProfileID: "profile-1",
|
||||
}
|
||||
|
||||
visibleIDs := []string{movieA, movieB, series}
|
||||
allIDs := []string{movieA, movieB, series, movieR}
|
||||
|
||||
batch, err := newService().GetItemDetailsByIDs(ctx, allIDs, filter)
|
||||
if err != nil {
|
||||
t.Fatalf("GetItemDetailsByIDs: %v", err)
|
||||
}
|
||||
|
||||
// Access-control parity: the R-rated item must be absent from the batch map
|
||||
// and rejected by per-item GetItemDetail, i.e. EnsureAccessibleIDs and
|
||||
// EnsureAccessible agree.
|
||||
if _, present := batch[movieR]; present {
|
||||
t.Fatalf("access-filtered item %s present in batch result", movieR)
|
||||
}
|
||||
if _, err := newService().GetItemDetail(ctx, movieR, filter); err == nil {
|
||||
t.Fatalf("per-item GetItemDetail(%s) succeeded, want access rejection", movieR)
|
||||
}
|
||||
|
||||
for _, id := range visibleIDs {
|
||||
batchDetail, ok := batch[id]
|
||||
if !ok {
|
||||
t.Fatalf("id %s missing from batch result", id)
|
||||
}
|
||||
perItem, err := newService().GetItemDetail(ctx, id, filter)
|
||||
if err != nil {
|
||||
t.Fatalf("GetItemDetail(%s): %v", id, err)
|
||||
}
|
||||
if !reflect.DeepEqual(batchDetail, perItem) {
|
||||
t.Fatalf("detail mismatch for %s:\nbatch: %#v\nperItem: %#v", id, batchDetail, perItem)
|
||||
}
|
||||
}
|
||||
|
||||
// Guard that the distinguishing data actually landed (otherwise the deep
|
||||
// equality above could pass vacuously on empty details).
|
||||
if got := batch[movieA]; got.Title != "Film A" || got.Overview != "Synopsis du film A (fr)" {
|
||||
t.Fatalf("movieA not localized into fr: title=%q overview=%q", got.Title, got.Overview)
|
||||
}
|
||||
if got := batch[movieA]; len(got.Cast) != 2 || len(got.Crew) != 1 {
|
||||
t.Fatalf("movieA credits = %d cast / %d crew, want 2/1", len(got.Cast), len(got.Crew))
|
||||
}
|
||||
if got := batch[movieA]; got.Cast[0].Character != "Hero" || got.Cast[1].Character != "Sidekick" {
|
||||
t.Fatalf("movieA cast order = %q,%q, want Hero,Sidekick", got.Cast[0].Character, got.Cast[1].Character)
|
||||
}
|
||||
if got := batch[movieA]; len(got.Versions) != 2 {
|
||||
t.Fatalf("movieA versions = %d, want 2 (file prefetch not applied)", len(got.Versions))
|
||||
}
|
||||
if got := batch[movieB]; got.PendingTranslationLanguage != "fr" {
|
||||
t.Fatalf("movieB pending translation language = %q, want fr", got.PendingTranslationLanguage)
|
||||
}
|
||||
if got := batch[series]; got.WorkID != "work-series-1" {
|
||||
t.Fatalf("series work summary not applied: WorkID=%q", got.WorkID)
|
||||
}
|
||||
}
|
||||
@@ -108,6 +108,9 @@ var SensitiveSettingKeys = map[string]bool{
|
||||
// value by the notifications system; clients receive the public half via
|
||||
// the capability endpoint, never from the settings store).
|
||||
"notifications.web_push.vapid_keypair": true,
|
||||
|
||||
// Silo push relay bearer credential for APNs/FCM delivery.
|
||||
"notifications.push_relay_api_key": true,
|
||||
}
|
||||
|
||||
// EncryptedSettingsRepo decorates a raw settings store, transparently
|
||||
|
||||
@@ -158,6 +158,7 @@ func TestSensitiveSettingKeys_Audited(t *testing.T) {
|
||||
"discord.client_secret",
|
||||
"discord.bot_token",
|
||||
"notifications.web_push.vapid_keypair",
|
||||
"notifications.push_relay_api_key",
|
||||
}
|
||||
for _, k := range mustHave {
|
||||
if !SensitiveSettingKeys[k] {
|
||||
|
||||
@@ -32,6 +32,31 @@ type episodeCatalogUserStatePlan struct {
|
||||
requireProgressRatio bool
|
||||
}
|
||||
|
||||
// applyEpisodeCatalogAccessFilter applies the section access constraints the
|
||||
// episode_catalog_entries table supports. The table has no `type` column, so
|
||||
// the compat audiobook/podcast media-type exclusion cannot be expressed here:
|
||||
// emitting `NOT (ece.type = ANY(...))` raised SQLSTATE 42703 and degraded the
|
||||
// whole query to empty. The exclusion is therefore dropped on this path.
|
||||
//
|
||||
// Dropping it is NOT a no-op masquerading as cosmetic: episode_catalog_entries
|
||||
// is NOT episodes-of-TV-series only — episode_libraries is populated from any
|
||||
// media_files row with a non-null episode_id, so podcast episodes
|
||||
// (media_items.type = 'podcast') also land here. What keeps non-TV episodes out
|
||||
// of episode-scoped results is library scoping plus the `si.type = 'series'`
|
||||
// guard in episodeCatalogSelectBody (hydration), not the media-type clause.
|
||||
// content_rating filtering is preserved here (that column exists).
|
||||
//
|
||||
// Because hydration's si.type='series' guard drops non-series rows AFTER the
|
||||
// page/count scans select them, the scan and hydration would otherwise diverge:
|
||||
// pages could under-fill and TotalRecordCount could count rows that never
|
||||
// render. episodeCatalogSeriesParentGuard re-expresses that same constraint at
|
||||
// the entry-scan level so both queries stay aligned with hydration.
|
||||
func applyEpisodeCatalogAccessFilter(access AccessFilter, whereParts *[]string, args *[]any, argIdx *int) {
|
||||
access.ExcludedMediaTypes = nil // access is a value param; caller unaffected
|
||||
ApplySectionAccessFilter("ece", access, whereParts, args, argIdx)
|
||||
*whereParts = append(*whereParts, episodeCatalogSeriesParentGuard)
|
||||
}
|
||||
|
||||
func (e *QueryExecutor) tryEpisodeCatalogUserStatePreviewPage(
|
||||
ctx context.Context,
|
||||
def QueryDefinition,
|
||||
@@ -84,7 +109,7 @@ func (e *QueryExecutor) tryEpisodeCatalogUserStatePreviewPage(
|
||||
argIdx++
|
||||
}
|
||||
|
||||
ApplySectionAccessFilter("ece", access, &whereParts, &args, &argIdx)
|
||||
applyEpisodeCatalogAccessFilter(access, &whereParts, &args, &argIdx)
|
||||
|
||||
if e.SnapshotAt != nil {
|
||||
whereParts = append(whereParts, fmt.Sprintf("ece.episode_created_at <= $%d", argIdx))
|
||||
@@ -242,7 +267,7 @@ func (e *QueryExecutor) tryEpisodeCatalogEntriesPreviewPage(
|
||||
argIdx++
|
||||
}
|
||||
|
||||
ApplySectionAccessFilter("ece", access, &whereParts, &args, &argIdx)
|
||||
applyEpisodeCatalogAccessFilter(access, &whereParts, &args, &argIdx)
|
||||
|
||||
if e.SnapshotAt != nil {
|
||||
whereParts = append(whereParts, fmt.Sprintf("ece.episode_created_at <= $%d", argIdx))
|
||||
|
||||
@@ -45,6 +45,31 @@ func TestEpisodeCatalogSelectBodyExposesQualifiedColumns(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestEpisodeCatalogSelectBodyExcludesNonSeriesEpisodes guards the invariant
|
||||
// that the episode catalog only ever hydrates episodes whose parent is a TV
|
||||
// series. episode_libraries (and episode_catalog_entries) also contain podcast
|
||||
// episodes (media_items.type = 'podcast'); the `si.type = 'series'` predicate in
|
||||
// episodeCatalogSelectBody is what keeps those non-TV rows out of episode-scoped
|
||||
// results regardless of which library a query resolves. Dropping it would let
|
||||
// podcast/audiobook episodes surface in a TV "episode" rail (fail-open).
|
||||
func TestEpisodeCatalogSelectBodyExcludesNonSeriesEpisodes(t *testing.T) {
|
||||
if !strings.Contains(episodeCatalogBaseRelation, "si.type = 'series'") {
|
||||
t.Fatalf("episodeCatalogBaseRelation must guard si.type = 'series' so non-TV "+
|
||||
"(podcast/audiobook) episodes never hydrate; got %q", episodeCatalogBaseRelation)
|
||||
}
|
||||
|
||||
// The library-scoped relation and the hydration relation render from the same
|
||||
// template, so the guard must travel with every rendering.
|
||||
scoped, _, handled := episodeCatalogBaseRelationForLibraries([]int{2}, nil, 1)
|
||||
if !handled {
|
||||
t.Fatal("expected episode library scope to be handled")
|
||||
}
|
||||
if !strings.Contains(scoped, "si.type = 'series'") {
|
||||
t.Fatalf("library-scoped episode relation must keep the si.type = 'series' "+
|
||||
"guard; got %q", scoped)
|
||||
}
|
||||
}
|
||||
|
||||
// columnsReferencedOnAlias returns every distinct column referenced as
|
||||
// "<alias>.<column>" within projection (e.g. mi.air_timezone -> air_timezone).
|
||||
func columnsReferencedOnAlias(projection, alias string) map[string]bool {
|
||||
|
||||
@@ -5,6 +5,18 @@ import (
|
||||
"strings"
|
||||
)
|
||||
|
||||
// episodeCatalogSelectBody is the hand-written subquery that hydrates episode
|
||||
// rows for the "episode" media scope, aliased "mi" so the shared catalog
|
||||
// projection can read it. The `si.type = 'series'` predicate is load-bearing,
|
||||
// not cosmetic: episode_libraries (and thus episode_catalog_entries) is
|
||||
// populated from every media_files row with a non-null episode_id, including
|
||||
// podcast episodes (media_items.type = 'podcast'). Without this guard those
|
||||
// non-TV episodes would hydrate into episode-scoped results, e.g. when a
|
||||
// library-scoped episode query happens to resolve a podcast/audiobook folder.
|
||||
// Restricting hydration to series-parented episodes keeps the episode catalog
|
||||
// to genuine TV episodes on every surface (native and Jellyfin-compat). Real
|
||||
// TV episodes always hang off a type='series' parent, so this never
|
||||
// over-filters legitimate episode sections.
|
||||
const episodeCatalogSelectBody = `(
|
||||
SELECT
|
||||
e.content_id,
|
||||
@@ -63,9 +75,27 @@ const episodeCatalogSelectBody = `(
|
||||
FROM episodes e
|
||||
JOIN media_items si ON si.content_id = e.series_id
|
||||
LEFT JOIN seasons s ON s.content_id = e.season_id
|
||||
WHERE %s
|
||||
WHERE (%s)
|
||||
AND si.type = 'series'
|
||||
) mi`
|
||||
|
||||
// episodeCatalogSeriesParentGuard mirrors the load-bearing `si.type = 'series'`
|
||||
// predicate in episodeCatalogSelectBody, but expressed against an entry-scan row
|
||||
// (ece.episode_id) instead of the hydration join. episode_catalog_entries rows
|
||||
// can reference episodes parented to non-series media_items (e.g. podcast
|
||||
// episodes), which hydration silently drops. Applying this same constraint to
|
||||
// the page and count scans keeps them aligned with hydration so pages never
|
||||
// under-fill and TotalRecordCount never counts rows that can't hydrate. It binds
|
||||
// no parameters, so it can be appended to any whereParts list without touching
|
||||
// the argument index.
|
||||
const episodeCatalogSeriesParentGuard = `EXISTS (
|
||||
SELECT 1
|
||||
FROM episodes e
|
||||
JOIN media_items si ON si.content_id = e.series_id
|
||||
WHERE e.content_id = ece.episode_id
|
||||
AND si.type = 'series'
|
||||
)`
|
||||
|
||||
const episodeCatalogActiveLibraryExists = `EXISTS (
|
||||
SELECT 1
|
||||
FROM episode_libraries el
|
||||
|
||||
@@ -320,7 +320,7 @@ func (r *EpisodeRepository) Upsert(ctx context.Context, ep *models.Episode) erro
|
||||
ON CONFLICT (series_id, season_number, episode_number) DO UPDATE SET
|
||||
season_id = COALESCE(EXCLUDED.season_id, episodes.season_id),
|
||||
title = EXCLUDED.title,
|
||||
default_metadata_language = COALESCE(NULLIF(episodes.default_metadata_language, ''), EXCLUDED.default_metadata_language),
|
||||
default_metadata_language = COALESCE(NULLIF(EXCLUDED.default_metadata_language, ''), episodes.default_metadata_language),
|
||||
overview = EXCLUDED.overview,
|
||||
air_date = EXCLUDED.air_date,
|
||||
runtime = EXCLUDED.runtime,
|
||||
@@ -503,7 +503,7 @@ func (r *EpisodeRepository) BulkUpsert(ctx context.Context, seriesID string, epi
|
||||
ON CONFLICT (series_id, season_number, episode_number) DO UPDATE SET
|
||||
season_id = COALESCE(EXCLUDED.season_id, episodes.season_id),
|
||||
title = EXCLUDED.title,
|
||||
default_metadata_language = COALESCE(NULLIF(episodes.default_metadata_language, ''), EXCLUDED.default_metadata_language),
|
||||
default_metadata_language = COALESCE(NULLIF(EXCLUDED.default_metadata_language, ''), episodes.default_metadata_language),
|
||||
overview = EXCLUDED.overview,
|
||||
air_date = EXCLUDED.air_date,
|
||||
runtime = EXCLUDED.runtime,
|
||||
@@ -827,6 +827,50 @@ func (r *EpisodeRepository) ListBySeason(ctx context.Context, seriesID string, s
|
||||
return scanEpisodes(rows)
|
||||
}
|
||||
|
||||
// ListAdjacentInSeries returns the episode at (seasonNumber, episodeNumber)
|
||||
// together with its immediate previous and next available neighbors within the
|
||||
// series, in natural (season, episode) order. It is the bounded backing query
|
||||
// for Jellyfin's AdjacentTo request (Wholphin autoplay/skip): instead of
|
||||
// materializing every episode of the series, each neighbor is found with a
|
||||
// bounded index range scan backed by the episodes_series_season_episode_key
|
||||
// UNIQUE index (series_id, season_number, episode_number). Neighbors are
|
||||
// resolved across season boundaries, so the result is at most three rows.
|
||||
func (r *EpisodeRepository) ListAdjacentInSeries(ctx context.Context, seriesID string, seasonNumber, episodeNumber int) ([]*models.Episode, error) {
|
||||
query := `SELECT ` + episodeColumns + ` FROM (
|
||||
(SELECT ` + episodeColumns + `
|
||||
FROM episodes
|
||||
WHERE series_id = $1
|
||||
AND (season_number, episode_number) < ($2, $3)
|
||||
AND ` + episodeAvailabilityPredicate + `
|
||||
ORDER BY season_number DESC, episode_number DESC
|
||||
LIMIT 1)
|
||||
UNION ALL
|
||||
(SELECT ` + episodeColumns + `
|
||||
FROM episodes
|
||||
WHERE series_id = $1
|
||||
AND season_number = $2
|
||||
AND episode_number = $3
|
||||
AND ` + episodeAvailabilityPredicate + `)
|
||||
UNION ALL
|
||||
(SELECT ` + episodeColumns + `
|
||||
FROM episodes
|
||||
WHERE series_id = $1
|
||||
AND (season_number, episode_number) > ($2, $3)
|
||||
AND ` + episodeAvailabilityPredicate + `
|
||||
ORDER BY season_number ASC, episode_number ASC
|
||||
LIMIT 1)
|
||||
) AS adjacent
|
||||
ORDER BY season_number ASC, episode_number ASC`
|
||||
|
||||
rows, err := r.pool.Query(ctx, query, seriesID, seasonNumber, episodeNumber)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("listing adjacent episodes in series: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
return scanEpisodes(rows)
|
||||
}
|
||||
|
||||
// ListBySeasonID returns all episodes for a specific season row, ordered by
|
||||
// episode number.
|
||||
func (r *EpisodeRepository) ListBySeasonID(ctx context.Context, seasonID string) ([]*models.Episode, error) {
|
||||
|
||||
@@ -90,7 +90,7 @@ func (r *ItemRepository) SetLocalPoster(ctx context.Context, contentID, posterPa
|
||||
tag, err := r.pool.Exec(ctx, `
|
||||
UPDATE media_items
|
||||
SET poster_path = $2,
|
||||
poster_thumbhash = NULLIF($3, ''),
|
||||
poster_thumbhash = $3,
|
||||
updated_at = NOW()
|
||||
WHERE content_id = $1
|
||||
AND (poster_path IS NULL OR poster_path = '' OR poster_path LIKE $4 || '%')
|
||||
@@ -467,7 +467,7 @@ func (r *ItemRepository) upsert(ctx context.Context, execer itemExecer, item *mo
|
||||
type = EXCLUDED.type,
|
||||
title = EXCLUDED.title,
|
||||
sort_title = EXCLUDED.sort_title,
|
||||
default_metadata_language = COALESCE(NULLIF(media_items.default_metadata_language, ''), EXCLUDED.default_metadata_language),
|
||||
default_metadata_language = COALESCE(NULLIF(EXCLUDED.default_metadata_language, ''), media_items.default_metadata_language),
|
||||
original_title = EXCLUDED.original_title,
|
||||
year = EXCLUDED.year,
|
||||
genres = EXCLUDED.genres,
|
||||
@@ -1333,6 +1333,69 @@ func (r *ItemRepository) EnsureAccessible(ctx context.Context, contentID string,
|
||||
return nil
|
||||
}
|
||||
|
||||
// EnsureAccessibleIDs is the batch form of EnsureAccessible: it returns the set
|
||||
// of content IDs from the input that the viewer may access, applying the exact
|
||||
// same predicates (library allow/deny via the single JOIN form, max content
|
||||
// rating, excluded media types). An item is accessible iff at least one
|
||||
// media_items row — joined per library link when a library filter is active —
|
||||
// satisfies the predicates, matching EnsureAccessible's `LIMIT 1` semantics.
|
||||
// IDs absent from the returned map are not accessible (mirrors EnsureAccessible
|
||||
// returning ErrItemNotFound). Used by GetItemDetailsByIDs to avoid a per-item
|
||||
// EnsureAccessible fan-out.
|
||||
func (r *ItemRepository) EnsureAccessibleIDs(ctx context.Context, contentIDs []string, filter AccessFilter) (map[string]bool, error) {
|
||||
result := make(map[string]bool, len(contentIDs))
|
||||
if len(contentIDs) == 0 {
|
||||
return result, nil
|
||||
}
|
||||
// An empty (non-nil) allowlist means "no libraries allowed" — nothing is
|
||||
// accessible, exactly as EnsureAccessible returns ErrItemNotFound.
|
||||
if filter.AllowedLibraryIDs != nil && len(filter.AllowedLibraryIDs) == 0 {
|
||||
return result, nil
|
||||
}
|
||||
|
||||
var conditions []string
|
||||
var args []any
|
||||
argIdx := 1
|
||||
|
||||
fromClause := "media_items mi"
|
||||
conditions = append(conditions, fmt.Sprintf("mi.content_id = ANY($%d)", argIdx))
|
||||
args = append(args, contentIDs)
|
||||
argIdx++
|
||||
|
||||
needsLibJoin := filter.AllowedLibraryIDs != nil || len(filter.DisabledLibraryIDs) > 0
|
||||
if filter.AllowedLibraryIDs != nil {
|
||||
conditions = append(conditions, fmt.Sprintf("mil.media_folder_id = ANY($%d)", argIdx))
|
||||
args = append(args, filter.AllowedLibraryIDs)
|
||||
argIdx++
|
||||
}
|
||||
if len(filter.DisabledLibraryIDs) > 0 {
|
||||
conditions = append(conditions, fmt.Sprintf("NOT (mil.media_folder_id = ANY($%d))", argIdx))
|
||||
args = append(args, filter.DisabledLibraryIDs)
|
||||
argIdx++
|
||||
}
|
||||
if needsLibJoin {
|
||||
fromClause = "media_items mi JOIN media_item_libraries mil ON mi.content_id = mil.content_id"
|
||||
}
|
||||
|
||||
applyAccessFilter("mi", AccessFilter{MaxContentRating: filter.MaxContentRating, ExcludedMediaTypes: filter.ExcludedMediaTypes}, &conditions, &args, &argIdx)
|
||||
|
||||
query := fmt.Sprintf("SELECT DISTINCT mi.content_id FROM %s WHERE %s", fromClause, strings.Join(conditions, " AND "))
|
||||
rows, err := r.pool.Query(ctx, query, args...)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("checking items access: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
for rows.Next() {
|
||||
var id string
|
||||
if err := rows.Scan(&id); err != nil {
|
||||
return nil, fmt.Errorf("scanning item access row: %w", err)
|
||||
}
|
||||
result[id] = true
|
||||
}
|
||||
return result, rows.Err()
|
||||
}
|
||||
|
||||
// GetItemsInLibrary returns a membership map for the provided content IDs within
|
||||
// a single library. Episode content IDs are matched directly against
|
||||
// episode_libraries so compat callers can filter mixed item/episode batches.
|
||||
@@ -1607,7 +1670,7 @@ func (r *ItemRepository) UpdateArtworkIfSourceMatches(ctx context.Context, conte
|
||||
UPDATE media_items
|
||||
SET poster_path = $3,
|
||||
poster_source_path = $2,
|
||||
poster_thumbhash = NULLIF($4, ''),
|
||||
poster_thumbhash = $4,
|
||||
updated_at = NOW()
|
||||
WHERE content_id = $1
|
||||
AND poster_source_path = $2`
|
||||
@@ -1617,7 +1680,7 @@ func (r *ItemRepository) UpdateArtworkIfSourceMatches(ctx context.Context, conte
|
||||
UPDATE media_items
|
||||
SET backdrop_path = $3,
|
||||
backdrop_source_path = $2,
|
||||
backdrop_thumbhash = NULLIF($4, ''),
|
||||
backdrop_thumbhash = $4,
|
||||
updated_at = NOW()
|
||||
WHERE content_id = $1
|
||||
AND backdrop_source_path = $2`
|
||||
|
||||
@@ -732,14 +732,29 @@ func (r *LibraryCollectionRepository) ReplaceItems(ctx context.Context, collecti
|
||||
return fmt.Errorf("clearing collection items: %w", err)
|
||||
}
|
||||
|
||||
// Dedupe by media_item_id before inserting. Different source entries can
|
||||
// resolve to the same library item (e.g. an IMDb and a TMDB entry pointing
|
||||
// at the same local content_id, or one item present in several of the
|
||||
// collection's bound folders), and library_collection_items has a composite
|
||||
// PK (collection_id, media_item_id). Inserting the same media_item_id twice
|
||||
// would raise a duplicate-key violation that fails the whole sync. First
|
||||
// occurrence wins, preserving rank order. Position is renumbered over the
|
||||
// surviving rows so it stays dense.
|
||||
seen := make(map[string]struct{}, len(items))
|
||||
position := 0
|
||||
for _, item := range items {
|
||||
if _, dup := seen[item.MediaItemID]; dup {
|
||||
continue
|
||||
}
|
||||
seen[item.MediaItemID] = struct{}{}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
INSERT INTO library_collection_items (
|
||||
collection_id, media_item_id, position, source_rank
|
||||
) VALUES ($1, $2, $3, $4)
|
||||
`, collectionID, item.MediaItemID, item.Position, item.SourceRank); err != nil {
|
||||
`, collectionID, item.MediaItemID, position, item.SourceRank); err != nil {
|
||||
return fmt.Errorf("inserting collection item %s: %w", item.MediaItemID, err)
|
||||
}
|
||||
position++
|
||||
}
|
||||
|
||||
if _, err := tx.Exec(ctx, "UPDATE library_collections SET updated_at = NOW() WHERE id = $1", collectionID); err != nil {
|
||||
|
||||
@@ -339,7 +339,7 @@ func (s *LibraryCollectionService) syncMDBListCollection(ctx context.Context, co
|
||||
// Single batched library membership query covering EVERY candidate across
|
||||
// all entries (preserves the libraryID filter from the legacy
|
||||
// resolveMDBListEntry).
|
||||
libraryMembers, err := s.libraryItems.GetItemsInFolder(ctx, candidateIDs, collection.LibraryID)
|
||||
libraryMembers, err := s.libraryItems.GetItemsInFolders(ctx, candidateIDs, collection.LibraryIDs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -355,7 +355,7 @@ func (s *LibraryCollectionService) syncMDBListCollection(ctx context.Context, co
|
||||
scannedEntries = index + 1
|
||||
r, ok := resolvedByIndex[index]
|
||||
if !ok {
|
||||
warnings = append(warnings, fmt.Sprintf("No match in library %d for %s", collection.LibraryID, entry.Title))
|
||||
warnings = append(warnings, fmt.Sprintf("No match in libraries %v for %s", collection.LibraryIDs, entry.Title))
|
||||
continue
|
||||
}
|
||||
var chosen string
|
||||
@@ -366,7 +366,7 @@ func (s *LibraryCollectionService) syncMDBListCollection(ctx context.Context, co
|
||||
}
|
||||
}
|
||||
if chosen == "" {
|
||||
warnings = append(warnings, fmt.Sprintf("No match in library %d for %s", collection.LibraryID, entry.Title))
|
||||
warnings = append(warnings, fmt.Sprintf("No match in libraries %v for %s", collection.LibraryIDs, entry.Title))
|
||||
continue
|
||||
}
|
||||
matchedItems = append(matchedItems, LibraryCollectionItemInput{
|
||||
@@ -474,7 +474,7 @@ func (s *LibraryCollectionService) syncTMDBPresetCollection(ctx context.Context,
|
||||
|
||||
for i, entry := range results {
|
||||
scannedEntries = i + 1
|
||||
item, err := s.resolveTMDBEntry(ctx, collection.LibraryID, entry)
|
||||
item, err := s.resolveTMDBEntry(ctx, collection.LibraryIDs, entry)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -488,7 +488,7 @@ func (s *LibraryCollectionService) syncTMDBPresetCollection(ctx context.Context,
|
||||
"tvdb_id", entry.TVDBID,
|
||||
)
|
||||
unmatchedCount++
|
||||
warnings = append(warnings, fmt.Sprintf("No match in library %d for %s", collection.LibraryID, entry.Title))
|
||||
warnings = append(warnings, fmt.Sprintf("No match in libraries %v for %s", collection.LibraryIDs, entry.Title))
|
||||
continue
|
||||
}
|
||||
if firstRank, exists := seenContentIDs[item.ContentID]; exists {
|
||||
@@ -638,7 +638,7 @@ func (s *LibraryCollectionService) syncTMDBFranchiseCollection(ctx context.Conte
|
||||
|
||||
for i, entry := range results {
|
||||
scannedEntries = i + 1
|
||||
item, err := s.resolveTMDBEntry(ctx, collection.LibraryID, entry)
|
||||
item, err := s.resolveTMDBEntry(ctx, collection.LibraryIDs, entry)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -650,7 +650,7 @@ func (s *LibraryCollectionService) syncTMDBFranchiseCollection(ctx context.Conte
|
||||
"imdb_id", entry.IMDbID,
|
||||
)
|
||||
unmatchedCount++
|
||||
warnings = append(warnings, fmt.Sprintf("No match in library %d for %s", collection.LibraryID, entry.Title))
|
||||
warnings = append(warnings, fmt.Sprintf("No match in libraries %v for %s", collection.LibraryIDs, entry.Title))
|
||||
continue
|
||||
}
|
||||
if firstRank, exists := seenContentIDs[item.ContentID]; exists {
|
||||
@@ -813,7 +813,7 @@ func (s *LibraryCollectionService) syncTMDBDiscoverCollection(ctx context.Contex
|
||||
|
||||
for i, entry := range results {
|
||||
scannedEntries = i + 1
|
||||
item, err := s.resolveTMDBEntry(ctx, collection.LibraryID, entry)
|
||||
item, err := s.resolveTMDBEntry(ctx, collection.LibraryIDs, entry)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -827,7 +827,7 @@ func (s *LibraryCollectionService) syncTMDBDiscoverCollection(ctx context.Contex
|
||||
"tvdb_id", entry.TVDBID,
|
||||
)
|
||||
unmatchedCount++
|
||||
warnings = append(warnings, fmt.Sprintf("No match in library %d for %s", collection.LibraryID, entry.Title))
|
||||
warnings = append(warnings, fmt.Sprintf("No match in libraries %v for %s", collection.LibraryIDs, entry.Title))
|
||||
continue
|
||||
}
|
||||
if firstRank, exists := seenContentIDs[item.ContentID]; exists {
|
||||
@@ -975,13 +975,13 @@ func (s *LibraryCollectionService) syncTraktPresetCollection(ctx context.Context
|
||||
|
||||
for i, entry := range results {
|
||||
scannedEntries = i + 1
|
||||
item, err := s.resolveTraktEntry(ctx, collection.LibraryID, entry)
|
||||
item, err := s.resolveTraktEntry(ctx, collection.LibraryIDs, entry)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if item == nil {
|
||||
unmatchedCount++
|
||||
warnings = append(warnings, fmt.Sprintf("No match in library %d for %s", collection.LibraryID, entry.Title))
|
||||
warnings = append(warnings, fmt.Sprintf("No match in libraries %v for %s", collection.LibraryIDs, entry.Title))
|
||||
continue
|
||||
}
|
||||
if firstRank, exists := seenContentIDs[item.ContentID]; exists {
|
||||
@@ -1071,7 +1071,7 @@ func (s *LibraryCollectionService) recordFailedCollectionSync(ctx context.Contex
|
||||
|
||||
// resolveTMDBEntry finds a media item in the library matching a TMDB preset entry.
|
||||
// It tries TMDB ID, IMDb ID, and TVDB ID (for TV shows) to maximize match rate.
|
||||
func (s *LibraryCollectionService) resolveTMDBEntry(ctx context.Context, libraryID int, entry TMDBCollectionEntry) (*models.MediaItem, error) {
|
||||
func (s *LibraryCollectionService) resolveTMDBEntry(ctx context.Context, libraryIDs []int, entry TMDBCollectionEntry) (*models.MediaItem, error) {
|
||||
itemType := "movie"
|
||||
if entry.MediaType == "tv" {
|
||||
itemType = "series"
|
||||
@@ -1094,7 +1094,7 @@ func (s *LibraryCollectionService) resolveTMDBEntry(ctx context.Context, library
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
membership, err := s.libraryItems.GetItemsInFolder(ctx, []string{item.ContentID}, libraryID)
|
||||
membership, err := s.libraryItems.GetItemsInFolders(ctx, []string{item.ContentID}, libraryIDs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -1106,7 +1106,7 @@ func (s *LibraryCollectionService) resolveTMDBEntry(ctx context.Context, library
|
||||
}
|
||||
|
||||
// resolveTraktEntry finds a local item matching a Trakt discovery entry.
|
||||
func (s *LibraryCollectionService) resolveTraktEntry(ctx context.Context, libraryID int, entry TraktCollectionEntry) (*models.MediaItem, error) {
|
||||
func (s *LibraryCollectionService) resolveTraktEntry(ctx context.Context, libraryIDs []int, entry TraktCollectionEntry) (*models.MediaItem, error) {
|
||||
itemType := "movie"
|
||||
if entry.MediaType == "tv" {
|
||||
itemType = "series"
|
||||
@@ -1130,7 +1130,7 @@ func (s *LibraryCollectionService) resolveTraktEntry(ctx context.Context, librar
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
membership, err := s.libraryItems.GetItemsInFolder(ctx, candidates, libraryID)
|
||||
membership, err := s.libraryItems.GetItemsInFolders(ctx, candidates, libraryIDs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -199,6 +199,7 @@ func buildListNextUpQuery(q NextUpQuery, limit int) (string, []interface{}) {
|
||||
WHERE uwp2.user_id = $1
|
||||
AND uwp2.profile_id = $2
|
||||
AND uwp2.media_item_id = e2.content_id
|
||||
AND (uwp2.completed = TRUE OR uwp2.position_seconds > 0)
|
||||
)
|
||||
ORDER BY e2.season_number, e2.episode_number
|
||||
LIMIT 1
|
||||
|
||||
@@ -20,6 +20,7 @@ func TestBuildListNextUpQuery_PrefersRecentCompletedOverOlderPartialProgress(t *
|
||||
"uwp_ip.updated_at > ce.updated_at",
|
||||
"FROM user_history_hidden_items hhi",
|
||||
"uwp.updated_at <= hhi.hidden_before",
|
||||
"AND (uwp2.completed = TRUE OR uwp2.position_seconds > 0)",
|
||||
"ORDER BY e.series_id, uwp.updated_at DESC, e.season_number DESC, e.episode_number DESC",
|
||||
"LIMIT $3",
|
||||
}
|
||||
|
||||
@@ -115,8 +115,8 @@ const orphanedMediaItemSafetyConditions = `NOT EXISTS (
|
||||
WHERE uwl.media_item_id = mi.content_id
|
||||
)
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM public.watch_provider_favorite_items wpfi
|
||||
WHERE wpfi.media_item_id = mi.content_id
|
||||
SELECT 1 FROM public.watch_provider_list_items wpli
|
||||
WHERE wpli.media_item_id = mi.content_id
|
||||
)
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM public.watch_provider_history_exports wphe
|
||||
|
||||
@@ -24,6 +24,7 @@ func TestOrphanedProvisionalPredicatePreservesDurableMediaItemReferences(t *test
|
||||
"public.user_personal_collection_items upci WHERE upci.media_item_id = mi.content_id",
|
||||
"public.user_series_playback_preferences uspp WHERE uspp.series_id = mi.content_id",
|
||||
"public.user_subtitle_preferences usp WHERE usp.series_id = mi.content_id",
|
||||
"public.watch_provider_list_items wpli WHERE wpli.media_item_id = mi.content_id",
|
||||
} {
|
||||
if !strings.Contains(predicate, normalizePredicateSQL(want)) {
|
||||
t.Fatalf("cleanup predicate missing durable reference guard %q", want)
|
||||
@@ -33,6 +34,7 @@ func TestOrphanedProvisionalPredicatePreservesDurableMediaItemReferences(t *test
|
||||
"abs_collection_items",
|
||||
"abs_playlist_items",
|
||||
"abs_playlists",
|
||||
"watch_provider_favorite_items",
|
||||
} {
|
||||
if strings.Contains(predicate, droppedTable) {
|
||||
t.Fatalf("cleanup predicate must not reference dropped legacy table %q", droppedTable)
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
package catalog
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
)
|
||||
|
||||
func TestConflictingExternalID(t *testing.T) {
|
||||
p := models.Person{TmdbID: "111", ImdbID: "nm222"}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
constraint string
|
||||
wantField string
|
||||
wantValue string
|
||||
wantOK bool
|
||||
}{
|
||||
{"tmdb", "idx_people_tmdb_id", "tmdb_id", "111", true},
|
||||
{"imdb", "idx_people_imdb_id", "imdb_id", "nm222", true},
|
||||
{"unknown constraint", "idx_people_name", "", "", false},
|
||||
{"empty constraint", "", "", "", false},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
field, value, ok := conflictingExternalID(tt.constraint, p)
|
||||
if field != tt.wantField || value != tt.wantValue || ok != tt.wantOK {
|
||||
t.Fatalf("conflictingExternalID(%q) = (%q,%q,%v), want (%q,%q,%v)",
|
||||
tt.constraint, field, value, ok, tt.wantField, tt.wantValue, tt.wantOK)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// A blank id must never be reported as the conflict source.
|
||||
if _, _, ok := conflictingExternalID("idx_people_tmdb_id", models.Person{}); ok {
|
||||
t.Fatal("expected ok=false when the conflicting field is empty")
|
||||
}
|
||||
}
|
||||
|
||||
func TestExternalIDsCompatible(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
a, b models.Person
|
||||
want bool
|
||||
}{
|
||||
{
|
||||
name: "tmdb-only vs imdb-only (the core dedup case)",
|
||||
a: models.Person{TmdbID: "111", ImdbID: "nm9"},
|
||||
b: models.Person{ImdbID: "nm9"},
|
||||
want: true,
|
||||
},
|
||||
{
|
||||
name: "identical ids",
|
||||
a: models.Person{TmdbID: "111", ImdbID: "nm9"},
|
||||
b: models.Person{TmdbID: "111", ImdbID: "nm9"},
|
||||
want: true,
|
||||
},
|
||||
{
|
||||
name: "conflicting tmdb ids",
|
||||
a: models.Person{TmdbID: "111", ImdbID: "nm9"},
|
||||
b: models.Person{TmdbID: "999", ImdbID: "nm9"},
|
||||
want: false,
|
||||
},
|
||||
{
|
||||
name: "conflicting tvdb ids",
|
||||
a: models.Person{ImdbID: "nm9", TvdbID: "5"},
|
||||
b: models.Person{ImdbID: "nm9", TvdbID: "6"},
|
||||
want: false,
|
||||
},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if got := externalIDsCompatible(tt.a, tt.b); got != tt.want {
|
||||
t.Fatalf("externalIDsCompatible = %v, want %v", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCanMergePeople(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
a, b models.Person
|
||||
want bool
|
||||
}{
|
||||
{
|
||||
name: "same name, compatible ids -> merge",
|
||||
a: models.Person{Name: "Jane Doe", TmdbID: "111"},
|
||||
b: models.Person{Name: "Jane Doe", ImdbID: "nm9"},
|
||||
want: true,
|
||||
},
|
||||
{
|
||||
name: "case/space-insensitive name match -> merge",
|
||||
a: models.Person{Name: " jane doe ", TmdbID: "111"},
|
||||
b: models.Person{Name: "Jane Doe", ImdbID: "nm9"},
|
||||
want: true,
|
||||
},
|
||||
{
|
||||
name: "different names, compatible ids -> do not merge",
|
||||
a: models.Person{Name: "Jane Doe", TmdbID: "111"},
|
||||
b: models.Person{Name: "John Smith", ImdbID: "nm9"},
|
||||
want: false,
|
||||
},
|
||||
{
|
||||
name: "same name but contradictory ids -> do not merge",
|
||||
a: models.Person{Name: "Jane Doe", TmdbID: "111"},
|
||||
b: models.Person{Name: "Jane Doe", TmdbID: "999"},
|
||||
want: false,
|
||||
},
|
||||
{
|
||||
name: "empty name on one side -> do not merge",
|
||||
a: models.Person{Name: "", TmdbID: "111"},
|
||||
b: models.Person{Name: "Jane Doe", ImdbID: "nm9"},
|
||||
want: false,
|
||||
},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if got := canMergePeople(tt.a, tt.b); got != tt.want {
|
||||
t.Fatalf("canMergePeople = %v, want %v", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergePersonFields(t *testing.T) {
|
||||
birth := time.Date(1980, 1, 2, 0, 0, 0, 0, time.UTC)
|
||||
|
||||
t.Run("backfills only empty fields, survivor wins", func(t *testing.T) {
|
||||
dst := models.Person{
|
||||
Name: "Jane Doe",
|
||||
TmdbID: "111",
|
||||
Bio: "survivor bio",
|
||||
}
|
||||
src := models.Person{
|
||||
Name: "Janey",
|
||||
TmdbID: "999", // must NOT overwrite a populated survivor id
|
||||
ImdbID: "nm9", // fills the gap
|
||||
Bio: "partner bio",
|
||||
Birthplace: "Springfield",
|
||||
BirthDate: &birth,
|
||||
PhotoPath: "/p.jpg",
|
||||
}
|
||||
mergePersonFields(&dst, src)
|
||||
|
||||
if dst.Name != "Jane Doe" {
|
||||
t.Errorf("Name overwritten: %q", dst.Name)
|
||||
}
|
||||
if dst.TmdbID != "111" {
|
||||
t.Errorf("populated TmdbID overwritten: %q", dst.TmdbID)
|
||||
}
|
||||
if dst.ImdbID != "nm9" {
|
||||
t.Errorf("empty ImdbID not backfilled: %q", dst.ImdbID)
|
||||
}
|
||||
if dst.Bio != "survivor bio" {
|
||||
t.Errorf("populated Bio overwritten: %q", dst.Bio)
|
||||
}
|
||||
if dst.Birthplace != "Springfield" {
|
||||
t.Errorf("empty Birthplace not backfilled: %q", dst.Birthplace)
|
||||
}
|
||||
if dst.BirthDate == nil || !dst.BirthDate.Equal(birth) {
|
||||
t.Errorf("nil BirthDate not backfilled: %v", dst.BirthDate)
|
||||
}
|
||||
if dst.PhotoPath != "/p.jpg" {
|
||||
t.Errorf("empty PhotoPath not backfilled: %q", dst.PhotoPath)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("backfills name when survivor is blank", func(t *testing.T) {
|
||||
dst := models.Person{TmdbID: "111"}
|
||||
src := models.Person{Name: "Jane Doe"}
|
||||
mergePersonFields(&dst, src)
|
||||
if dst.Name != "Jane Doe" {
|
||||
t.Errorf("blank survivor Name not backfilled: %q", dst.Name)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("photo trio is folded together", func(t *testing.T) {
|
||||
dst := models.Person{} // no photo at all
|
||||
src := models.Person{
|
||||
PhotoPath: "/p.jpg",
|
||||
PhotoSourcePath: "tmdb://p",
|
||||
PhotoThumbhash: "hash",
|
||||
}
|
||||
mergePersonFields(&dst, src)
|
||||
if dst.PhotoPath != "/p.jpg" || dst.PhotoSourcePath != "tmdb://p" || dst.PhotoThumbhash != "hash" {
|
||||
t.Errorf("photo trio not folded together: %+v", dst)
|
||||
}
|
||||
|
||||
// A survivor that already has a photo keeps its own trio intact.
|
||||
keep := models.Person{PhotoPath: "/keep.jpg"}
|
||||
mergePersonFields(&keep, src)
|
||||
if keep.PhotoPath != "/keep.jpg" || keep.PhotoSourcePath != "" || keep.PhotoThumbhash != "" {
|
||||
t.Errorf("survivor photo trio was disturbed: %+v", keep)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestExternalIDValueAndSet(t *testing.T) {
|
||||
p := models.Person{TmdbID: "111", ImdbID: "nm9", TvdbID: "5", PlexGUID: "g"}
|
||||
|
||||
for field, want := range map[string]string{
|
||||
"tmdb_id": "111", "imdb_id": "nm9", "tvdb_id": "5", "plex_guid": "g", "nope": "",
|
||||
} {
|
||||
if got := externalIDValue(p, field); got != want {
|
||||
t.Errorf("externalIDValue(%q) = %q, want %q", field, got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// Setting a field to a new value replaces only that column.
|
||||
updated := p
|
||||
setExternalIDField(&updated, "imdb_id", "nm12345")
|
||||
if updated.ImdbID != "nm12345" {
|
||||
t.Errorf("setExternalIDField did not set imdb_id: %q", updated.ImdbID)
|
||||
}
|
||||
if updated.TmdbID != "111" {
|
||||
t.Errorf("setExternalIDField touched the wrong field: %q", updated.TmdbID)
|
||||
}
|
||||
|
||||
// Setting a field to "" blanks it, matching the old clear behavior.
|
||||
cleared := p
|
||||
setExternalIDField(&cleared, "imdb_id", "")
|
||||
if cleared.ImdbID != "" {
|
||||
t.Errorf("setExternalIDField did not blank imdb_id: %q", cleared.ImdbID)
|
||||
}
|
||||
if cleared.TmdbID != "111" {
|
||||
t.Errorf("setExternalIDField touched the wrong field: %q", cleared.TmdbID)
|
||||
}
|
||||
}
|
||||
+387
-10
@@ -2,7 +2,9 @@ package catalog
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -493,6 +495,13 @@ func (r *PersonRepository) GetByName(ctx context.Context, name string) (*models.
|
||||
}
|
||||
|
||||
// Search finds persons by name substring (case-insensitive), ordered by name.
|
||||
//
|
||||
// The predicate is `name ILIKE '%term%'` rather than `LOWER(name) LIKE ...` so
|
||||
// the pg_trgm GIN index idx_people_name_trgm can serve it: for a rare 3+ char
|
||||
// term that index turns a ~300-400ms full name-index scan into a ~10ms bitmap
|
||||
// scan. ILIKE is itself case-insensitive, so this stays equivalent to the prior
|
||||
// LOWER(name) comparison (including its existing treatment of % and _ in the
|
||||
// term as LIKE wildcards).
|
||||
func (r *PersonRepository) Search(ctx context.Context, query string, limit int) ([]models.Person, error) {
|
||||
if limit <= 0 {
|
||||
limit = 20
|
||||
@@ -500,7 +509,7 @@ func (r *PersonRepository) Search(ctx context.Context, query string, limit int)
|
||||
rows, err := r.pool.Query(ctx, `
|
||||
SELECT id, name, sort_name, bio, birth_date, death_date, birthplace, homepage,
|
||||
photo_path, photo_source_path, photo_thumbhash, tmdb_id, imdb_id, tvdb_id, plex_guid, created_at, updated_at
|
||||
FROM people WHERE LOWER(name) LIKE '%' || LOWER($1) || '%'
|
||||
FROM people WHERE name ILIKE '%' || $1 || '%'
|
||||
ORDER BY name LIMIT $2`, query, limit,
|
||||
)
|
||||
if err != nil {
|
||||
@@ -521,15 +530,114 @@ func (r *PersonRepository) Search(ctx context.Context, query string, limit int)
|
||||
return people, rows.Err()
|
||||
}
|
||||
|
||||
// maxPersonConflictResolutions bounds how many external-id collisions a single
|
||||
// Update is allowed to resolve before giving up. One per unique external-id
|
||||
// field is the natural ceiling (today only tmdb_id and imdb_id are unique).
|
||||
const maxPersonConflictResolutions = 4
|
||||
|
||||
// Update updates all non-key fields on a person.
|
||||
//
|
||||
// A person's resolved external ids (tmdb_id / imdb_id) can collide with another
|
||||
// row that turns out to be the same human discovered through a different id —
|
||||
// e.g. one credit ingested with only a tmdb_id and another with only an imdb_id,
|
||||
// later reconciled when a refresh fills in the missing cross-id. Rather than
|
||||
// failing that write (SQLSTATE 23505) and letting the refresh worker retry the
|
||||
// same row forever, Update detects the collision and resolves it: it merges the
|
||||
// two people when they are the same human, or drops just the conflicting id when
|
||||
// they are not, then retries the write so updated_at advances and the row leaves
|
||||
// the stale set.
|
||||
//
|
||||
// The common, no-collision path runs as a single plain transaction. Only when a
|
||||
// 23505 surfaces does Update fall back to updateResolvingConflicts, which does
|
||||
// the whole reconciliation — merge or drop, plus the field write and reindex —
|
||||
// inside one transaction so it commits atomically (or not at all). Returns
|
||||
// pgx.ErrNoRows if the row no longer exists (e.g. merged away concurrently).
|
||||
func (r *PersonRepository) Update(ctx context.Context, p models.Person) error {
|
||||
err := r.applyUpdate(ctx, p)
|
||||
if err == nil || !isDuplicateKeyError(err) {
|
||||
return err
|
||||
}
|
||||
return r.updateResolvingConflicts(ctx, p)
|
||||
}
|
||||
|
||||
// applyUpdate writes all non-key fields on a person and enqueues a search
|
||||
// reindex for the items they appear in, all in one transaction.
|
||||
func (r *PersonRepository) applyUpdate(ctx context.Context, p models.Person) error {
|
||||
tx, err := r.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("begin person update tx: %w", err)
|
||||
}
|
||||
defer func() { _ = tx.Rollback(ctx) }()
|
||||
|
||||
_, err = tx.Exec(ctx, `
|
||||
affected, err := execPersonUpdate(ctx, tx, p)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if affected == 0 {
|
||||
// The row was deleted out from under us (e.g. merged away concurrently).
|
||||
return pgx.ErrNoRows
|
||||
}
|
||||
if err := reindexPersonItems(ctx, tx, p.ID); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return fmt.Errorf("commit person update tx: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// updateResolvingConflicts performs the person write inside a single transaction,
|
||||
// reconciling external-id collisions (merge or drop) as they arise. The write is
|
||||
// retried after each resolution via a savepoint so a failed attempt does not
|
||||
// poison the transaction; the whole thing commits atomically once the write
|
||||
// lands. Bounded by maxPersonConflictResolutions so it cannot spin.
|
||||
func (r *PersonRepository) updateResolvingConflicts(ctx context.Context, p models.Person) error {
|
||||
tx, err := r.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("begin person merge tx: %w", err)
|
||||
}
|
||||
defer func() { _ = tx.Rollback(ctx) }()
|
||||
|
||||
var affected int64
|
||||
for attempt := 0; ; attempt++ {
|
||||
affected, err = tryPersonUpdate(ctx, tx, p)
|
||||
if err == nil {
|
||||
break
|
||||
}
|
||||
if !isDuplicateKeyError(err) || attempt >= maxPersonConflictResolutions {
|
||||
return err
|
||||
}
|
||||
field, value, ok := conflictingExternalID(extractConstraint(err), p)
|
||||
if !ok {
|
||||
// Unique violation on a constraint we do not know how to reconcile.
|
||||
return err
|
||||
}
|
||||
resolved, resolveErr := r.resolveExternalIDConflict(ctx, tx, &p, field, value)
|
||||
if resolveErr != nil {
|
||||
return resolveErr
|
||||
}
|
||||
if !resolved {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
if affected == 0 {
|
||||
// Survivor was deleted concurrently (e.g. merged into another row); there
|
||||
// is nothing left to persist for this id.
|
||||
return pgx.ErrNoRows
|
||||
}
|
||||
if err := reindexPersonItems(ctx, tx, p.ID); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return fmt.Errorf("commit person merge tx: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// execPersonUpdate writes all non-key person fields and returns the rows affected.
|
||||
func execPersonUpdate(ctx context.Context, tx pgx.Tx, p models.Person) (int64, error) {
|
||||
tag, err := tx.Exec(ctx, `
|
||||
UPDATE people SET name=$2, sort_name=$3, bio=$4, birth_date=$5, death_date=$6,
|
||||
birthplace=$7, homepage=$8, photo_path=$9, photo_source_path=$10, photo_thumbhash=$11,
|
||||
tmdb_id=$12, imdb_id=$13, tvdb_id=$14, plex_guid=$15, updated_at=now()
|
||||
@@ -539,30 +647,299 @@ func (r *PersonRepository) Update(ctx context.Context, p models.Person) error {
|
||||
p.TmdbID, p.ImdbID, p.TvdbID, p.PlexGUID,
|
||||
)
|
||||
if err != nil {
|
||||
return err
|
||||
return 0, err
|
||||
}
|
||||
return tag.RowsAffected(), nil
|
||||
}
|
||||
|
||||
// tryPersonUpdate runs execPersonUpdate inside a savepoint so a unique violation
|
||||
// rolls back only the attempt, leaving the surrounding transaction usable.
|
||||
func tryPersonUpdate(ctx context.Context, tx pgx.Tx, p models.Person) (int64, error) {
|
||||
sp, err := tx.Begin(ctx)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("begin person update savepoint: %w", err)
|
||||
}
|
||||
affected, err := execPersonUpdate(ctx, sp, p)
|
||||
if err != nil {
|
||||
_ = sp.Rollback(ctx)
|
||||
return 0, err
|
||||
}
|
||||
if err := sp.Commit(ctx); err != nil {
|
||||
return 0, fmt.Errorf("release person update savepoint: %w", err)
|
||||
}
|
||||
return affected, nil
|
||||
}
|
||||
|
||||
// reindexPersonItems enqueues a catalog-search reindex for every item the person
|
||||
// appears in. After a merge the survivor's link set already includes the merged
|
||||
// person's items, so this covers all items whose denormalized people text changed.
|
||||
func reindexPersonItems(ctx context.Context, tx pgx.Tx, personID int64) error {
|
||||
rows, err := tx.Query(ctx, `
|
||||
SELECT DISTINCT content_id
|
||||
FROM item_people
|
||||
WHERE person_id = $1
|
||||
`, p.ID)
|
||||
`, personID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("listing items linked to person %d: %w", p.ID, err)
|
||||
return fmt.Errorf("listing items linked to person %d: %w", personID, err)
|
||||
}
|
||||
contentIDs, err := pgx.CollectRows(rows, pgx.RowTo[string])
|
||||
if err != nil {
|
||||
return fmt.Errorf("collecting items linked to person %d: %w", p.ID, err)
|
||||
return fmt.Errorf("collecting items linked to person %d: %w", personID, err)
|
||||
}
|
||||
if err := EnqueueSearchIndexUpserts(ctx, tx, contentIDs); err != nil {
|
||||
return fmt.Errorf("enqueueing catalog search person update for person %d: %w", p.ID, err)
|
||||
}
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return fmt.Errorf("commit person update tx: %w", err)
|
||||
return fmt.Errorf("enqueueing catalog search person update for person %d: %w", personID, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// resolveExternalIDConflict reconciles a unique-violation on (field=value) raised
|
||||
// while updating *p, operating within the caller's transaction tx. It mutates *p
|
||||
// in place (folding in the partner's data, or dropping the conflicting id) and
|
||||
// reports whether the caller should retry the write. resolved=false means the
|
||||
// conflict could not be reconciled and the original error should be surfaced.
|
||||
func (r *PersonRepository) resolveExternalIDConflict(
|
||||
ctx context.Context,
|
||||
tx pgx.Tx,
|
||||
p *models.Person,
|
||||
field string,
|
||||
value string,
|
||||
) (resolved bool, err error) {
|
||||
if value == "" {
|
||||
return false, nil
|
||||
}
|
||||
|
||||
// Identify the row that currently owns the colliding id.
|
||||
var partnerID int64
|
||||
lookup := fmt.Sprintf("SELECT id FROM people WHERE %s = $1", field)
|
||||
switch scanErr := tx.QueryRow(ctx, lookup, value).Scan(&partnerID); {
|
||||
case errors.Is(scanErr, pgx.ErrNoRows):
|
||||
// The conflicting owner vanished (concurrent merge); retry the plain write.
|
||||
return true, nil
|
||||
case scanErr != nil:
|
||||
return false, fmt.Errorf("locate person owning %s=%q: %w", field, value, scanErr)
|
||||
}
|
||||
if partnerID == p.ID {
|
||||
// Self-conflict is impossible under the unique index; bail rather than loop.
|
||||
return false, nil
|
||||
}
|
||||
|
||||
// Lock both rows in a deterministic order so concurrent merges cannot deadlock.
|
||||
people, err := scanPeopleByIDsForUpdate(ctx, tx, p.ID, partnerID)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if _, ok := people[p.ID]; !ok {
|
||||
// The survivor was deleted under us (concurrent merge); retry the plain
|
||||
// write, which will affect zero rows and surface as pgx.ErrNoRows.
|
||||
return true, nil
|
||||
}
|
||||
partner, ok := people[partnerID]
|
||||
if !ok {
|
||||
// Partner already gone; retry the plain write.
|
||||
return true, nil
|
||||
}
|
||||
// Re-confirm the partner still holds the colliding value after locking.
|
||||
if externalIDValue(partner, field) != value {
|
||||
return true, nil
|
||||
}
|
||||
|
||||
if !canMergePeople(*p, partner) {
|
||||
// Not confidently the same human (contradictory ids, or names disagree):
|
||||
// restore the survivor's currently-persisted value for this field so the
|
||||
// retried write becomes a no-op on that column. This lets the write commit
|
||||
// instead of looping forever, without destructively deleting a
|
||||
// possibly-distinct person and without blanking an id the survivor already
|
||||
// held (blanking would silently drop a previously-valid provider id, e.g.
|
||||
// on the admin PATCH path that mutates an existing id into a colliding one).
|
||||
// Writing the row's own current value back can never violate the unique
|
||||
// index, so this field will not re-trigger the conflict.
|
||||
setExternalIDField(p, field, externalIDValue(people[p.ID], field))
|
||||
slog.Warn("person merge: declining to merge, preserving survivor's existing id",
|
||||
"person_id", p.ID, "partner_id", partnerID, "field", field, "value", value,
|
||||
"person_name", p.Name, "partner_name", partner.Name)
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// Fold the partner's ids and any metadata this write lacks onto the survivor,
|
||||
// so the retried write persists data only the partner had.
|
||||
mergePersonFields(p, partner)
|
||||
|
||||
// Repoint the partner's credits onto the survivor, skipping links that would
|
||||
// duplicate an existing (content_id, kind, character) credit on the survivor.
|
||||
if _, err := tx.Exec(ctx, `
|
||||
UPDATE item_people ip SET person_id = $1
|
||||
WHERE ip.person_id = $2
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM item_people e
|
||||
WHERE e.person_id = $1
|
||||
AND e.content_id = ip.content_id
|
||||
AND e.kind = ip.kind
|
||||
AND e.character = ip.character
|
||||
)`, p.ID, partnerID); err != nil {
|
||||
return false, fmt.Errorf("repoint item_people from %d to %d: %w", partnerID, p.ID, err)
|
||||
}
|
||||
|
||||
// Delete the partner; any leftover duplicate credits cascade away.
|
||||
if _, err := tx.Exec(ctx, `DELETE FROM people WHERE id = $1`, partnerID); err != nil {
|
||||
return false, fmt.Errorf("delete merged person %d: %w", partnerID, err)
|
||||
}
|
||||
|
||||
slog.Info("person merge: folded duplicate person",
|
||||
"survivor_id", p.ID, "merged_id", partnerID, "field", field, "value", value)
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// scanPeopleByIDsForUpdate loads the given people, locking them FOR UPDATE in
|
||||
// ascending id order so concurrent merges acquire locks consistently.
|
||||
func scanPeopleByIDsForUpdate(ctx context.Context, tx pgx.Tx, ids ...int64) (map[int64]models.Person, error) {
|
||||
rows, err := tx.Query(ctx, `
|
||||
SELECT id, name, sort_name, bio, birth_date, death_date, birthplace, homepage,
|
||||
photo_path, photo_source_path, photo_thumbhash, tmdb_id, imdb_id, tvdb_id, plex_guid, created_at, updated_at
|
||||
FROM people
|
||||
WHERE id = ANY($1)
|
||||
ORDER BY id
|
||||
FOR UPDATE`, ids)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("lock people for merge: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
out := make(map[int64]models.Person, len(ids))
|
||||
for rows.Next() {
|
||||
var p models.Person
|
||||
if err := rows.Scan(&p.ID, &p.Name, &p.SortName, &p.Bio, &p.BirthDate, &p.DeathDate, &p.Birthplace, &p.Homepage,
|
||||
&p.PhotoPath, &p.PhotoSourcePath, &p.PhotoThumbhash, &p.TmdbID, &p.ImdbID, &p.TvdbID, &p.PlexGUID, &p.CreatedAt, &p.UpdatedAt,
|
||||
); err != nil {
|
||||
return nil, fmt.Errorf("scan locked person: %w", err)
|
||||
}
|
||||
out[p.ID] = p
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("iterate locked people: %w", err)
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// conflictingExternalID maps a violated unique constraint to the person field
|
||||
// and value that caused it. ok is false for constraints Update cannot reconcile.
|
||||
func conflictingExternalID(constraint string, p models.Person) (field, value string, ok bool) {
|
||||
switch constraint {
|
||||
case "idx_people_tmdb_id":
|
||||
return "tmdb_id", p.TmdbID, p.TmdbID != ""
|
||||
case "idx_people_imdb_id":
|
||||
return "imdb_id", p.ImdbID, p.ImdbID != ""
|
||||
default:
|
||||
return "", "", false
|
||||
}
|
||||
}
|
||||
|
||||
// externalIDValue returns the value of the named external-id column on p.
|
||||
func externalIDValue(p models.Person, field string) string {
|
||||
switch field {
|
||||
case "tmdb_id":
|
||||
return p.TmdbID
|
||||
case "imdb_id":
|
||||
return p.ImdbID
|
||||
case "tvdb_id":
|
||||
return p.TvdbID
|
||||
case "plex_guid":
|
||||
return p.PlexGUID
|
||||
default:
|
||||
return ""
|
||||
}
|
||||
}
|
||||
|
||||
// setExternalIDField sets the named external-id column on p to value.
|
||||
func setExternalIDField(p *models.Person, field, value string) {
|
||||
switch field {
|
||||
case "tmdb_id":
|
||||
p.TmdbID = value
|
||||
case "imdb_id":
|
||||
p.ImdbID = value
|
||||
case "tvdb_id":
|
||||
p.TvdbID = value
|
||||
case "plex_guid":
|
||||
p.PlexGUID = value
|
||||
}
|
||||
}
|
||||
|
||||
// externalIDsCompatible reports whether a and b can be the same person: no
|
||||
// external-id field where both hold a different non-empty value. A field where
|
||||
// one side is empty (or both match) is compatible.
|
||||
func externalIDsCompatible(a, b models.Person) bool {
|
||||
return idsCompatible(a.TmdbID, b.TmdbID) &&
|
||||
idsCompatible(a.ImdbID, b.ImdbID) &&
|
||||
idsCompatible(a.TvdbID, b.TvdbID) &&
|
||||
idsCompatible(a.PlexGUID, b.PlexGUID)
|
||||
}
|
||||
|
||||
func idsCompatible(x, y string) bool {
|
||||
return x == "" || y == "" || x == y
|
||||
}
|
||||
|
||||
// canMergePeople reports whether two rows that collided on a shared external id
|
||||
// are confidently the same human and may be folded together. A shared id is a
|
||||
// strong signal, but providers occasionally hand the same id to two distinct
|
||||
// people; requiring the names to agree as well guards against destructively
|
||||
// deleting a genuinely different person. When this returns false the caller
|
||||
// drops the conflicting id instead (non-destructive).
|
||||
func canMergePeople(a, b models.Person) bool {
|
||||
return externalIDsCompatible(a, b) && personNamesMatch(a.Name, b.Name)
|
||||
}
|
||||
|
||||
func personNamesMatch(a, b string) bool {
|
||||
a = strings.TrimSpace(a)
|
||||
b = strings.TrimSpace(b)
|
||||
if a == "" || b == "" {
|
||||
return false
|
||||
}
|
||||
return strings.EqualFold(a, b)
|
||||
}
|
||||
|
||||
// mergePersonFields fills empty external ids and metadata on dst from src,
|
||||
// preserving dst's existing values. dst is the refreshed survivor, so its
|
||||
// populated fields win; src (the row being merged away) only backfills gaps.
|
||||
func mergePersonFields(dst *models.Person, src models.Person) {
|
||||
if dst.Name == "" {
|
||||
dst.Name = src.Name
|
||||
}
|
||||
if dst.TmdbID == "" {
|
||||
dst.TmdbID = src.TmdbID
|
||||
}
|
||||
if dst.ImdbID == "" {
|
||||
dst.ImdbID = src.ImdbID
|
||||
}
|
||||
if dst.TvdbID == "" {
|
||||
dst.TvdbID = src.TvdbID
|
||||
}
|
||||
if dst.PlexGUID == "" {
|
||||
dst.PlexGUID = src.PlexGUID
|
||||
}
|
||||
if dst.Bio == "" {
|
||||
dst.Bio = src.Bio
|
||||
}
|
||||
if dst.Birthplace == "" {
|
||||
dst.Birthplace = src.Birthplace
|
||||
}
|
||||
if dst.Homepage == "" {
|
||||
dst.Homepage = src.Homepage
|
||||
}
|
||||
if dst.SortName == "" {
|
||||
dst.SortName = src.SortName
|
||||
}
|
||||
if dst.BirthDate == nil {
|
||||
dst.BirthDate = src.BirthDate
|
||||
}
|
||||
if dst.DeathDate == nil {
|
||||
dst.DeathDate = src.DeathDate
|
||||
}
|
||||
if dst.PhotoPath == "" {
|
||||
dst.PhotoPath = src.PhotoPath
|
||||
dst.PhotoSourcePath = src.PhotoSourcePath
|
||||
dst.PhotoThumbhash = src.PhotoThumbhash
|
||||
}
|
||||
}
|
||||
|
||||
func (r *PersonRepository) UpdatePhotoIfSourceMatches(ctx context.Context, personID int64, sourcePath, cachedPath, thumbhash string) (bool, error) {
|
||||
if r == nil || r.pool == nil {
|
||||
return false, pgx.ErrNoRows
|
||||
|
||||
@@ -258,7 +258,7 @@ func (i *CatalogSearchIndexer) Rebuild(ctx context.Context, progress SearchIndex
|
||||
if err := client.WaitTask(ctx, taskID); err != nil {
|
||||
return stats, err
|
||||
}
|
||||
taskID, err = client.UpdateSettings(ctx, buildIndexUID, catalogSearchMeilisearchSettings(settings.Embedder))
|
||||
taskID, err = client.UpdateSettings(ctx, buildIndexUID, catalogSearchMeilisearchSettings(settings.Embedder, settings.SemanticEnabled))
|
||||
if err != nil {
|
||||
return stats, err
|
||||
}
|
||||
@@ -438,12 +438,8 @@ func (i *CatalogSearchIndexer) CheckConnection(ctx context.Context, settings Cat
|
||||
return client.Health(ctx)
|
||||
}
|
||||
|
||||
func catalogSearchMeilisearchSettings(embedder string) map[string]any {
|
||||
embedder, err := NormalizeCatalogSearchEmbedderName(embedder)
|
||||
if err != nil {
|
||||
embedder = DefaultMeilisearchEmbedder
|
||||
}
|
||||
return map[string]any{
|
||||
func catalogSearchMeilisearchSettings(embedder string, semanticEnabled bool) map[string]any {
|
||||
settings := map[string]any{
|
||||
"displayedAttributes": []string{"content_id", "type"},
|
||||
"filterableAttributes": []string{"type"},
|
||||
"searchableAttributes": []string{
|
||||
@@ -462,8 +458,15 @@ func catalogSearchMeilisearchSettings(embedder string) map[string]any {
|
||||
"pagination": map[string]any{
|
||||
"maxTotalHits": meilisearchDefaultCandidateScanCap,
|
||||
},
|
||||
"embedders": catalogSearchMeilisearchEmbedderSettings(embedder),
|
||||
}
|
||||
if semanticEnabled {
|
||||
embedder, err := NormalizeCatalogSearchEmbedderName(embedder)
|
||||
if err != nil {
|
||||
embedder = DefaultMeilisearchEmbedder
|
||||
}
|
||||
settings["embedders"] = catalogSearchMeilisearchEmbedderSettings(embedder)
|
||||
}
|
||||
return settings
|
||||
}
|
||||
|
||||
func coalesceSearchIndexEvents(events []SearchIndexEvent) (upsertIDs []string, deleteIDs []string) {
|
||||
|
||||
@@ -127,6 +127,29 @@ func TestCatalogSearchDocumentVectorsUseEmbedderAndOptOutMissing(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestCatalogSearchMeilisearchSettingsOmitEmbeddersWhenSemanticDisabled(t *testing.T) {
|
||||
settings := catalogSearchMeilisearchSettings("silo_recommendations", false)
|
||||
|
||||
if _, ok := settings["embedders"]; ok {
|
||||
t.Fatalf("semantic-disabled settings should not include embedders: %#v", settings["embedders"])
|
||||
}
|
||||
if _, ok := settings["searchableAttributes"]; !ok {
|
||||
t.Fatal("semantic-disabled settings should still configure keyword searchable attributes")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCatalogSearchMeilisearchSettingsIncludeEmbeddersWhenSemanticEnabled(t *testing.T) {
|
||||
settings := catalogSearchMeilisearchSettings("custom_embedder", true)
|
||||
|
||||
embedders, ok := settings["embedders"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("embedders = %#v, want map[string]any", settings["embedders"])
|
||||
}
|
||||
if _, ok := embedders["custom_embedder"]; !ok {
|
||||
t.Fatalf("embedders = %#v, want custom_embedder", embedders)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCatalogSearchDocumentPayloadBatchesSplitVectorDocs(t *testing.T) {
|
||||
vector := make([]float32, 128)
|
||||
docs := []catalogSearchDocument{
|
||||
|
||||
@@ -112,7 +112,7 @@ func (r *SeasonRepository) Upsert(ctx context.Context, s *models.Season) error {
|
||||
)
|
||||
ON CONFLICT (series_id, season_number) DO UPDATE SET
|
||||
title = EXCLUDED.title,
|
||||
default_metadata_language = COALESCE(NULLIF(seasons.default_metadata_language, ''), EXCLUDED.default_metadata_language),
|
||||
default_metadata_language = COALESCE(NULLIF(EXCLUDED.default_metadata_language, ''), seasons.default_metadata_language),
|
||||
overview = EXCLUDED.overview,
|
||||
air_date = EXCLUDED.air_date,
|
||||
poster_path = EXCLUDED.poster_path,
|
||||
@@ -198,7 +198,7 @@ func (r *SeasonRepository) BulkUpsert(ctx context.Context, seasons []*models.Sea
|
||||
)
|
||||
ON CONFLICT (series_id, season_number) DO UPDATE SET
|
||||
title = EXCLUDED.title,
|
||||
default_metadata_language = COALESCE(NULLIF(seasons.default_metadata_language, ''), EXCLUDED.default_metadata_language),
|
||||
default_metadata_language = COALESCE(NULLIF(EXCLUDED.default_metadata_language, ''), seasons.default_metadata_language),
|
||||
overview = EXCLUDED.overview,
|
||||
air_date = EXCLUDED.air_date,
|
||||
poster_path = EXCLUDED.poster_path,
|
||||
|
||||
@@ -1828,7 +1828,10 @@ func bulkUpdatePeople(ctx context.Context, tx pgx.Tx, people map[int64]*importPe
|
||||
FROM (VALUES `,
|
||||
rows,
|
||||
7,
|
||||
nil,
|
||||
// A standalone VALUES clause has no target column to infer types
|
||||
// from, so v.id must be cast explicitly or it defaults to text and
|
||||
// the join predicate becomes bigint = text (SQLSTATE 42883).
|
||||
map[int]string{0: "::bigint"},
|
||||
`) AS v(id, tmdb_id, imdb_id, tvdb_id, plex_guid, photo_path, photo_thumbhash)
|
||||
WHERE p.id = v.id`,
|
||||
nil,
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
package catalogseed
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
func newCatalogSeedTestPool(t *testing.T) *pgxpool.Pool {
|
||||
t.Helper()
|
||||
dsn := os.Getenv("SILO_TEST_DATABASE_URL")
|
||||
if dsn == "" {
|
||||
t.Skip("SILO_TEST_DATABASE_URL is not set")
|
||||
}
|
||||
pool, err := pgxpool.New(context.Background(), dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("connect test database: %v", err)
|
||||
}
|
||||
t.Cleanup(pool.Close)
|
||||
return pool
|
||||
}
|
||||
|
||||
func TestBulkUpdatePeopleComparesIDsAsBigint(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
pool := newCatalogSeedTestPool(t)
|
||||
|
||||
tx, err := pool.Begin(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("begin tx: %v", err)
|
||||
}
|
||||
defer func() { _ = tx.Rollback(ctx) }()
|
||||
|
||||
personID := time.Now().UnixNano()
|
||||
if _, err := tx.Exec(ctx, `
|
||||
INSERT INTO people (id, name, sort_name, tmdb_id)
|
||||
VALUES ($1, 'Import Test Person', 'import test person', '111')
|
||||
`, personID); err != nil {
|
||||
t.Fatalf("seed person: %v", err)
|
||||
}
|
||||
|
||||
people := map[int64]*importPersonState{
|
||||
personID: {
|
||||
ID: personID,
|
||||
Name: "Import Test Person",
|
||||
TmdbID: "222",
|
||||
NeedsUpdate: true,
|
||||
},
|
||||
}
|
||||
if err := bulkUpdatePeople(ctx, tx, people); err != nil {
|
||||
t.Fatalf("bulkUpdatePeople: %v", err)
|
||||
}
|
||||
|
||||
var tmdbID string
|
||||
if err := tx.QueryRow(ctx, `SELECT tmdb_id FROM people WHERE id = $1`, personID).Scan(&tmdbID); err != nil {
|
||||
t.Fatalf("read back person: %v", err)
|
||||
}
|
||||
if tmdbID != "222" {
|
||||
t.Fatalf("tmdb_id = %q, want %q", tmdbID, "222")
|
||||
}
|
||||
}
|
||||
@@ -303,6 +303,13 @@ type DownloadConfig struct {
|
||||
MaxConcurrentPerUser int `yaml:"-"` // max simultaneous downloads per user (0 = unlimited)
|
||||
MaxPerPeriod int `yaml:"-"` // max downloads per user per period (0 = unlimited)
|
||||
PeriodDuration time.Duration `yaml:"-"` // rolling window for MaxPerPeriod
|
||||
|
||||
// Downloads v2 (offline sync for mobile). The prepare-to-file pipeline that
|
||||
// consumes these ships in Phase 3; they are parsed default-off from Phase 0.
|
||||
TranscodeEnabled bool `yaml:"-"` // server gate for transcode-to-file (default false)
|
||||
ArtifactDir string `yaml:"-"` // prepared-artifact output volume ("" = default under the transcode dir)
|
||||
MaxConcurrentPrepares int `yaml:"-"` // encode/remux worker-pool size (default 2)
|
||||
ArtifactMaxBytes int64 `yaml:"-"` // LRU eviction budget for prepared artifacts (0 = unlimited)
|
||||
}
|
||||
|
||||
// MetadataConfig holds metadata pipeline settings.
|
||||
@@ -365,6 +372,10 @@ var defaultJellyfinCompatServerID = uuid.NewSHA1(
|
||||
[]byte("https://silo.local/jellycompat"),
|
||||
).String()
|
||||
|
||||
// DefaultTranscodeDir is the fallback playback.transcode_dir; download
|
||||
// artifacts default to a sibling directory (see downloads.effectiveArtifactDir).
|
||||
const DefaultTranscodeDir = "/tmp/silo-transcode"
|
||||
|
||||
const DefaultJellyfinCompatEmulatedServerVersion = "10.12.0"
|
||||
const DefaultJellyfinWebVersion = "10.11.6"
|
||||
const DefaultJellyfinWebInstallDir = "/var/lib/silo/compat/jellyfin-web"
|
||||
@@ -432,7 +443,7 @@ func setDefaults() *configRaw {
|
||||
},
|
||||
Playback: PlaybackConfig{
|
||||
FFmpegPath: "/usr/lib/jellyfin-ffmpeg/ffmpeg",
|
||||
TranscodeDir: "/tmp/silo-transcode",
|
||||
TranscodeDir: DefaultTranscodeDir,
|
||||
HWAccel: "auto",
|
||||
ChapterThumbnailWorkers: 1,
|
||||
ChapterThumbnailExecution: "local",
|
||||
|
||||
@@ -302,7 +302,7 @@ func LoadFromDB(m map[string]string) (*Config, error) {
|
||||
|
||||
// Playback
|
||||
cfg.Playback.FFmpegPath = stringOr(m, "playback.ffmpeg_path", "/usr/lib/jellyfin-ffmpeg/ffmpeg")
|
||||
cfg.Playback.TranscodeDir = stringOr(m, "playback.transcode_dir", "/tmp/silo-transcode")
|
||||
cfg.Playback.TranscodeDir = stringOr(m, "playback.transcode_dir", DefaultTranscodeDir)
|
||||
cfg.Playback.HWAccel = stringOr(m, "playback.hw_accel", "auto")
|
||||
cfg.Playback.HWDevice = stringOr(m, "playback.hw_device", "")
|
||||
chapterThumbnailWorkers, err := intOr(m, "playback.chapter_thumbnail_workers", 1)
|
||||
@@ -565,5 +565,30 @@ func LoadFromDB(m map[string]string) (*Config, error) {
|
||||
cfg.Download.MaxPerPeriod = maxPerPeriod
|
||||
cfg.Download.PeriodDuration = periodDuration
|
||||
|
||||
// Downloads v2 (offline sync). Default-off; the prepare-to-file pipeline that
|
||||
// consumes these ships in Phase 3.
|
||||
downloadTranscodeEnabled, err := boolOr(m, "download.transcode_enabled", false)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
maxConcurrentPrepares, err := intOr(m, "download.max_concurrent_prepares", 2)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
artifactMaxBytes, err := int64Or(m, "download.artifact_max_bytes", 0)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if maxConcurrentPrepares < 0 {
|
||||
return nil, fmt.Errorf("invalid value for %q: must be non-negative", "download.max_concurrent_prepares")
|
||||
}
|
||||
if artifactMaxBytes < 0 {
|
||||
return nil, fmt.Errorf("invalid value for %q: must be non-negative", "download.artifact_max_bytes")
|
||||
}
|
||||
cfg.Download.TranscodeEnabled = downloadTranscodeEnabled
|
||||
cfg.Download.ArtifactDir = stringOr(m, "download.artifact_dir", "")
|
||||
cfg.Download.MaxConcurrentPrepares = maxConcurrentPrepares
|
||||
cfg.Download.ArtifactMaxBytes = artifactMaxBytes
|
||||
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
@@ -42,6 +42,8 @@ func TestRestartRequired(t *testing.T) {
|
||||
{"overlays.enabled", false},
|
||||
{"markers.mode", false},
|
||||
{"download.enabled", false},
|
||||
{"download.transcode_enabled", false},
|
||||
{"download.max_concurrent_prepares", false},
|
||||
{"allow_4k_transcode", false},
|
||||
{"defaults.card_overlays", false},
|
||||
// Unknown keys default to live.
|
||||
|
||||
@@ -1,50 +0,0 @@
|
||||
package download
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Download status constants.
|
||||
const (
|
||||
StatusQueued = "queued"
|
||||
StatusDownloading = "downloading"
|
||||
StatusCompleted = "completed"
|
||||
StatusFailed = "failed"
|
||||
StatusCancelled = "cancelled"
|
||||
)
|
||||
|
||||
// Download kind constants.
|
||||
const (
|
||||
KindDirect = "direct"
|
||||
KindQueued = "queued"
|
||||
)
|
||||
|
||||
// Sentinel errors.
|
||||
var (
|
||||
ErrNotFound = errors.New("download not found")
|
||||
ErrDownloadNotAllowed = errors.New("user is not allowed to download")
|
||||
ErrFeatureDisabled = errors.New("downloads are disabled")
|
||||
ErrConcurrentLimitReached = errors.New("concurrent download limit reached")
|
||||
ErrPeriodLimitReached = errors.New("download period limit reached")
|
||||
ErrDownloadNotActive = errors.New("download is not in an active state")
|
||||
ErrStatusConflict = errors.New("download status transition conflict")
|
||||
)
|
||||
|
||||
// Download represents a row in the downloads table.
|
||||
type Download struct {
|
||||
ID string
|
||||
UserID int
|
||||
MediaFileID int
|
||||
ContentID string
|
||||
EpisodeID string
|
||||
BatchID string
|
||||
Kind string // direct or queued
|
||||
Status string // queued, downloading, completed, failed, cancelled
|
||||
FileSize int64
|
||||
BytesSent int64
|
||||
ErrorMessage string
|
||||
CreatedAt time.Time
|
||||
UpdatedAt time.Time
|
||||
CompletedAt *time.Time
|
||||
}
|
||||
@@ -1,239 +0,0 @@
|
||||
package download
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
const downloadColumns = `id, user_id, media_file_id, content_id, episode_id, batch_id,
|
||||
kind, status, file_size, bytes_sent, error_message,
|
||||
created_at, updated_at, completed_at`
|
||||
|
||||
// Repository provides CRUD operations for the downloads table.
|
||||
type Repository struct {
|
||||
pool *pgxpool.Pool
|
||||
}
|
||||
|
||||
// NewRepository creates a new Repository backed by the given pool.
|
||||
func NewRepository(pool *pgxpool.Pool) *Repository {
|
||||
return &Repository{pool: pool}
|
||||
}
|
||||
|
||||
func scanDownload(row pgx.Row) (*Download, error) {
|
||||
var d Download
|
||||
var episodeID, batchID *string
|
||||
err := row.Scan(
|
||||
&d.ID, &d.UserID, &d.MediaFileID, &d.ContentID, &episodeID, &batchID,
|
||||
&d.Kind, &d.Status, &d.FileSize, &d.BytesSent, &d.ErrorMessage,
|
||||
&d.CreatedAt, &d.UpdatedAt, &d.CompletedAt,
|
||||
)
|
||||
if err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil, ErrNotFound
|
||||
}
|
||||
return nil, fmt.Errorf("scanning download: %w", err)
|
||||
}
|
||||
if episodeID != nil {
|
||||
d.EpisodeID = *episodeID
|
||||
}
|
||||
if batchID != nil {
|
||||
d.BatchID = *batchID
|
||||
}
|
||||
return &d, nil
|
||||
}
|
||||
|
||||
func scanDownloads(rows pgx.Rows) ([]*Download, error) {
|
||||
var downloads []*Download
|
||||
for rows.Next() {
|
||||
var d Download
|
||||
var episodeID, batchID *string
|
||||
err := rows.Scan(
|
||||
&d.ID, &d.UserID, &d.MediaFileID, &d.ContentID, &episodeID, &batchID,
|
||||
&d.Kind, &d.Status, &d.FileSize, &d.BytesSent, &d.ErrorMessage,
|
||||
&d.CreatedAt, &d.UpdatedAt, &d.CompletedAt,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("scanning download row: %w", err)
|
||||
}
|
||||
if episodeID != nil {
|
||||
d.EpisodeID = *episodeID
|
||||
}
|
||||
if batchID != nil {
|
||||
d.BatchID = *batchID
|
||||
}
|
||||
downloads = append(downloads, &d)
|
||||
}
|
||||
return downloads, rows.Err()
|
||||
}
|
||||
|
||||
// Create inserts a new download record.
|
||||
func (r *Repository) Create(ctx context.Context, d *Download) error {
|
||||
_, err := r.pool.Exec(ctx,
|
||||
`INSERT INTO downloads (id, user_id, media_file_id, content_id, episode_id, batch_id,
|
||||
kind, status, file_size, bytes_sent, error_message, created_at, updated_at, completed_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14)`,
|
||||
d.ID, d.UserID, d.MediaFileID, d.ContentID,
|
||||
nilIfEmpty(d.EpisodeID), nilIfEmpty(d.BatchID),
|
||||
d.Kind, d.Status, d.FileSize, d.BytesSent, d.ErrorMessage,
|
||||
d.CreatedAt, d.UpdatedAt, d.CompletedAt,
|
||||
)
|
||||
if err != nil {
|
||||
return fmt.Errorf("inserting download: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// CreateBatch inserts multiple download records in a single transaction.
|
||||
func (r *Repository) CreateBatch(ctx context.Context, downloads []*Download) error {
|
||||
tx, err := r.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("beginning batch insert: %w", err)
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
|
||||
for _, d := range downloads {
|
||||
_, err := tx.Exec(ctx,
|
||||
`INSERT INTO downloads (id, user_id, media_file_id, content_id, episode_id, batch_id,
|
||||
kind, status, file_size, bytes_sent, error_message, created_at, updated_at, completed_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14)`,
|
||||
d.ID, d.UserID, d.MediaFileID, d.ContentID,
|
||||
nilIfEmpty(d.EpisodeID), nilIfEmpty(d.BatchID),
|
||||
d.Kind, d.Status, d.FileSize, d.BytesSent, d.ErrorMessage,
|
||||
d.CreatedAt, d.UpdatedAt, d.CompletedAt,
|
||||
)
|
||||
if err != nil {
|
||||
return fmt.Errorf("inserting batch download: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return fmt.Errorf("committing batch insert: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetByID retrieves a download by its ID.
|
||||
func (r *Repository) GetByID(ctx context.Context, id string) (*Download, error) {
|
||||
query := `SELECT ` + downloadColumns + ` FROM downloads WHERE id = $1`
|
||||
return scanDownload(r.pool.QueryRow(ctx, query, id))
|
||||
}
|
||||
|
||||
// ListByUser returns all downloads for a user, most recent first.
|
||||
func (r *Repository) ListByUser(ctx context.Context, userID int) ([]*Download, error) {
|
||||
query := `SELECT ` + downloadColumns + ` FROM downloads
|
||||
WHERE user_id = $1
|
||||
ORDER BY created_at DESC`
|
||||
rows, err := r.pool.Query(ctx, query, userID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("listing downloads for user: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
result, err := scanDownloads(rows)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("scanning download rows: %w", err)
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// CountActiveByUser returns the number of active (queued or downloading) downloads for a user.
|
||||
func (r *Repository) CountActiveByUser(ctx context.Context, userID int) (int, error) {
|
||||
var count int
|
||||
err := r.pool.QueryRow(ctx,
|
||||
`SELECT COUNT(*) FROM downloads WHERE user_id = $1 AND status IN ('queued', 'downloading')`,
|
||||
userID,
|
||||
).Scan(&count)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("counting active downloads: %w", err)
|
||||
}
|
||||
return count, nil
|
||||
}
|
||||
|
||||
// CountByUserSince returns the number of successful downloads created since the given time.
|
||||
// Cancelled and failed downloads are excluded so transient failures don't consume quota.
|
||||
func (r *Repository) CountByUserSince(ctx context.Context, userID int, since time.Time) (int, error) {
|
||||
var count int
|
||||
err := r.pool.QueryRow(ctx,
|
||||
`SELECT COUNT(*) FROM downloads WHERE user_id = $1 AND created_at >= $2 AND status NOT IN ('cancelled', 'failed')`,
|
||||
userID, since,
|
||||
).Scan(&count)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("counting downloads in period: %w", err)
|
||||
}
|
||||
return count, nil
|
||||
}
|
||||
|
||||
// UpdateStatus sets the status and optionally the bytes_sent and completed_at fields.
|
||||
func (r *Repository) UpdateStatus(ctx context.Context, id, status string, bytesSent int64, completedAt *time.Time) error {
|
||||
tag, err := r.pool.Exec(ctx,
|
||||
`UPDATE downloads SET status = $1, bytes_sent = $2, completed_at = $3, updated_at = now() WHERE id = $4`,
|
||||
status, bytesSent, completedAt, id,
|
||||
)
|
||||
if err != nil {
|
||||
return fmt.Errorf("updating download status: %w", err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return ErrNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// TransitionStatus atomically transitions a download from expectedStatus to newStatus.
|
||||
// Returns ErrStatusConflict if the row is not in expectedStatus (another request
|
||||
// already transitioned it).
|
||||
func (r *Repository) TransitionStatus(ctx context.Context, id, expectedStatus, newStatus string, bytesSent int64, completedAt *time.Time) error {
|
||||
tag, err := r.pool.Exec(ctx,
|
||||
`UPDATE downloads SET status = $1, bytes_sent = $2, completed_at = $3, updated_at = now()
|
||||
WHERE id = $4 AND status = $5`,
|
||||
newStatus, bytesSent, completedAt, id, expectedStatus,
|
||||
)
|
||||
if err != nil {
|
||||
return fmt.Errorf("transitioning download status: %w", err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return ErrStatusConflict
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Delete removes a download record. Returns ErrNotFound if the row doesn't exist
|
||||
// or doesn't belong to the given user.
|
||||
func (r *Repository) Delete(ctx context.Context, id string, userID int) error {
|
||||
tag, err := r.pool.Exec(ctx,
|
||||
`DELETE FROM downloads WHERE id = $1 AND user_id = $2`, id, userID,
|
||||
)
|
||||
if err != nil {
|
||||
return fmt.Errorf("deleting download: %w", err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return ErrNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// CancelByID sets a download to cancelled if it is still queued or downloading.
|
||||
func (r *Repository) CancelByID(ctx context.Context, id string, userID int) error {
|
||||
tag, err := r.pool.Exec(ctx,
|
||||
`UPDATE downloads SET status = 'cancelled', updated_at = now()
|
||||
WHERE id = $1 AND user_id = $2 AND status IN ('queued', 'downloading')`,
|
||||
id, userID,
|
||||
)
|
||||
if err != nil {
|
||||
return fmt.Errorf("cancelling download: %w", err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return ErrNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func nilIfEmpty(s string) *string {
|
||||
if s == "" {
|
||||
return nil
|
||||
}
|
||||
return &s
|
||||
}
|
||||
@@ -1,518 +0,0 @@
|
||||
package download
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/config"
|
||||
"github.com/Silo-Server/silo-server/internal/idgen"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
"github.com/Silo-Server/silo-server/internal/playback"
|
||||
)
|
||||
|
||||
// FileResolver looks up media files by various keys.
|
||||
type FileResolver interface {
|
||||
GetByID(ctx context.Context, id int) (*models.MediaFile, error)
|
||||
GetByContentID(ctx context.Context, contentID string) ([]*models.MediaFile, error)
|
||||
GetByEpisodeID(ctx context.Context, episodeID string) ([]*models.MediaFile, error)
|
||||
}
|
||||
|
||||
// ItemResolver looks up media items.
|
||||
type ItemResolver interface {
|
||||
GetByID(ctx context.Context, contentID string) (*models.MediaItem, error)
|
||||
}
|
||||
|
||||
// EpisodeResolver lists episodes for a series.
|
||||
type EpisodeResolver interface {
|
||||
ListBySeries(ctx context.Context, seriesID string) ([]*models.Episode, error)
|
||||
}
|
||||
|
||||
// UserResolver looks up users.
|
||||
type UserResolver interface {
|
||||
GetByID(ctx context.Context, id int) (*models.User, error)
|
||||
}
|
||||
|
||||
// ItemAccessChecker checks library/content-rating access.
|
||||
type ItemAccessChecker interface {
|
||||
EnsureAccessible(ctx context.Context, contentID string, filter catalog.AccessFilter) error
|
||||
}
|
||||
|
||||
// SettingsReader loads all server settings as a flat map.
|
||||
type SettingsReader interface {
|
||||
GetAll(ctx context.Context) (map[string]string, error)
|
||||
}
|
||||
|
||||
const configCacheTTL = 30 * time.Second
|
||||
|
||||
// Service orchestrates download permission checks, quota enforcement,
|
||||
// file resolution, and file serving.
|
||||
type Service struct {
|
||||
repo *Repository
|
||||
bandwidth *BandwidthManager
|
||||
limiter *QuantityLimiter
|
||||
fileRepo FileResolver
|
||||
itemRepo ItemResolver
|
||||
episodeRepo EpisodeResolver
|
||||
userRepo UserResolver
|
||||
itemAccess ItemAccessChecker
|
||||
settings SettingsReader
|
||||
|
||||
cfgMu sync.RWMutex
|
||||
cfg config.DownloadConfig
|
||||
cfgLoadedAt time.Time
|
||||
}
|
||||
|
||||
// NewService creates a new download service with the given dependencies.
|
||||
func NewService(
|
||||
repo *Repository,
|
||||
bandwidth *BandwidthManager,
|
||||
limiter *QuantityLimiter,
|
||||
fileRepo FileResolver,
|
||||
itemRepo ItemResolver,
|
||||
episodeRepo EpisodeResolver,
|
||||
userRepo UserResolver,
|
||||
itemAccess ItemAccessChecker,
|
||||
settings SettingsReader,
|
||||
initialCfg *config.DownloadConfig,
|
||||
) *Service {
|
||||
s := &Service{
|
||||
repo: repo,
|
||||
bandwidth: bandwidth,
|
||||
limiter: limiter,
|
||||
fileRepo: fileRepo,
|
||||
itemRepo: itemRepo,
|
||||
episodeRepo: episodeRepo,
|
||||
userRepo: userRepo,
|
||||
itemAccess: itemAccess,
|
||||
settings: settings,
|
||||
}
|
||||
if initialCfg != nil {
|
||||
s.cfg = *initialCfg
|
||||
s.cfgLoadedAt = time.Now()
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// loadConfig returns the current download config, refreshing from DB if stale.
|
||||
func (s *Service) loadConfig(ctx context.Context) config.DownloadConfig {
|
||||
s.cfgMu.RLock()
|
||||
if time.Since(s.cfgLoadedAt) < configCacheTTL {
|
||||
cfg := s.cfg
|
||||
s.cfgMu.RUnlock()
|
||||
return cfg
|
||||
}
|
||||
s.cfgMu.RUnlock()
|
||||
|
||||
s.cfgMu.Lock()
|
||||
defer s.cfgMu.Unlock()
|
||||
|
||||
// Double-check after acquiring write lock.
|
||||
if time.Since(s.cfgLoadedAt) < configCacheTTL {
|
||||
return s.cfg
|
||||
}
|
||||
|
||||
if s.settings == nil {
|
||||
return s.cfg
|
||||
}
|
||||
|
||||
allSettings, err := s.settings.GetAll(ctx)
|
||||
if err != nil {
|
||||
slog.Warn("failed to reload download config from DB, using cached", "error", err)
|
||||
return s.cfg
|
||||
}
|
||||
|
||||
newFullCfg, err := config.LoadFromDB(allSettings)
|
||||
if err != nil {
|
||||
slog.Warn("failed to parse download config from DB, using cached", "error", err)
|
||||
return s.cfg
|
||||
}
|
||||
|
||||
oldCfg := s.cfg
|
||||
s.cfg = newFullCfg.Download
|
||||
s.cfgLoadedAt = time.Now()
|
||||
|
||||
// Update bandwidth manager if limits changed.
|
||||
if s.bandwidth != nil && (oldCfg.ServerBandwidthBPS != s.cfg.ServerBandwidthBPS || oldCfg.UserBandwidthBPS != s.cfg.UserBandwidthBPS) {
|
||||
s.bandwidth.Reload(s.cfg.ServerBandwidthBPS, s.cfg.UserBandwidthBPS)
|
||||
slog.Info("download bandwidth config reloaded", "server_bps", s.cfg.ServerBandwidthBPS, "user_bps", s.cfg.UserBandwidthBPS)
|
||||
}
|
||||
|
||||
// Update quantity limiter if limits changed.
|
||||
if s.limiter != nil && (oldCfg.MaxConcurrentPerUser != s.cfg.MaxConcurrentPerUser || oldCfg.MaxPerPeriod != s.cfg.MaxPerPeriod || oldCfg.PeriodDuration != s.cfg.PeriodDuration) {
|
||||
s.limiter.Reload(s.cfg.MaxConcurrentPerUser, s.cfg.MaxPerPeriod, s.cfg.PeriodDuration)
|
||||
slog.Info("download quantity limits reloaded", "max_concurrent", s.cfg.MaxConcurrentPerUser, "max_per_period", s.cfg.MaxPerPeriod, "period", s.cfg.PeriodDuration)
|
||||
}
|
||||
|
||||
return s.cfg
|
||||
}
|
||||
|
||||
// CreateRequest holds the parameters for creating a download.
|
||||
type CreateRequest struct {
|
||||
ContentID string
|
||||
EpisodeID string
|
||||
FileID int
|
||||
}
|
||||
|
||||
// CreateQueued creates a queued download for a single item (movie or episode).
|
||||
func (s *Service) CreateQueued(ctx context.Context, userID int, req CreateRequest, filter catalog.AccessFilter) (*Download, error) {
|
||||
if err := s.checkEnabled(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
user, err := s.userRepo.GetByID(ctx, userID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("loading user: %w", err)
|
||||
}
|
||||
if !user.DownloadAllowed {
|
||||
return nil, ErrDownloadNotAllowed
|
||||
}
|
||||
|
||||
file, err := s.resolveFile(ctx, req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if err := s.itemAccess.EnsureAccessible(ctx, file.ContentID, filter); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if err := s.limiter.Check(ctx, userID, 1); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
id, err := idgen.NextID()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("generating download ID: %w", err)
|
||||
}
|
||||
|
||||
now := time.Now()
|
||||
d := &Download{
|
||||
ID: id,
|
||||
UserID: userID,
|
||||
MediaFileID: file.ID,
|
||||
ContentID: file.ContentID,
|
||||
EpisodeID: file.EpisodeID,
|
||||
Kind: KindQueued,
|
||||
Status: StatusQueued,
|
||||
FileSize: file.FileSize,
|
||||
CreatedAt: now,
|
||||
UpdatedAt: now,
|
||||
}
|
||||
|
||||
if err := s.repo.Create(ctx, d); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return d, nil
|
||||
}
|
||||
|
||||
// CreateQueuedBatch creates download records for all episodes in a series.
|
||||
// Returns the created downloads and a shared batch ID.
|
||||
func (s *Service) CreateQueuedBatch(ctx context.Context, userID int, seriesContentID string, filter catalog.AccessFilter) ([]*Download, string, error) {
|
||||
if err := s.checkEnabled(ctx); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
|
||||
user, err := s.userRepo.GetByID(ctx, userID)
|
||||
if err != nil {
|
||||
return nil, "", fmt.Errorf("loading user: %w", err)
|
||||
}
|
||||
if !user.DownloadAllowed {
|
||||
return nil, "", ErrDownloadNotAllowed
|
||||
}
|
||||
|
||||
item, err := s.itemRepo.GetByID(ctx, seriesContentID)
|
||||
if err != nil {
|
||||
return nil, "", fmt.Errorf("loading series: %w", err)
|
||||
}
|
||||
if item.Type != "series" {
|
||||
return nil, "", fmt.Errorf("content_id is not a series")
|
||||
}
|
||||
|
||||
if err := s.itemAccess.EnsureAccessible(ctx, seriesContentID, filter); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
|
||||
episodes, err := s.episodeRepo.ListBySeries(ctx, seriesContentID)
|
||||
if err != nil {
|
||||
return nil, "", fmt.Errorf("listing episodes: %w", err)
|
||||
}
|
||||
|
||||
var downloads []*Download
|
||||
now := time.Now()
|
||||
|
||||
for _, ep := range episodes {
|
||||
files, err := s.fileRepo.GetByEpisodeID(ctx, ep.ContentID)
|
||||
if err != nil {
|
||||
return nil, "", fmt.Errorf("resolving files for episode %s: %w", ep.ContentID, err)
|
||||
}
|
||||
if len(files) == 0 {
|
||||
continue // skip episodes with no files
|
||||
}
|
||||
file := pickBestFile(files)
|
||||
|
||||
id, err := idgen.NextID()
|
||||
if err != nil {
|
||||
return nil, "", fmt.Errorf("generating download ID: %w", err)
|
||||
}
|
||||
|
||||
downloads = append(downloads, &Download{
|
||||
ID: id,
|
||||
UserID: userID,
|
||||
MediaFileID: file.ID,
|
||||
ContentID: seriesContentID,
|
||||
EpisodeID: ep.ContentID,
|
||||
Kind: KindQueued,
|
||||
Status: StatusQueued,
|
||||
FileSize: file.FileSize,
|
||||
CreatedAt: now,
|
||||
UpdatedAt: now,
|
||||
})
|
||||
}
|
||||
|
||||
if len(downloads) == 0 {
|
||||
return nil, "", fmt.Errorf("no downloadable episodes found")
|
||||
}
|
||||
|
||||
if err := s.limiter.Check(ctx, userID, len(downloads)); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
|
||||
batchID, err := idgen.NextID()
|
||||
if err != nil {
|
||||
return nil, "", fmt.Errorf("generating batch ID: %w", err)
|
||||
}
|
||||
for _, d := range downloads {
|
||||
d.BatchID = batchID
|
||||
}
|
||||
|
||||
if err := s.repo.CreateBatch(ctx, downloads); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
return downloads, batchID, nil
|
||||
}
|
||||
|
||||
// ServeDirect validates permissions and serves a file directly for browser download.
|
||||
// No persistent download record is created.
|
||||
func (s *Service) ServeDirect(ctx context.Context, w http.ResponseWriter, r *http.Request, userID int, fileID int, filter catalog.AccessFilter) error {
|
||||
if err := s.checkEnabled(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
user, err := s.userRepo.GetByID(ctx, userID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("loading user: %w", err)
|
||||
}
|
||||
if !user.DownloadAllowed {
|
||||
return ErrDownloadNotAllowed
|
||||
}
|
||||
|
||||
file, err := s.fileRepo.GetByID(ctx, fileID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("loading media file: %w", err)
|
||||
}
|
||||
if file == nil || file.MissingSince != nil {
|
||||
return catalog.ErrItemNotFound
|
||||
}
|
||||
|
||||
if err := s.itemAccess.EnsureAccessible(ctx, file.ContentID, filter); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return s.serveFileDownload(ctx, w, r, file, userID)
|
||||
}
|
||||
|
||||
// ServeFile serves a queued download's file, verifying ownership and current policy.
|
||||
func (s *Service) ServeFile(ctx context.Context, w http.ResponseWriter, r *http.Request, userID int, downloadID string) error {
|
||||
// Re-check policy — admin may have disabled downloads or revoked permission
|
||||
// after this download was queued.
|
||||
if err := s.checkEnabled(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
user, err := s.userRepo.GetByID(ctx, userID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("loading user: %w", err)
|
||||
}
|
||||
if !user.DownloadAllowed {
|
||||
return ErrDownloadNotAllowed
|
||||
}
|
||||
|
||||
dl, err := s.repo.GetByID(ctx, downloadID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if dl.UserID != userID {
|
||||
return ErrNotFound // don't reveal existence
|
||||
}
|
||||
if dl.Status == StatusCancelled || dl.Status == StatusFailed {
|
||||
return fmt.Errorf("download is %s: %w", dl.Status, ErrDownloadNotActive)
|
||||
}
|
||||
|
||||
file, err := s.fileRepo.GetByID(ctx, dl.MediaFileID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("loading media file: %w", err)
|
||||
}
|
||||
if file == nil || file.MissingSince != nil {
|
||||
return catalog.ErrItemNotFound
|
||||
}
|
||||
|
||||
// Atomically transition queued → downloading. If another request already
|
||||
// claimed this download, return a conflict error.
|
||||
if dl.Status == StatusQueued {
|
||||
if err := s.repo.TransitionStatus(ctx, dl.ID, StatusQueued, StatusDownloading, 0, nil); err != nil {
|
||||
if errors.Is(err, ErrStatusConflict) {
|
||||
return fmt.Errorf("download already in progress: %w", ErrDownloadNotActive)
|
||||
}
|
||||
slog.Warn("failed to transition download to downloading", "download_id", dl.ID, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
if err := s.serveFileDownload(ctx, w, r, file, userID); err != nil {
|
||||
if updateErr := s.repo.UpdateStatus(ctx, dl.ID, StatusFailed, 0, nil); updateErr != nil {
|
||||
slog.Error("failed to mark download as failed", "download_id", dl.ID, "error", updateErr)
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
now := time.Now()
|
||||
if err := s.repo.UpdateStatus(ctx, dl.ID, StatusCompleted, file.FileSize, &now); err != nil {
|
||||
slog.Error("failed to mark download as completed", "download_id", dl.ID, "error", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// List returns all downloads for a user.
|
||||
func (s *Service) List(ctx context.Context, userID int) ([]*Download, error) {
|
||||
return s.repo.ListByUser(ctx, userID)
|
||||
}
|
||||
|
||||
// Cancel cancels a queued/active download or deletes a completed/failed one.
|
||||
func (s *Service) Cancel(ctx context.Context, userID int, downloadID string) error {
|
||||
dl, err := s.repo.GetByID(ctx, downloadID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if dl.UserID != userID {
|
||||
return ErrNotFound
|
||||
}
|
||||
|
||||
switch dl.Status {
|
||||
case StatusQueued, StatusDownloading:
|
||||
return s.repo.CancelByID(ctx, downloadID, userID)
|
||||
default:
|
||||
return s.repo.Delete(ctx, downloadID, userID)
|
||||
}
|
||||
}
|
||||
|
||||
func (s *Service) checkEnabled(ctx context.Context) error {
|
||||
cfg := s.loadConfig(ctx)
|
||||
if !cfg.Enabled {
|
||||
return ErrFeatureDisabled
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (s *Service) resolveFile(ctx context.Context, req CreateRequest) (*models.MediaFile, error) {
|
||||
if req.FileID > 0 {
|
||||
file, err := s.fileRepo.GetByID(ctx, req.FileID)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("loading media file: %w", err)
|
||||
}
|
||||
if file == nil || file.MissingSince != nil {
|
||||
return nil, catalog.ErrItemNotFound
|
||||
}
|
||||
return file, nil
|
||||
}
|
||||
|
||||
var files []*models.MediaFile
|
||||
var err error
|
||||
if req.EpisodeID != "" {
|
||||
files, err = s.fileRepo.GetByEpisodeID(ctx, req.EpisodeID)
|
||||
} else {
|
||||
files, err = s.fileRepo.GetByContentID(ctx, req.ContentID)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolving files: %w", err)
|
||||
}
|
||||
if len(files) == 0 {
|
||||
return nil, catalog.ErrItemNotFound
|
||||
}
|
||||
|
||||
return pickBestFile(files), nil
|
||||
}
|
||||
|
||||
func (s *Service) serveFileDownload(ctx context.Context, w http.ResponseWriter, r *http.Request, file *models.MediaFile, userID int) error {
|
||||
f, err := os.Open(file.FilePath)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return catalog.ErrItemNotFound
|
||||
}
|
||||
return fmt.Errorf("opening file: %w", err)
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
stat, err := f.Stat()
|
||||
if err != nil {
|
||||
return fmt.Errorf("stat file: %w", err)
|
||||
}
|
||||
|
||||
filename := sanitizeFilename(filepath.Base(file.FilePath))
|
||||
w.Header().Set("Content-Disposition", fmt.Sprintf(`attachment; filename="%s"`, filename))
|
||||
w.Header().Set("Content-Type", playback.MimeFromExtension(file.FilePath))
|
||||
|
||||
var reader io.ReadSeeker = f
|
||||
if s.bandwidth != nil {
|
||||
reader = s.bandwidth.ThrottledReader(ctx, f, userID)
|
||||
}
|
||||
|
||||
http.ServeContent(w, r, stat.Name(), stat.ModTime(), reader)
|
||||
return nil
|
||||
}
|
||||
|
||||
// pickBestFile selects the highest-resolution file from a list.
|
||||
func pickBestFile(files []*models.MediaFile) *models.MediaFile {
|
||||
if len(files) == 1 {
|
||||
return files[0]
|
||||
}
|
||||
best := files[0]
|
||||
for _, f := range files[1:] {
|
||||
if resolutionRank(f.Resolution) > resolutionRank(best.Resolution) {
|
||||
best = f
|
||||
}
|
||||
}
|
||||
return best
|
||||
}
|
||||
|
||||
func resolutionRank(res string) int {
|
||||
switch strings.ToLower(res) {
|
||||
case "2160p":
|
||||
return 4
|
||||
case "1080p":
|
||||
return 3
|
||||
case "720p":
|
||||
return 2
|
||||
case "480p":
|
||||
return 1
|
||||
default:
|
||||
return 0
|
||||
}
|
||||
}
|
||||
|
||||
func sanitizeFilename(name string) string {
|
||||
return strings.Map(func(r rune) rune {
|
||||
switch r {
|
||||
case '/', '\\', '"', '<', '>', '|', '?', '*', ':':
|
||||
return '_'
|
||||
}
|
||||
return r
|
||||
}, name)
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
package downloads
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/config"
|
||||
)
|
||||
|
||||
// Artifact status constants (download_artifacts.status).
|
||||
const (
|
||||
ArtifactQueued = "queued"
|
||||
ArtifactRunning = "running"
|
||||
ArtifactReady = "ready"
|
||||
ArtifactFailed = "failed"
|
||||
)
|
||||
|
||||
// ErrNoArtifactJob is returned by the queue when no claimable job exists.
|
||||
var ErrNoArtifactJob = errors.New("no claimable artifact job")
|
||||
|
||||
// Artifact is a prepared (remux/transcode) file, deduplicated by
|
||||
// (media_file_id, format, params_hash), and a row in the durable encode queue.
|
||||
type Artifact struct {
|
||||
ID string
|
||||
MediaFileID int
|
||||
Format string // remux | transcode
|
||||
ParamsHash string
|
||||
Container string
|
||||
CodecVideo string
|
||||
CodecAudio string
|
||||
Resolution string
|
||||
AudioTrackIndex int
|
||||
TargetBitrateKbps int
|
||||
OutputPath string
|
||||
FileSize int64
|
||||
Status string
|
||||
ErrorMessage string
|
||||
Attempts int
|
||||
MaxAttempts int
|
||||
LeaseOwner string
|
||||
LeaseExpiresAt *time.Time
|
||||
NextRetryAt *time.Time
|
||||
CreatedAt time.Time
|
||||
CompletedAt *time.Time
|
||||
LastUsedAt time.Time
|
||||
}
|
||||
|
||||
// paramsHash is the dedup key for an encode target:
|
||||
// sha256(format | container | codec_video | codec_audio | resolution | audio_track_index | bitrate | subtitle_burn_in).
|
||||
func paramsHash(format, container, codecVideo, codecAudio, resolution string, audioTrackIndex, targetBitrateKbps int, subtitleBurnIn bool) string {
|
||||
sum := sha256.Sum256([]byte(fmt.Sprintf("%s|%s|%s|%s|%s|%d|%d|%t",
|
||||
format, container, codecVideo, codecAudio, resolution, audioTrackIndex, targetBitrateKbps, subtitleBurnIn)))
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// defaultTranscodeDir is config's TranscodeDir default and roots
|
||||
// prepared download artifacts when neither download.artifact_dir nor the
|
||||
// transcode dir is configured. Keeping it absolute avoids writing artifacts
|
||||
// relative to the process working directory.
|
||||
const defaultTranscodeDir = config.DefaultTranscodeDir
|
||||
|
||||
// effectiveArtifactDir resolves where prepared artifacts are written: the
|
||||
// configured download.artifact_dir when set, otherwise a dedicated directory
|
||||
// alongside the transcode dir. The result is always rooted at a real volume,
|
||||
// never "" (which would land in the process cwd).
|
||||
//
|
||||
// Artifacts live as a SIBLING of the transcode dir, never inside it:
|
||||
// CleanupOrphanedTranscodeDirs deletes every non-active subdirectory of the
|
||||
// transcode dir, so an artifact dir nested under it would be wiped on the next
|
||||
// transcode sweep.
|
||||
func effectiveArtifactDir(artifactDir, transcodeDir string) string {
|
||||
if artifactDir != "" {
|
||||
return artifactDir
|
||||
}
|
||||
if transcodeDir == "" {
|
||||
transcodeDir = defaultTranscodeDir
|
||||
}
|
||||
return filepath.Join(filepath.Dir(transcodeDir), "silo-download-artifacts")
|
||||
}
|
||||
|
||||
// artifactOutputPath derives a deterministic output path from
|
||||
// (media_file_id, format, params_hash) so a reclaimed job targets the same file.
|
||||
func artifactOutputPath(dir string, mediaFileID int, format, hash string) string {
|
||||
short := hash
|
||||
if len(short) > 16 {
|
||||
short = short[:16]
|
||||
}
|
||||
return filepath.Join(dir, fmt.Sprintf("%d_%s_%s.mp4", mediaFileID, format, short))
|
||||
}
|
||||
@@ -0,0 +1,348 @@
|
||||
package downloads
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
const artifactColumns = `id, media_file_id, format, params_hash, container, codec_video, codec_audio,
|
||||
resolution, audio_track_index, target_bitrate_kbps, output_path, file_size, status, error_message,
|
||||
attempts, max_attempts, lease_owner, lease_expires_at, next_retry_at,
|
||||
created_at, completed_at, last_used_at`
|
||||
|
||||
// ArtifactRepository provides CRUD + durable-queue operations for
|
||||
// download_artifacts.
|
||||
type ArtifactRepository struct {
|
||||
pool *pgxpool.Pool
|
||||
}
|
||||
|
||||
// NewArtifactRepository creates an ArtifactRepository.
|
||||
func NewArtifactRepository(pool *pgxpool.Pool) *ArtifactRepository {
|
||||
return &ArtifactRepository{pool: pool}
|
||||
}
|
||||
|
||||
func scanArtifact(row pgx.Row) (*Artifact, error) {
|
||||
var a Artifact
|
||||
var leaseOwner *string
|
||||
if err := row.Scan(
|
||||
&a.ID, &a.MediaFileID, &a.Format, &a.ParamsHash, &a.Container, &a.CodecVideo, &a.CodecAudio,
|
||||
&a.Resolution, &a.AudioTrackIndex, &a.TargetBitrateKbps, &a.OutputPath, &a.FileSize, &a.Status, &a.ErrorMessage,
|
||||
&a.Attempts, &a.MaxAttempts, &leaseOwner, &a.LeaseExpiresAt, &a.NextRetryAt,
|
||||
&a.CreatedAt, &a.CompletedAt, &a.LastUsedAt,
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
a.LeaseOwner = deref(leaseOwner)
|
||||
return &a, nil
|
||||
}
|
||||
|
||||
// EnsureQueued inserts the artifact if no row exists for its
|
||||
// (media_file_id, format, params_hash), then returns the current row (existing
|
||||
// or freshly queued) and whether it was newly created.
|
||||
func (r *ArtifactRepository) EnsureQueued(ctx context.Context, a *Artifact) (*Artifact, bool, error) {
|
||||
tag, err := r.pool.Exec(ctx,
|
||||
`INSERT INTO download_artifacts
|
||||
(id, media_file_id, format, params_hash, container, codec_video, codec_audio,
|
||||
resolution, audio_track_index, target_bitrate_kbps, output_path, status, max_attempts)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, 'queued', $12)
|
||||
ON CONFLICT (media_file_id, format, params_hash) DO NOTHING`,
|
||||
a.ID, a.MediaFileID, a.Format, a.ParamsHash, a.Container, a.CodecVideo, a.CodecAudio,
|
||||
a.Resolution, a.AudioTrackIndex, a.TargetBitrateKbps, a.OutputPath, a.MaxAttempts,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, false, fmt.Errorf("ensuring artifact: %w", err)
|
||||
}
|
||||
row, err := r.GetByKey(ctx, a.MediaFileID, a.Format, a.ParamsHash)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
return row, tag.RowsAffected() > 0, nil
|
||||
}
|
||||
|
||||
// GetByID returns an artifact by id, or ErrNotFound.
|
||||
func (r *ArtifactRepository) GetByID(ctx context.Context, id string) (*Artifact, error) {
|
||||
a, err := scanArtifact(r.pool.QueryRow(ctx, `SELECT `+artifactColumns+` FROM download_artifacts WHERE id = $1`, id))
|
||||
if err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil, ErrNotFound
|
||||
}
|
||||
return nil, fmt.Errorf("getting artifact: %w", err)
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// GetByKey returns the artifact for a (media_file_id, format, params_hash), or ErrNotFound.
|
||||
func (r *ArtifactRepository) GetByKey(ctx context.Context, mediaFileID int, format, paramsHash string) (*Artifact, error) {
|
||||
a, err := scanArtifact(r.pool.QueryRow(ctx,
|
||||
`SELECT `+artifactColumns+` FROM download_artifacts
|
||||
WHERE media_file_id = $1 AND format = $2 AND params_hash = $3`,
|
||||
mediaFileID, format, paramsHash,
|
||||
))
|
||||
if err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil, ErrNotFound
|
||||
}
|
||||
return nil, fmt.Errorf("getting artifact by key: %w", err)
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// ClaimNext atomically claims one runnable job: a queued row whose backoff has
|
||||
// elapsed, or a running row whose lease has expired (lease stealing). FOR UPDATE
|
||||
// SKIP LOCKED makes concurrent workers (and nodes) safe without double-encoding.
|
||||
// Returns ErrNoArtifactJob when nothing is claimable.
|
||||
func (r *ArtifactRepository) ClaimNext(ctx context.Context, owner string, lease time.Duration) (*Artifact, error) {
|
||||
leaseSecs := int(lease.Seconds())
|
||||
if leaseSecs <= 0 {
|
||||
leaseSecs = 60
|
||||
}
|
||||
a, err := scanArtifact(r.pool.QueryRow(ctx,
|
||||
`UPDATE download_artifacts
|
||||
SET status = 'running', lease_owner = $1, lease_expires_at = now() + make_interval(secs => $2),
|
||||
attempts = attempts + 1
|
||||
WHERE id = (
|
||||
SELECT id FROM download_artifacts
|
||||
WHERE (status = 'queued' AND (next_retry_at IS NULL OR next_retry_at <= now()))
|
||||
OR (status = 'running' AND lease_expires_at < now())
|
||||
ORDER BY created_at
|
||||
LIMIT 1
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
RETURNING `+artifactColumns,
|
||||
owner, leaseSecs,
|
||||
))
|
||||
if err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil, ErrNoArtifactJob
|
||||
}
|
||||
return nil, fmt.Errorf("claiming artifact job: %w", err)
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// Heartbeat extends a running job's lease while it is still owned by owner.
|
||||
// Returns false when the lease was lost (another worker stole it).
|
||||
func (r *ArtifactRepository) Heartbeat(ctx context.Context, id, owner string, lease time.Duration) (bool, error) {
|
||||
leaseSecs := int(lease.Seconds())
|
||||
if leaseSecs <= 0 {
|
||||
leaseSecs = 60
|
||||
}
|
||||
tag, err := r.pool.Exec(ctx,
|
||||
`UPDATE download_artifacts SET lease_expires_at = now() + make_interval(secs => $3)
|
||||
WHERE id = $1 AND lease_owner = $2 AND status = 'running'`,
|
||||
id, owner, leaseSecs,
|
||||
)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("heartbeating artifact: %w", err)
|
||||
}
|
||||
return tag.RowsAffected() > 0, nil
|
||||
}
|
||||
|
||||
// MarkReady transitions a job to ready, records its size/path, and clears the
|
||||
// lease. The write is fenced on (lease_owner, status='running') so a worker that
|
||||
// lost its lease — e.g. a slow encode whose lease expired and was reclaimed by
|
||||
// another node — cannot flip a row it no longer owns. Returns false when the
|
||||
// fence rejected the write (the lease was lost); the caller must then NOT flip
|
||||
// linked downloads, leaving that to the current owner.
|
||||
func (r *ArtifactRepository) MarkReady(ctx context.Context, id, owner, outputPath string, fileSize int64) (bool, error) {
|
||||
tag, err := r.pool.Exec(ctx,
|
||||
`UPDATE download_artifacts
|
||||
SET status = 'ready', output_path = $2, file_size = $3, error_message = '',
|
||||
completed_at = now(), last_used_at = now(),
|
||||
lease_owner = NULL, lease_expires_at = NULL, next_retry_at = NULL
|
||||
WHERE id = $1 AND lease_owner = $4 AND status = 'running'`,
|
||||
id, outputPath, fileSize, owner,
|
||||
)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("marking artifact ready: %w", err)
|
||||
}
|
||||
return tag.RowsAffected() > 0, nil
|
||||
}
|
||||
|
||||
// MarkFailedOrRetry records a failed attempt. attempts was already incremented
|
||||
// at claim time, so attempts >= max_attempts means terminal (status=failed);
|
||||
// otherwise the row returns to queued behind a backoff gate. The write is fenced
|
||||
// on (lease_owner, status='running'): terminal reports whether the job went
|
||||
// terminal, and applied is false when the fence rejected the write (lease lost),
|
||||
// in which case the caller must not fail linked downloads.
|
||||
func (r *ArtifactRepository) MarkFailedOrRetry(ctx context.Context, id, owner, errMsg string, backoff time.Duration) (terminal bool, applied bool, err error) {
|
||||
backoffSecs := int(backoff.Seconds())
|
||||
if backoffSecs <= 0 {
|
||||
backoffSecs = 30
|
||||
}
|
||||
err = r.pool.QueryRow(ctx,
|
||||
`UPDATE download_artifacts
|
||||
SET status = CASE WHEN attempts >= max_attempts THEN 'failed' ELSE 'queued' END,
|
||||
error_message = $2,
|
||||
next_retry_at = CASE WHEN attempts >= max_attempts THEN NULL ELSE now() + make_interval(secs => $3) END,
|
||||
completed_at = CASE WHEN attempts >= max_attempts THEN now() ELSE NULL END,
|
||||
lease_owner = NULL, lease_expires_at = NULL
|
||||
WHERE id = $1 AND lease_owner = $4 AND status = 'running'
|
||||
RETURNING status = 'failed'`,
|
||||
id, errMsg, backoffSecs, owner,
|
||||
).Scan(&terminal)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return false, false, nil // lease lost; the current owner resolves the job
|
||||
}
|
||||
if err != nil {
|
||||
return false, false, fmt.Errorf("marking artifact failed/retry: %w", err)
|
||||
}
|
||||
return terminal, true, nil
|
||||
}
|
||||
|
||||
// reclaimedArtifact reports a row recovered by the startup sweep.
|
||||
type reclaimedArtifact struct {
|
||||
ID string
|
||||
Terminal bool // true when the row exhausted attempts and is now failed
|
||||
}
|
||||
|
||||
// ReclaimExpiredLeases resets running rows whose lease has expired: back to
|
||||
// queued, or to failed when attempts are exhausted. This is the startup sweep
|
||||
// that guarantees no crash can strand a job in `running` forever.
|
||||
func (r *ArtifactRepository) ReclaimExpiredLeases(ctx context.Context) ([]reclaimedArtifact, error) {
|
||||
rows, err := r.pool.Query(ctx,
|
||||
`UPDATE download_artifacts
|
||||
SET status = CASE WHEN attempts >= max_attempts THEN 'failed' ELSE 'queued' END,
|
||||
lease_owner = NULL, lease_expires_at = NULL,
|
||||
error_message = CASE WHEN attempts >= max_attempts THEN 'exceeded max attempts after lease expiry' ELSE error_message END,
|
||||
completed_at = CASE WHEN attempts >= max_attempts THEN now() ELSE completed_at END
|
||||
WHERE status = 'running' AND lease_expires_at < now()
|
||||
RETURNING id, status = 'failed'`,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("reclaiming expired leases: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
var out []reclaimedArtifact
|
||||
for rows.Next() {
|
||||
var rc reclaimedArtifact
|
||||
if err := rows.Scan(&rc.ID, &rc.Terminal); err != nil {
|
||||
return nil, fmt.Errorf("scanning reclaimed artifact: %w", err)
|
||||
}
|
||||
out = append(out, rc)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// Requeue forces a ready/failed artifact back to queued (e.g. when its
|
||||
// output_path is missing on disk). The deterministic output_path is preserved.
|
||||
// Returns ErrNotFound when the row no longer exists (e.g. a concurrent sweep
|
||||
// deleted it) so callers never keep using a dead artifact id.
|
||||
func (r *ArtifactRepository) Requeue(ctx context.Context, id string) error {
|
||||
tag, err := r.pool.Exec(ctx,
|
||||
`UPDATE download_artifacts
|
||||
SET status = 'queued', attempts = 0, error_message = '', next_retry_at = NULL,
|
||||
lease_owner = NULL, lease_expires_at = NULL, completed_at = NULL
|
||||
WHERE id = $1`,
|
||||
id,
|
||||
)
|
||||
if err != nil {
|
||||
return fmt.Errorf("requeuing artifact: %w", err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return ErrNotFound
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// TouchLastUsed bumps last_used_at for LRU accounting (called on serve).
|
||||
func (r *ArtifactRepository) TouchLastUsed(ctx context.Context, id string) error {
|
||||
_, err := r.pool.Exec(ctx, `UPDATE download_artifacts SET last_used_at = now() WHERE id = $1`, id)
|
||||
if err != nil {
|
||||
return fmt.Errorf("touching artifact: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ListReady returns ready artifacts ordered by least-recently-used first.
|
||||
func (r *ArtifactRepository) ListReady(ctx context.Context) ([]*Artifact, error) {
|
||||
rows, err := r.pool.Query(ctx,
|
||||
`SELECT `+artifactColumns+` FROM download_artifacts WHERE status = 'ready' ORDER BY last_used_at ASC`)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("listing ready artifacts: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
return scanArtifacts(rows)
|
||||
}
|
||||
|
||||
func scanArtifacts(rows pgx.Rows) ([]*Artifact, error) {
|
||||
var out []*Artifact
|
||||
for rows.Next() {
|
||||
a, err := scanArtifact(rows)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("scanning artifact row: %w", err)
|
||||
}
|
||||
out = append(out, a)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// TotalReadyBytes returns the sum of ready artifact sizes (LRU budget input).
|
||||
func (r *ArtifactRepository) TotalReadyBytes(ctx context.Context) (int64, error) {
|
||||
var total int64
|
||||
if err := r.pool.QueryRow(ctx,
|
||||
`SELECT COALESCE(SUM(file_size), 0) FROM download_artifacts WHERE status = 'ready'`).Scan(&total); err != nil {
|
||||
return 0, fmt.Errorf("summing ready artifacts: %w", err)
|
||||
}
|
||||
return total, nil
|
||||
}
|
||||
|
||||
// HasActiveLink reports whether any valid download row — managed or ephemeral
|
||||
// (device-less web) — still references the artifact. Completed rows are
|
||||
// retained because they remain re-downloadable handles; evicting an artifact a
|
||||
// live row references would 404 a download the API advertises as servable.
|
||||
func (r *ArtifactRepository) HasActiveLink(ctx context.Context, artifactID string) (bool, error) {
|
||||
var exists bool
|
||||
if err := r.pool.QueryRow(ctx,
|
||||
`SELECT EXISTS(SELECT 1 FROM downloads
|
||||
WHERE artifact_id = $1
|
||||
AND status NOT IN ('cancelled','failed','revoked'))`,
|
||||
artifactID,
|
||||
).Scan(&exists); err != nil {
|
||||
return false, fmt.Errorf("checking artifact links: %w", err)
|
||||
}
|
||||
return exists, nil
|
||||
}
|
||||
|
||||
// ListFailedBefore returns terminally-failed artifacts cold since cutoff
|
||||
// (last_used_at). Their linked downloads were already flipped to 'failed' by
|
||||
// reconciliation, so the rows serve nothing and only block re-attempts.
|
||||
func (r *ArtifactRepository) ListFailedBefore(ctx context.Context, cutoff time.Time) ([]*Artifact, error) {
|
||||
rows, err := r.pool.Query(ctx,
|
||||
`SELECT `+artifactColumns+` FROM download_artifacts
|
||||
WHERE status = 'failed' AND last_used_at < $1`, cutoff)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("listing failed artifacts: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
return scanArtifacts(rows)
|
||||
}
|
||||
|
||||
// ListUnlinkedReadyBefore returns ready artifacts referenced by NO download
|
||||
// row at all (every linking row deleted) and unused since cutoff. These are
|
||||
// pure orphans: nothing can ever serve them again.
|
||||
func (r *ArtifactRepository) ListUnlinkedReadyBefore(ctx context.Context, cutoff time.Time) ([]*Artifact, error) {
|
||||
rows, err := r.pool.Query(ctx,
|
||||
`SELECT `+artifactColumns+` FROM download_artifacts a
|
||||
WHERE a.status = 'ready' AND a.last_used_at < $1
|
||||
AND NOT EXISTS (SELECT 1 FROM downloads d WHERE d.artifact_id = a.id)`, cutoff)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("listing unlinked artifacts: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
return scanArtifacts(rows)
|
||||
}
|
||||
|
||||
// DeleteArtifact removes an artifact row.
|
||||
func (r *ArtifactRepository) DeleteArtifact(ctx context.Context, id string) error {
|
||||
_, err := r.pool.Exec(ctx, `DELETE FROM download_artifacts WHERE id = $1`, id)
|
||||
if err != nil {
|
||||
return fmt.Errorf("deleting artifact: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,257 @@
|
||||
package downloads
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/idgen"
|
||||
)
|
||||
|
||||
func newArtifactTestRepo(t *testing.T) (*ArtifactRepository, *pgxpool.Pool, int) {
|
||||
t.Helper()
|
||||
pool := newDownloadsTestPool(t)
|
||||
ctx := context.Background()
|
||||
var present *string
|
||||
if err := pool.QueryRow(ctx, `SELECT to_regclass('public.download_artifacts')::text`).Scan(&present); err != nil {
|
||||
t.Fatalf("check download_artifacts: %v", err)
|
||||
}
|
||||
if present == nil {
|
||||
t.Skip("download_artifacts migration has not been applied")
|
||||
}
|
||||
|
||||
suffix := time.Now().UnixNano()
|
||||
var folderID int
|
||||
if err := pool.QueryRow(ctx,
|
||||
`INSERT INTO media_folders (type, name) VALUES ('movies', $1) RETURNING id`,
|
||||
fmt.Sprintf("Artifacts Test %d", suffix),
|
||||
).Scan(&folderID); err != nil {
|
||||
t.Fatalf("seed folder: %v", err)
|
||||
}
|
||||
var fileID int
|
||||
if err := pool.QueryRow(ctx,
|
||||
`INSERT INTO media_files (media_folder_id, file_path) VALUES ($1, $2) RETURNING id`,
|
||||
folderID, fmt.Sprintf("/tmp/artifact-%d.mkv", suffix),
|
||||
).Scan(&fileID); err != nil {
|
||||
t.Fatalf("seed media file: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
_, _ = pool.Exec(ctx, `DELETE FROM download_artifacts WHERE media_file_id = $1`, fileID)
|
||||
_, _ = pool.Exec(ctx, `DELETE FROM media_files WHERE id = $1`, fileID)
|
||||
_, _ = pool.Exec(ctx, `DELETE FROM media_folders WHERE id = $1`, folderID)
|
||||
})
|
||||
return NewArtifactRepository(pool), pool, fileID
|
||||
}
|
||||
|
||||
func newArtifact(t *testing.T, fileID int, hash string) *Artifact {
|
||||
t.Helper()
|
||||
id, err := idgen.NextID()
|
||||
if err != nil {
|
||||
t.Fatalf("id: %v", err)
|
||||
}
|
||||
return &Artifact{
|
||||
ID: id, MediaFileID: fileID, Format: "transcode", ParamsHash: hash,
|
||||
Container: "mp4", CodecVideo: "h264", CodecAudio: "aac", Resolution: "1080p",
|
||||
AudioTrackIndex: -1, OutputPath: "/tmp/" + id + ".mp4", MaxAttempts: 3,
|
||||
}
|
||||
}
|
||||
|
||||
// TestArtifactQueueClaimAndLeaseRecovery is the Phase 3 / invariant-3 acceptance
|
||||
// test: a crash mid-encode (an expired lease) is recovered on the next sweep so
|
||||
// the job re-enqueues and reaches ready, and concurrent workers never claim the
|
||||
// same job twice (no double-encode).
|
||||
func TestArtifactQueueClaimAndLeaseRecovery(t *testing.T) {
|
||||
repo, pool, fileID := newArtifactTestRepo(t)
|
||||
ctx := context.Background()
|
||||
|
||||
a := newArtifact(t, fileID, "hash-recovery")
|
||||
row, created, err := repo.EnsureQueued(ctx, a)
|
||||
if err != nil || !created || row.Status != ArtifactQueued {
|
||||
t.Fatalf("EnsureQueued = (%+v, created=%v, %v), want new queued row", row, created, err)
|
||||
}
|
||||
|
||||
// Dedup: a second ensure for the same key returns the same row, not a new one.
|
||||
dup, created2, err := repo.EnsureQueued(ctx, newArtifact(t, fileID, "hash-recovery"))
|
||||
if err != nil || created2 || dup.ID != row.ID {
|
||||
t.Fatalf("dedup EnsureQueued = (%s, created=%v, %v), want existing %s", dup.ID, created2, err, row.ID)
|
||||
}
|
||||
|
||||
// Worker 1 claims the job; worker 2 finds nothing (no double-encode).
|
||||
claim, err := repo.ClaimNext(ctx, "worker-1", time.Minute)
|
||||
if err != nil || claim.ID != row.ID || claim.Status != ArtifactRunning || claim.Attempts != 1 {
|
||||
t.Fatalf("ClaimNext = (%+v, %v), want running attempts=1", claim, err)
|
||||
}
|
||||
if _, err := repo.ClaimNext(ctx, "worker-2", time.Minute); !errors.Is(err, ErrNoArtifactJob) {
|
||||
t.Fatalf("second ClaimNext err = %v, want ErrNoArtifactJob", err)
|
||||
}
|
||||
|
||||
// Simulate a crash: expire the lease, then run the startup sweep.
|
||||
if _, err := pool.Exec(ctx, `UPDATE download_artifacts SET lease_expires_at = now() - interval '1 minute' WHERE id = $1`, row.ID); err != nil {
|
||||
t.Fatalf("expire lease: %v", err)
|
||||
}
|
||||
reclaimed, err := repo.ReclaimExpiredLeases(ctx)
|
||||
if err != nil {
|
||||
t.Fatalf("reclaim: %v", err)
|
||||
}
|
||||
if len(reclaimed) != 1 || reclaimed[0].ID != row.ID || reclaimed[0].Terminal {
|
||||
t.Fatalf("reclaimed = %+v, want one non-terminal %s", reclaimed, row.ID)
|
||||
}
|
||||
back, err := repo.GetByID(ctx, row.ID)
|
||||
if err != nil || back.Status != ArtifactQueued {
|
||||
t.Fatalf("after reclaim status = %v (%v), want queued (no permanent running)", back.Status, err)
|
||||
}
|
||||
|
||||
// Another worker reclaims and completes it.
|
||||
claim2, err := repo.ClaimNext(ctx, "worker-2", time.Minute)
|
||||
if err != nil || claim2.ID != row.ID || claim2.Attempts != 2 {
|
||||
t.Fatalf("reclaim ClaimNext = (%+v, %v), want attempts=2", claim2, err)
|
||||
}
|
||||
if applied, err := repo.MarkReady(ctx, row.ID, "worker-2", claim2.OutputPath, 4242); err != nil || !applied {
|
||||
t.Fatalf("MarkReady = (%v, %v), want (true, nil)", applied, err)
|
||||
}
|
||||
done, err := repo.GetByKey(ctx, fileID, "transcode", "hash-recovery")
|
||||
if err != nil || done.Status != ArtifactReady || done.FileSize != 4242 {
|
||||
t.Fatalf("final = (%+v, %v), want ready size=4242", done, err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestArtifactRetryUntilTerminal verifies attempt counting and backoff: a job
|
||||
// retries behind its backoff gate until max_attempts, then goes terminal-failed.
|
||||
func TestArtifactRetryUntilTerminal(t *testing.T) {
|
||||
repo, pool, fileID := newArtifactTestRepo(t)
|
||||
ctx := context.Background()
|
||||
|
||||
row, _, err := repo.EnsureQueued(ctx, newArtifact(t, fileID, "hash-retry"))
|
||||
if err != nil {
|
||||
t.Fatalf("EnsureQueued: %v", err)
|
||||
}
|
||||
|
||||
for attempt := 1; attempt <= 3; attempt++ {
|
||||
claim, err := repo.ClaimNext(ctx, "worker", time.Minute)
|
||||
if err != nil {
|
||||
t.Fatalf("attempt %d ClaimNext: %v", attempt, err)
|
||||
}
|
||||
if claim.Attempts != attempt {
|
||||
t.Fatalf("attempt %d: attempts = %d", attempt, claim.Attempts)
|
||||
}
|
||||
terminal, applied, err := repo.MarkFailedOrRetry(ctx, row.ID, "worker", "boom", 30*time.Second)
|
||||
if err != nil || !applied {
|
||||
t.Fatalf("attempt %d MarkFailedOrRetry = (%v, %v, %v)", attempt, terminal, applied, err)
|
||||
}
|
||||
if attempt < 3 {
|
||||
if terminal {
|
||||
t.Fatalf("attempt %d went terminal too early", attempt)
|
||||
}
|
||||
// Behind the backoff gate the job is not yet claimable.
|
||||
if _, err := repo.ClaimNext(ctx, "worker", time.Minute); !errors.Is(err, ErrNoArtifactJob) {
|
||||
t.Fatalf("attempt %d: job claimable during backoff", attempt)
|
||||
}
|
||||
if _, err := pool.Exec(ctx, `UPDATE download_artifacts SET next_retry_at = now() - interval '1 second' WHERE id = $1`, row.ID); err != nil {
|
||||
t.Fatalf("clear backoff: %v", err)
|
||||
}
|
||||
} else if !terminal {
|
||||
t.Fatalf("final attempt should be terminal")
|
||||
}
|
||||
}
|
||||
|
||||
failed, err := repo.GetByID(ctx, row.ID)
|
||||
if err != nil || failed.Status != ArtifactFailed {
|
||||
t.Fatalf("final status = %v (%v), want failed", failed.Status, err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestArtifactMarkFencedByOwner verifies MarkReady/MarkFailedOrRetry only apply
|
||||
// for the worker that currently holds the lease, so a worker whose lease was
|
||||
// stolen (e.g. a slow encode reclaimed by another node) cannot flip a job it no
|
||||
// longer owns — the double-encode guard behind invariant 3.
|
||||
func TestArtifactMarkFencedByOwner(t *testing.T) {
|
||||
repo, _, fileID := newArtifactTestRepo(t)
|
||||
ctx := context.Background()
|
||||
|
||||
row, _, err := repo.EnsureQueued(ctx, newArtifact(t, fileID, "hash-fence"))
|
||||
if err != nil {
|
||||
t.Fatalf("EnsureQueued: %v", err)
|
||||
}
|
||||
if _, err := repo.ClaimNext(ctx, "owner-1", time.Minute); err != nil {
|
||||
t.Fatalf("ClaimNext: %v", err)
|
||||
}
|
||||
|
||||
// A non-owner cannot mark the job ready or failed.
|
||||
if applied, err := repo.MarkReady(ctx, row.ID, "owner-2", "/tmp/x.mp4", 10); err != nil || applied {
|
||||
t.Fatalf("MarkReady(non-owner) = (%v, %v), want (false, nil)", applied, err)
|
||||
}
|
||||
if _, applied, err := repo.MarkFailedOrRetry(ctx, row.ID, "owner-2", "boom", time.Second); err != nil || applied {
|
||||
t.Fatalf("MarkFailedOrRetry(non-owner) applied = %v (%v), want false", applied, err)
|
||||
}
|
||||
|
||||
// The job remains claimable-state 'running' and untouched.
|
||||
mid, err := repo.GetByID(ctx, row.ID)
|
||||
if err != nil || mid.Status != ArtifactRunning {
|
||||
t.Fatalf("status after fenced writes = %v (%v), want running", mid.Status, err)
|
||||
}
|
||||
|
||||
// The real owner succeeds.
|
||||
if applied, err := repo.MarkReady(ctx, row.ID, "owner-1", "/tmp/x.mp4", 10); err != nil || !applied {
|
||||
t.Fatalf("MarkReady(owner) = (%v, %v), want (true, nil)", applied, err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHasActiveLinkCoversEphemeralRows pins the eviction guard: an ephemeral
|
||||
// (device-less web) download row must protect its artifact from LRU cleanup
|
||||
// exactly like a managed row does, and terminal rows must not.
|
||||
func TestHasActiveLinkCoversEphemeralRows(t *testing.T) {
|
||||
repo, pool, fileID := newArtifactTestRepo(t)
|
||||
ctx := context.Background()
|
||||
|
||||
art := newArtifact(t, fileID, fmt.Sprintf("hash-link-%d", time.Now().UnixNano()))
|
||||
if _, _, err := repo.EnsureQueued(ctx, art); err != nil {
|
||||
t.Fatalf("ensure artifact: %v", err)
|
||||
}
|
||||
|
||||
var userID int
|
||||
if err := pool.QueryRow(ctx,
|
||||
`INSERT INTO users (username, role, download_allowed) VALUES ($1, 'user', true) RETURNING id`,
|
||||
fmt.Sprintf("linkuser-%d", time.Now().UnixNano()),
|
||||
).Scan(&userID); err != nil {
|
||||
t.Fatalf("seed user: %v", err)
|
||||
}
|
||||
contentID := fmt.Sprintf("link-content-%d", time.Now().UnixNano())
|
||||
t.Cleanup(func() {
|
||||
_, _ = pool.Exec(ctx, `DELETE FROM downloads WHERE user_id = $1`, userID)
|
||||
_, _ = pool.Exec(ctx, `DELETE FROM users WHERE id = $1`, userID)
|
||||
})
|
||||
|
||||
dlRepo := NewRepository(pool)
|
||||
now := time.Now()
|
||||
dlID := fmt.Sprintf("dl-link-%d", now.UnixNano())
|
||||
if err := dlRepo.Create(ctx, &Download{
|
||||
ID: dlID, UserID: userID, MediaFileID: fileID, ContentID: contentID,
|
||||
Kind: KindQueued, Status: StatusReady, Format: FormatTranscode,
|
||||
ArtifactID: art.ID, FileSize: 1024, CreatedAt: now, UpdatedAt: now,
|
||||
}); err != nil {
|
||||
t.Fatalf("create ephemeral download: %v", err)
|
||||
}
|
||||
|
||||
active, err := repo.HasActiveLink(ctx, art.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("HasActiveLink: %v", err)
|
||||
}
|
||||
if !active {
|
||||
t.Fatal("ephemeral ready row must protect its artifact from eviction")
|
||||
}
|
||||
|
||||
if _, err := pool.Exec(ctx, `UPDATE downloads SET status = 'cancelled' WHERE id = $1`, dlID); err != nil {
|
||||
t.Fatalf("cancel download: %v", err)
|
||||
}
|
||||
active, err = repo.HasActiveLink(ctx, art.ID)
|
||||
if err != nil {
|
||||
t.Fatalf("HasActiveLink after cancel: %v", err)
|
||||
}
|
||||
if active {
|
||||
t.Fatal("terminal-only links must not protect an artifact")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,155 @@
|
||||
package downloads
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/config"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
)
|
||||
|
||||
func TestParamsHashStableAndDistinct(t *testing.T) {
|
||||
base := paramsHash("transcode", "mp4", "h264", "aac", "1080p", -1, 10000, false)
|
||||
if base == "" || len(base) != 64 {
|
||||
t.Fatalf("params hash should be a 64-char sha256 hex, got %q", base)
|
||||
}
|
||||
// Deterministic for identical inputs (dedup key).
|
||||
if again := paramsHash("transcode", "mp4", "h264", "aac", "1080p", -1, 10000, false); again != base {
|
||||
t.Fatalf("params hash not stable: %q != %q", base, again)
|
||||
}
|
||||
// Distinct when any parameter differs.
|
||||
for _, other := range []string{
|
||||
paramsHash("remux", "mp4", "h264", "aac", "1080p", -1, 10000, false),
|
||||
paramsHash("transcode", "mp4", "hevc", "aac", "1080p", -1, 10000, false),
|
||||
paramsHash("transcode", "mp4", "h264", "aac", "720p", -1, 10000, false),
|
||||
paramsHash("transcode", "mp4", "h264", "aac", "1080p", 1, 10000, false),
|
||||
paramsHash("transcode", "mp4", "h264", "aac", "1080p", -1, 5000, false),
|
||||
paramsHash("transcode", "mp4", "h264", "aac", "1080p", -1, 10000, true),
|
||||
} {
|
||||
if other == base {
|
||||
t.Fatalf("params hash collision: %q", other)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtifactOutputPathDeterministic(t *testing.T) {
|
||||
p1 := artifactOutputPath("/var/artifacts", 42, "transcode", "abcdef0123456789deadbeef")
|
||||
p2 := artifactOutputPath("/var/artifacts", 42, "transcode", "abcdef0123456789deadbeef")
|
||||
if p1 != p2 {
|
||||
t.Fatalf("output path not deterministic: %q != %q", p1, p2)
|
||||
}
|
||||
if !strings.HasPrefix(p1, "/var/artifacts/") || !strings.HasSuffix(p1, ".mp4") {
|
||||
t.Fatalf("unexpected output path %q", p1)
|
||||
}
|
||||
if !strings.Contains(p1, "42_transcode_") {
|
||||
t.Fatalf("output path missing identity components: %q", p1)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEffectiveArtifactDir(t *testing.T) {
|
||||
// Explicit config wins verbatim.
|
||||
if got := effectiveArtifactDir("/data/artifacts", "/tmp/silo-transcode"); got != "/data/artifacts" {
|
||||
t.Fatalf("explicit dir = %q, want /data/artifacts", got)
|
||||
}
|
||||
// Unset: a sibling of the transcode dir, never inside it (the transcode
|
||||
// cleanup sweep would otherwise delete a nested artifact dir) and never the
|
||||
// process cwd (a relative/empty path).
|
||||
got := effectiveArtifactDir("", "/var/lib/silo/transcode")
|
||||
if got != "/var/lib/silo/silo-download-artifacts" {
|
||||
t.Fatalf("default dir = %q, want sibling of transcode dir", got)
|
||||
}
|
||||
if strings.HasPrefix(got, "/var/lib/silo/transcode/") {
|
||||
t.Fatalf("default dir %q is nested under the transcode dir", got)
|
||||
}
|
||||
// Unset transcode dir falls back to the absolute default root, not "".
|
||||
fallback := effectiveArtifactDir("", "")
|
||||
if !strings.HasPrefix(fallback, "/") {
|
||||
t.Fatalf("fallback dir %q is not absolute", fallback)
|
||||
}
|
||||
}
|
||||
|
||||
type fakeUserRepo struct{ user *models.User }
|
||||
|
||||
func (f fakeUserRepo) GetByID(context.Context, int) (*models.User, error) { return f.user, nil }
|
||||
|
||||
func TestCapabilityQualityPresetsGating(t *testing.T) {
|
||||
newSvc := func(user *models.User, transcodeEnabled bool) *Service {
|
||||
cfg := config.DownloadConfig{Enabled: true, TranscodeEnabled: transcodeEnabled}
|
||||
return NewService(nil, nil, nil, nil, nil, nil, fakeUserRepo{user}, nil, nil, &cfg)
|
||||
}
|
||||
allowAll := &models.User{DownloadAllowed: true, DownloadTranscodeAllowed: true}
|
||||
|
||||
// No artifact pipeline wired → only original is fulfillable.
|
||||
svc := newSvc(allowAll, true)
|
||||
capInfo, err := svc.Capability(context.Background(), 1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := strings.Join(capInfo.QualityPresets, ","); got != "original" {
|
||||
t.Fatalf("quality presets without pipeline = %q, want original", got)
|
||||
}
|
||||
|
||||
// Pipeline wired + transcode server/user gates open → full bitrate ladder.
|
||||
svc = newSvc(allowAll, true)
|
||||
svc.SetArtifactManager(&ArtifactManager{})
|
||||
capInfo, _ = svc.Capability(context.Background(), 1)
|
||||
if got := strings.Join(capInfo.QualityPresets, ","); got != "original,20mbps,10mbps,5mbps,2mbps,1mbps" {
|
||||
t.Fatalf("quality presets with pipeline = %q, want full ladder", got)
|
||||
}
|
||||
|
||||
// Transcode gated off (user flag) → original only.
|
||||
svc = newSvc(&models.User{DownloadAllowed: true, DownloadTranscodeAllowed: false}, true)
|
||||
svc.SetArtifactManager(&ArtifactManager{})
|
||||
capInfo, _ = svc.Capability(context.Background(), 1)
|
||||
if got := strings.Join(capInfo.QualityPresets, ","); got != "original" {
|
||||
t.Fatalf("quality presets with transcode gated = %q, want original", got)
|
||||
}
|
||||
|
||||
// Download permission revoked → an EMPTY array, never nil: the capability
|
||||
// contract documents quality_presets as an array, and a nil slice would
|
||||
// serialize as JSON null and break typed clients.
|
||||
svc = newSvc(&models.User{DownloadAllowed: false}, true)
|
||||
capInfo, _ = svc.Capability(context.Background(), 1)
|
||||
if capInfo.QualityPresets == nil {
|
||||
t.Fatal("quality presets for a denied user must be an empty array, not nil")
|
||||
}
|
||||
if len(capInfo.QualityPresets) != 0 {
|
||||
t.Fatalf("quality presets for a denied user = %v, want empty", capInfo.QualityPresets)
|
||||
}
|
||||
if b, err := json.Marshal(capInfo.QualityPresets); err != nil || string(b) != "[]" {
|
||||
t.Fatalf("quality presets serialize to %s (%v), want []", b, err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTriggerDrainDoesNotBlockCaller pins the async-dispatch contract:
|
||||
// Ensure runs on request goroutines and the kick drains the whole encode
|
||||
// queue (ffmpeg included), so triggerDrain must return without waiting on it.
|
||||
func TestTriggerDrainDoesNotBlockCaller(t *testing.T) {
|
||||
m := &ArtifactManager{}
|
||||
started := make(chan struct{})
|
||||
release := make(chan struct{})
|
||||
m.SetKick(func() {
|
||||
close(started)
|
||||
<-release
|
||||
})
|
||||
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
m.triggerDrain()
|
||||
close(done)
|
||||
}()
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("triggerDrain blocked on the kick; it must dispatch asynchronously")
|
||||
}
|
||||
select {
|
||||
case <-started:
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("kick was never invoked")
|
||||
}
|
||||
close(release)
|
||||
}
|
||||
@@ -0,0 +1,582 @@
|
||||
package downloads
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"os"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/config"
|
||||
"github.com/Silo-Server/silo-server/internal/idgen"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
"github.com/Silo-Server/silo-server/internal/playback"
|
||||
)
|
||||
|
||||
const (
|
||||
artifactLease = 2 * time.Minute
|
||||
artifactHeartbeat = 40 * time.Second
|
||||
artifactMaxAttempts = 3
|
||||
)
|
||||
|
||||
// EncodePreparer produces a single finalized file for an artifact. The default
|
||||
// implementation calls playback.PrepareFile; tests substitute a fake.
|
||||
type EncodePreparer interface {
|
||||
PrepareFile(ctx context.Context, opts playback.TranscodeOpts, outputPath string) error
|
||||
}
|
||||
|
||||
type playbackPreparer struct{}
|
||||
|
||||
func (playbackPreparer) PrepareFile(ctx context.Context, opts playback.TranscodeOpts, outputPath string) error {
|
||||
return playback.PrepareFile(ctx, opts, outputPath)
|
||||
}
|
||||
|
||||
// NewPlaybackPreparer returns the production EncodePreparer (ffmpeg-backed).
|
||||
func NewPlaybackPreparer() EncodePreparer { return playbackPreparer{} }
|
||||
|
||||
// ArtifactNotifier publishes an event when a linked download changes state.
|
||||
type ArtifactNotifier func(ctx context.Context, d *Download)
|
||||
|
||||
// ArtifactManager owns the durable encode queue: it ensures/deduplicates encode
|
||||
// jobs, drains them through a bounded worker pool with leased heartbeats, and
|
||||
// recovers stranded jobs on startup.
|
||||
type ArtifactManager struct {
|
||||
repo *ArtifactRepository
|
||||
downloads *Repository
|
||||
fileRepo FileResolver
|
||||
preparer EncodePreparer
|
||||
owner string
|
||||
liveCfg func() *config.Config
|
||||
notify ArtifactNotifier
|
||||
|
||||
mu sync.Mutex
|
||||
kick func()
|
||||
lastDiskSweep time.Time
|
||||
lastStaleSweep time.Time
|
||||
}
|
||||
|
||||
// maintenanceInterval spaces the disk-presence and stale-row sweeps: both are
|
||||
// O(cache size) (stats / extra queries) and their failure modes self-heal, so
|
||||
// running them on every 30s task tick is steady-state waste that grows with
|
||||
// the artifact cache. The first run after startup always executes.
|
||||
const maintenanceInterval = time.Hour
|
||||
|
||||
// maintenanceDue reports whether the sweep guarded by last is due, advancing
|
||||
// the stamp when it is.
|
||||
func (m *ArtifactManager) maintenanceDue(last *time.Time) bool {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if !last.IsZero() && time.Since(*last) < maintenanceInterval {
|
||||
return false
|
||||
}
|
||||
*last = time.Now()
|
||||
return true
|
||||
}
|
||||
|
||||
// NewArtifactManager constructs an ArtifactManager. liveCfg reads the current
|
||||
// config (artifact dir, worker-pool size, byte budget, ffmpeg/hwaccel); owner is
|
||||
// this node's id for lease ownership; notify (optional) publishes ready/failed.
|
||||
func NewArtifactManager(
|
||||
repo *ArtifactRepository,
|
||||
downloadRepo *Repository,
|
||||
fileRepo FileResolver,
|
||||
preparer EncodePreparer,
|
||||
owner string,
|
||||
liveCfg func() *config.Config,
|
||||
notify ArtifactNotifier,
|
||||
) *ArtifactManager {
|
||||
if preparer == nil {
|
||||
preparer = playbackPreparer{}
|
||||
}
|
||||
if owner == "" {
|
||||
owner = "node"
|
||||
}
|
||||
return &ArtifactManager{
|
||||
repo: repo, downloads: downloadRepo, fileRepo: fileRepo, preparer: preparer,
|
||||
owner: owner, liveCfg: liveCfg, notify: notify,
|
||||
}
|
||||
}
|
||||
|
||||
// SetKick wires a low-latency drain trigger (e.g. taskmanager RunTask) invoked
|
||||
// when a new job is enqueued.
|
||||
func (m *ArtifactManager) SetKick(kick func()) {
|
||||
m.mu.Lock()
|
||||
m.kick = kick
|
||||
m.mu.Unlock()
|
||||
}
|
||||
|
||||
// Ready returns a ready artifact for serving and bumps its LRU timestamp.
|
||||
// Returns ErrDownloadNotActive when the artifact is not yet ready.
|
||||
func (m *ArtifactManager) Ready(ctx context.Context, id string) (*Artifact, error) {
|
||||
a, err := m.repo.GetByID(ctx, id)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if a.Status != ArtifactReady {
|
||||
return nil, fmt.Errorf("artifact is %s: %w", a.Status, ErrDownloadNotActive)
|
||||
}
|
||||
_ = m.repo.TouchLastUsed(ctx, id)
|
||||
return a, nil
|
||||
}
|
||||
|
||||
func (m *ArtifactManager) downloadConfig() config.DownloadConfig {
|
||||
if m.liveCfg != nil {
|
||||
if c := m.liveCfg(); c != nil {
|
||||
return c.Download
|
||||
}
|
||||
}
|
||||
return config.DownloadConfig{}
|
||||
}
|
||||
|
||||
// artifactDir resolves the effective output directory for prepared artifacts,
|
||||
// defaulting under the transcode dir when download.artifact_dir is unset so
|
||||
// encodes never write relative to the process working directory.
|
||||
func (m *ArtifactManager) artifactDir() string {
|
||||
var artifactDir, transcodeDir string
|
||||
if m.liveCfg != nil {
|
||||
if c := m.liveCfg(); c != nil {
|
||||
artifactDir = c.Download.ArtifactDir
|
||||
transcodeDir = c.Playback.TranscodeDir
|
||||
}
|
||||
}
|
||||
return effectiveArtifactDir(artifactDir, transcodeDir)
|
||||
}
|
||||
|
||||
// Ensure deduplicates and (when new) enqueues an encode job for file in the
|
||||
// given format, returning the current artifact row. The deterministic
|
||||
// output_path keeps a reclaimed job idempotent.
|
||||
func (m *ArtifactManager) Ensure(ctx context.Context, file *models.MediaFile, format string, target playback.PrepareTarget) (*Artifact, error) {
|
||||
hash := paramsHash(format, target.Container, target.CodecVideo, target.CodecAudio, target.Resolution, target.AudioTrackIndex, target.TargetBitrateKbps, false)
|
||||
id, err := idgen.NextID()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
a := &Artifact{
|
||||
ID: id,
|
||||
MediaFileID: file.ID,
|
||||
Format: format,
|
||||
ParamsHash: hash,
|
||||
Container: target.Container,
|
||||
CodecVideo: target.CodecVideo,
|
||||
CodecAudio: target.CodecAudio,
|
||||
Resolution: target.Resolution,
|
||||
AudioTrackIndex: target.AudioTrackIndex,
|
||||
TargetBitrateKbps: target.TargetBitrateKbps,
|
||||
OutputPath: artifactOutputPath(m.artifactDir(), file.ID, format, hash),
|
||||
MaxAttempts: artifactMaxAttempts,
|
||||
}
|
||||
row, created, err := m.repo.EnsureQueued(ctx, a)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if row.Status == ArtifactReady {
|
||||
_ = m.repo.TouchLastUsed(ctx, row.ID)
|
||||
return row, nil
|
||||
}
|
||||
// A terminally-failed dedup row would otherwise strand every new download
|
||||
// linked to it in 'preparing' forever (no drain is triggered for an existing
|
||||
// row). Requeue it for a fresh attempt so the new download can resolve — or
|
||||
// fail cleanly via reconciliation once the encode is exhausted again.
|
||||
if row.Status == ArtifactFailed {
|
||||
switch err := m.repo.Requeue(ctx, row.ID); {
|
||||
case errors.Is(err, ErrNotFound):
|
||||
// The failed row was swept between EnsureQueued and Requeue:
|
||||
// create a fresh job instead of linking to a dead artifact id.
|
||||
if row, _, err = m.repo.EnsureQueued(ctx, a); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
case err != nil:
|
||||
return nil, err
|
||||
default:
|
||||
row.Status = ArtifactQueued
|
||||
}
|
||||
m.triggerDrain()
|
||||
return row, nil
|
||||
}
|
||||
if created {
|
||||
m.triggerDrain()
|
||||
}
|
||||
return row, nil
|
||||
}
|
||||
|
||||
func (m *ArtifactManager) triggerDrain() {
|
||||
m.mu.Lock()
|
||||
kick := m.kick
|
||||
m.mu.Unlock()
|
||||
if kick != nil {
|
||||
// Ensure is called on request goroutines and the kick runs the encode
|
||||
// task to completion (the task manager serializes concurrent runs), so
|
||||
// it must never execute inline: a POST /downloads would otherwise block
|
||||
// on the entire queue drain, ffmpeg encodes included.
|
||||
go kick()
|
||||
}
|
||||
}
|
||||
|
||||
// RunOnce performs a startup-safe recovery sweep and then drains the queue
|
||||
// until empty. It is safe to call concurrently across nodes; FOR UPDATE SKIP
|
||||
// LOCKED prevents double-encoding.
|
||||
func (m *ArtifactManager) RunOnce(ctx context.Context) error {
|
||||
m.recover(ctx)
|
||||
return m.drain(ctx)
|
||||
}
|
||||
|
||||
// recover repairs state a crash or lost lease can leave behind: it reclaims
|
||||
// expired-lease running rows (back to queued, or failed when attempts are
|
||||
// exhausted), reconciles linked downloads against terminal artifact states, and
|
||||
// re-queues ready artifacts whose output file is missing on disk. Safe to run
|
||||
// repeatedly (each step is idempotent).
|
||||
func (m *ArtifactManager) recover(ctx context.Context) {
|
||||
if _, err := m.repo.ReclaimExpiredLeases(ctx); err != nil {
|
||||
slog.Warn("download artifact lease reclaim failed", "error", err)
|
||||
}
|
||||
|
||||
// Reconcile downloads stranded in 'preparing' against their artifact's
|
||||
// terminal state: this closes the non-transactional window between an
|
||||
// artifact's MarkReady and its MarkLinkedDownloadsReady, and fails the links
|
||||
// of any artifact that reached 'failed' (including the rows just reclaimed to
|
||||
// failed above) so a download can never sit 'preparing' forever.
|
||||
readyFlipped, failedFlipped, err := m.downloads.ReconcileLinkedDownloads(ctx)
|
||||
if err != nil {
|
||||
slog.Warn("reconciling linked downloads failed", "error", err)
|
||||
} else {
|
||||
for _, d := range readyFlipped {
|
||||
m.publish(ctx, d)
|
||||
}
|
||||
for _, d := range failedFlipped {
|
||||
m.publish(ctx, d)
|
||||
}
|
||||
}
|
||||
|
||||
// Disk-presence sweep: stats every ready file, so it runs on the startup
|
||||
// pass and then hourly rather than on every tick.
|
||||
if !m.maintenanceDue(&m.lastDiskSweep) {
|
||||
return
|
||||
}
|
||||
ready, err := m.repo.ListReady(ctx)
|
||||
if err != nil {
|
||||
slog.Warn("download artifact ready scan failed", "error", err)
|
||||
return
|
||||
}
|
||||
for _, a := range ready {
|
||||
if a.OutputPath == "" {
|
||||
continue
|
||||
}
|
||||
if _, statErr := os.Stat(a.OutputPath); statErr != nil {
|
||||
slog.Warn("download artifact output missing, re-queuing", "artifact_id", a.ID, "path", a.OutputPath)
|
||||
if err := m.repo.Requeue(ctx, a.ID); err != nil {
|
||||
slog.Warn("re-queue artifact failed", "artifact_id", a.ID, "error", err)
|
||||
continue
|
||||
}
|
||||
m.triggerDrain()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// drain claims and encodes jobs through a bounded worker pool until the queue is
|
||||
// empty or the context is canceled.
|
||||
func (m *ArtifactManager) drain(ctx context.Context) error {
|
||||
maxConcurrent := m.downloadConfig().MaxConcurrentPrepares
|
||||
if maxConcurrent <= 0 {
|
||||
maxConcurrent = 2
|
||||
}
|
||||
sem := make(chan struct{}, maxConcurrent)
|
||||
var wg sync.WaitGroup
|
||||
for {
|
||||
// Acquire a worker slot BEFORE claiming. A claimed job is leased but only
|
||||
// heartbeated once encodeOne runs; claiming first and then blocking for a
|
||||
// slot would leave the job leased-but-unattended, so its lease could lapse
|
||||
// while it waits — letting another node steal it and encode the same
|
||||
// output path concurrently. Reserving the slot first closes that window.
|
||||
select {
|
||||
case sem <- struct{}{}:
|
||||
case <-ctx.Done():
|
||||
wg.Wait()
|
||||
return ctx.Err()
|
||||
}
|
||||
job, err := m.repo.ClaimNext(ctx, m.owner, artifactLease)
|
||||
if err != nil {
|
||||
<-sem // release the slot we reserved but won't use
|
||||
if errors.Is(err, ErrNoArtifactJob) {
|
||||
break
|
||||
}
|
||||
wg.Wait()
|
||||
return err // includes context cancellation (pgx honors ctx)
|
||||
}
|
||||
wg.Add(1)
|
||||
go func(a *Artifact) {
|
||||
defer wg.Done()
|
||||
defer func() { <-sem }()
|
||||
m.encodeOne(ctx, a)
|
||||
}(job)
|
||||
}
|
||||
wg.Wait()
|
||||
return nil
|
||||
}
|
||||
|
||||
// encodeOne runs one claimed job to completion, extending its lease via a
|
||||
// heartbeat, and links/notifies the dependent download rows on the outcome.
|
||||
func (m *ArtifactManager) encodeOne(ctx context.Context, a *Artifact) {
|
||||
hbCtx, cancelHB := context.WithCancel(ctx)
|
||||
defer cancelHB()
|
||||
// heartbeatLoop cancels hbCtx if the lease is lost; PrepareFile runs on hbCtx
|
||||
// so that cancellation aborts ffmpeg, ensuring we never keep writing the
|
||||
// output path after another worker has taken the job.
|
||||
go m.heartbeatLoop(hbCtx, cancelHB, a.ID)
|
||||
|
||||
file, err := m.fileRepo.GetByID(ctx, a.MediaFileID)
|
||||
if err != nil || file == nil {
|
||||
m.failJob(ctx, a, "source media file unavailable")
|
||||
return
|
||||
}
|
||||
|
||||
opts := m.buildOpts(file, a)
|
||||
if err := m.preparer.PrepareFile(hbCtx, opts, a.OutputPath); err != nil {
|
||||
switch {
|
||||
case ctx.Err() != nil:
|
||||
// Parent shutting down: leave the job 'running'; its lease expires and
|
||||
// recovery (here or on another node) reclaims it.
|
||||
return
|
||||
case hbCtx.Err() != nil:
|
||||
// We lost the lease mid-encode; another worker now owns the job.
|
||||
slog.Warn("download artifact encode aborted; lease lost", "artifact_id", a.ID)
|
||||
return
|
||||
default:
|
||||
slog.Warn("download artifact encode failed", "artifact_id", a.ID, "error", err)
|
||||
m.failJob(ctx, a, err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
var size int64
|
||||
if fi, statErr := os.Stat(a.OutputPath); statErr == nil {
|
||||
size = fi.Size()
|
||||
}
|
||||
// Fenced on lease ownership: if we lost the lease between encode and commit,
|
||||
// applied is false and the current owner is responsible for flipping links —
|
||||
// do not flip them here or we would race/duplicate that owner's work.
|
||||
applied, err := m.repo.MarkReady(ctx, a.ID, m.owner, a.OutputPath, size)
|
||||
if err != nil {
|
||||
slog.Error("marking artifact ready failed", "artifact_id", a.ID, "error", err)
|
||||
return
|
||||
}
|
||||
if !applied {
|
||||
slog.Warn("download artifact ready skipped; lease lost", "artifact_id", a.ID)
|
||||
return
|
||||
}
|
||||
flipped, err := m.downloads.MarkLinkedDownloadsReady(ctx, a.ID, size)
|
||||
if err != nil {
|
||||
slog.Error("flipping linked downloads ready failed", "artifact_id", a.ID, "error", err)
|
||||
return
|
||||
}
|
||||
for _, d := range flipped {
|
||||
m.publish(ctx, d)
|
||||
}
|
||||
}
|
||||
|
||||
func (m *ArtifactManager) failJob(ctx context.Context, a *Artifact, msg string) {
|
||||
terminal, applied, err := m.repo.MarkFailedOrRetry(ctx, a.ID, m.owner, msg, backoffFor(a.Attempts))
|
||||
if err != nil {
|
||||
slog.Error("marking artifact failed/retry errored", "artifact_id", a.ID, "error", err)
|
||||
return
|
||||
}
|
||||
if !applied {
|
||||
// Lease lost; the current owner is responsible for the job's outcome.
|
||||
return
|
||||
}
|
||||
if terminal {
|
||||
m.failLinkedDownloads(ctx, a.ID, msg)
|
||||
} else {
|
||||
m.triggerDrain()
|
||||
}
|
||||
}
|
||||
|
||||
func (m *ArtifactManager) failLinkedDownloads(ctx context.Context, artifactID, msg string) {
|
||||
flipped, err := m.downloads.MarkLinkedDownloadsFailed(ctx, artifactID, msg)
|
||||
if err != nil {
|
||||
slog.Error("flipping linked downloads failed errored", "artifact_id", artifactID, "error", err)
|
||||
return
|
||||
}
|
||||
for _, d := range flipped {
|
||||
m.publish(ctx, d)
|
||||
}
|
||||
}
|
||||
|
||||
func (m *ArtifactManager) publish(ctx context.Context, d *Download) {
|
||||
if m.notify != nil {
|
||||
m.notify(ctx, d)
|
||||
}
|
||||
}
|
||||
|
||||
// heartbeatLoop extends the job's lease until ctx is done. If the lease is lost
|
||||
// (another worker stole it, or the row is gone) it calls cancel to abort the
|
||||
// encode so two workers never write the same output path. A transient DB error
|
||||
// is retried on the next tick rather than aborting a healthy encode.
|
||||
func (m *ArtifactManager) heartbeatLoop(ctx context.Context, cancel context.CancelFunc, id string) {
|
||||
ticker := time.NewTicker(artifactHeartbeat)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
ok, err := m.repo.Heartbeat(ctx, id, m.owner, artifactLease)
|
||||
switch {
|
||||
case err != nil && ctx.Err() != nil:
|
||||
return // encode finished or shutting down
|
||||
case err != nil:
|
||||
slog.Warn("download artifact heartbeat errored", "artifact_id", id, "error", err)
|
||||
case !ok:
|
||||
slog.Warn("download artifact lease lost; aborting encode", "artifact_id", id)
|
||||
cancel()
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (m *ArtifactManager) buildOpts(file *models.MediaFile, a *Artifact) playback.TranscodeOpts {
|
||||
cfg := config.Config{}
|
||||
if m.liveCfg != nil {
|
||||
if c := m.liveCfg(); c != nil {
|
||||
cfg = *c
|
||||
}
|
||||
}
|
||||
return playback.TranscodeOpts{
|
||||
InputPath: file.FilePath,
|
||||
SourceVideoCodec: file.CodecVideo,
|
||||
TargetCodecVideo: a.CodecVideo,
|
||||
TargetCodecAudio: a.CodecAudio,
|
||||
TargetResolution: a.Resolution,
|
||||
TargetBitrateKbps: a.TargetBitrateKbps,
|
||||
AudioTrackIndex: a.AudioTrackIndex,
|
||||
SubtitleTrackIndex: -1,
|
||||
FFmpegPath: cfg.Playback.FFmpegPath,
|
||||
HWAccel: cfg.Playback.HWAccel,
|
||||
HWDevice: cfg.Playback.HWDevice,
|
||||
TotalDuration: float64(file.Duration),
|
||||
}
|
||||
}
|
||||
|
||||
// Hygiene retention windows. These remove only rows nothing can serve again —
|
||||
// terminally-failed jobs (linked downloads already flipped to failed by
|
||||
// reconciliation) and ready artifacts whose every referencing download row was
|
||||
// deleted — plus ephemeral web rows past their convenience-record lifetime.
|
||||
// The server-disk *quota* is download.artifact_max_bytes (see the download
|
||||
// limits & restrictions design); this sweep is not a quota.
|
||||
const (
|
||||
failedArtifactRetention = 24 * time.Hour
|
||||
unlinkedArtifactRetention = 30 * 24 * time.Hour
|
||||
ephemeralDownloadRetention = 7 * 24 * time.Hour
|
||||
)
|
||||
|
||||
// Cleanup runs the hygiene sweep, then evicts ready artifacts (LRU first) once
|
||||
// the total exceeds the byte budget, never removing one still linked by any
|
||||
// active download row (managed or ephemeral) — only artifacts whose links are
|
||||
// all terminal are evictable.
|
||||
func (m *ArtifactManager) Cleanup(ctx context.Context) error {
|
||||
m.sweepStale(ctx)
|
||||
budget := m.downloadConfig().ArtifactMaxBytes
|
||||
if budget <= 0 {
|
||||
return nil // unlimited
|
||||
}
|
||||
total, err := m.repo.TotalReadyBytes(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if total <= budget {
|
||||
return nil
|
||||
}
|
||||
candidates, err := m.repo.ListReady(ctx) // least-recently-used first
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, a := range candidates {
|
||||
if total <= budget {
|
||||
break
|
||||
}
|
||||
active, err := m.repo.HasActiveLink(ctx, a.ID)
|
||||
if err != nil {
|
||||
slog.Warn("artifact link check failed", "artifact_id", a.ID, "error", err)
|
||||
continue
|
||||
}
|
||||
if active {
|
||||
continue
|
||||
}
|
||||
if a.OutputPath != "" {
|
||||
if err := os.Remove(a.OutputPath); err != nil && !os.IsNotExist(err) {
|
||||
slog.Warn("removing evicted artifact file failed", "artifact_id", a.ID, "error", err)
|
||||
}
|
||||
}
|
||||
if err := m.repo.DeleteArtifact(ctx, a.ID); err != nil {
|
||||
slog.Warn("deleting evicted artifact row failed", "artifact_id", a.ID, "error", err)
|
||||
continue
|
||||
}
|
||||
slog.Info("evicted download artifact (LRU)", "artifact_id", a.ID, "bytes", a.FileSize)
|
||||
total -= a.FileSize
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// sweepStale is the age-based hygiene pass: cold terminally-failed artifacts
|
||||
// (with their leftover .part files), orphaned ready artifacts no download row
|
||||
// references, and expired ephemeral download rows. Best-effort; every step
|
||||
// logs and continues.
|
||||
func (m *ArtifactManager) sweepStale(ctx context.Context) {
|
||||
if !m.maintenanceDue(&m.lastStaleSweep) {
|
||||
return
|
||||
}
|
||||
now := time.Now()
|
||||
if failed, err := m.repo.ListFailedBefore(ctx, now.Add(-failedArtifactRetention)); err != nil {
|
||||
slog.Warn("failed-artifact sweep list failed", "error", err)
|
||||
} else {
|
||||
for _, a := range failed {
|
||||
m.removeArtifact(ctx, a, "failed")
|
||||
}
|
||||
}
|
||||
if orphans, err := m.repo.ListUnlinkedReadyBefore(ctx, now.Add(-unlinkedArtifactRetention)); err != nil {
|
||||
slog.Warn("unlinked-artifact sweep list failed", "error", err)
|
||||
} else {
|
||||
for _, a := range orphans {
|
||||
m.removeArtifact(ctx, a, "unlinked")
|
||||
}
|
||||
}
|
||||
if m.downloads != nil {
|
||||
if n, err := m.downloads.PruneEphemeralOlderThan(ctx, now.Add(-ephemeralDownloadRetention)); err != nil {
|
||||
slog.Warn("ephemeral download prune failed", "error", err)
|
||||
} else if n > 0 {
|
||||
slog.Info("pruned expired ephemeral downloads", "rows", n)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// removeArtifact deletes an artifact's output file, its .part leftover, and
|
||||
// its row. Used by the hygiene sweep for rows nothing can serve again.
|
||||
func (m *ArtifactManager) removeArtifact(ctx context.Context, a *Artifact, reason string) {
|
||||
if a.OutputPath != "" {
|
||||
if err := os.Remove(a.OutputPath); err != nil && !os.IsNotExist(err) {
|
||||
slog.Warn("removing swept artifact file failed", "artifact_id", a.ID, "error", err)
|
||||
}
|
||||
if err := os.Remove(a.OutputPath + ".part"); err != nil && !os.IsNotExist(err) {
|
||||
slog.Warn("removing swept artifact partial failed", "artifact_id", a.ID, "error", err)
|
||||
}
|
||||
}
|
||||
if err := m.repo.DeleteArtifact(ctx, a.ID); err != nil {
|
||||
slog.Warn("deleting swept artifact row failed", "artifact_id", a.ID, "error", err)
|
||||
return
|
||||
}
|
||||
slog.Info("swept stale download artifact", "artifact_id", a.ID, "reason", reason, "bytes", a.FileSize)
|
||||
}
|
||||
|
||||
// backoffFor returns the retry delay for the next attempt after a failure.
|
||||
func backoffFor(attempts int) time.Duration {
|
||||
if attempts < 1 {
|
||||
attempts = 1
|
||||
}
|
||||
d := time.Duration(attempts) * 30 * time.Second
|
||||
if d > 5*time.Minute {
|
||||
d = 5 * time.Minute
|
||||
}
|
||||
return d
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package download
|
||||
package downloads
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -65,12 +65,14 @@ func (bm *BandwidthManager) getUserLimiter(userID int) *rate.Limiter {
|
||||
return nil
|
||||
}
|
||||
if v, ok := bm.userLimiters.Load(userID); ok {
|
||||
return v.(*rate.Limiter)
|
||||
lim, _ := v.(*rate.Limiter)
|
||||
return lim
|
||||
}
|
||||
burst := max(int(ubps/4), 32*1024)
|
||||
limiter := rate.NewLimiter(rate.Limit(ubps), burst)
|
||||
actual, _ := bm.userLimiters.LoadOrStore(userID, limiter)
|
||||
return actual.(*rate.Limiter)
|
||||
lim, _ := actual.(*rate.Limiter)
|
||||
return lim
|
||||
}
|
||||
|
||||
// ThrottledReader wraps an io.ReadSeeker with bandwidth limiting for the given user.
|
||||
@@ -1,4 +1,4 @@
|
||||
package download
|
||||
package downloads
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -0,0 +1,490 @@
|
||||
package downloads
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
"github.com/Silo-Server/silo-server/internal/subtitles"
|
||||
)
|
||||
|
||||
// manifestVersion is bumped whenever the OfflineManifest DTO shape changes.
|
||||
const manifestVersion = 2
|
||||
|
||||
const apiDownloadsPrefix = "/api/v1/downloads/"
|
||||
|
||||
// ManifestSource assembles catalog detail for a content id. GetItemDetail
|
||||
// enforces per-profile content/library access via its filter, which doubles as
|
||||
// the manifest/artwork access re-check.
|
||||
type ManifestSource interface {
|
||||
GetItemDetail(ctx context.Context, contentID string, filter catalog.AccessFilter) (*catalog.ItemDetail, error)
|
||||
}
|
||||
|
||||
// SubtitleSource enumerates and fetches downloaded (S3) subtitle assets.
|
||||
type SubtitleSource interface {
|
||||
ListDownloadedSubtitles(ctx context.Context, mediaFileID int) ([]subtitles.DownloadedSubtitle, error)
|
||||
GetSubtitleContent(ctx context.Context, id int) (*subtitles.DownloadedSubtitle, []byte, error)
|
||||
}
|
||||
|
||||
// Marker is a time range (intro/credits/recap/preview) in seconds.
|
||||
type Marker struct {
|
||||
Start float64 `json:"start"`
|
||||
End float64 `json:"end"`
|
||||
}
|
||||
|
||||
// OfflineChapter is a chapter with only stable references (no presigned URL).
|
||||
type OfflineChapter struct {
|
||||
Index int `json:"index"`
|
||||
Title string `json:"title,omitempty"`
|
||||
StartSeconds float64 `json:"start_seconds"`
|
||||
EndSeconds float64 `json:"end_seconds"`
|
||||
ThumbnailThumbhash string `json:"thumbnail_thumbhash,omitempty"`
|
||||
}
|
||||
|
||||
// OfflineSubtitle is one downloadable subtitle asset. FetchURL is an
|
||||
// authenticated proxy endpoint, never a presigned URL.
|
||||
type OfflineSubtitle struct {
|
||||
Language string `json:"language"`
|
||||
Format string `json:"format"`
|
||||
Forced bool `json:"forced"`
|
||||
HearingImpaired bool `json:"hearing_impaired"`
|
||||
External bool `json:"external"`
|
||||
FetchURL string `json:"fetch_url"`
|
||||
FileSize int64 `json:"file_size,omitempty"`
|
||||
}
|
||||
|
||||
// OfflineAudioTrack describes audio streams the client may expose offline.
|
||||
type OfflineAudioTrack struct {
|
||||
Index int `json:"index"`
|
||||
Title string `json:"title,omitempty"`
|
||||
Language string `json:"language,omitempty"`
|
||||
Codec string `json:"codec,omitempty"`
|
||||
Layout string `json:"layout,omitempty"`
|
||||
Channels int `json:"channels,omitempty"`
|
||||
Bitrate int `json:"bitrate,omitempty"`
|
||||
SampleRate int `json:"sample_rate,omitempty"`
|
||||
Default bool `json:"default"`
|
||||
}
|
||||
|
||||
// OfflineIdentity mirrors userstore.WatchIdentity so a client can re-resolve
|
||||
// content_id after a server-side rescan.
|
||||
type OfflineIdentity struct {
|
||||
StableType string `json:"stable_type,omitempty"`
|
||||
ProviderIDs map[string]string `json:"provider_ids,omitempty"`
|
||||
SeriesProviderIDs map[string]string `json:"series_provider_ids,omitempty"`
|
||||
Season *int `json:"season,omitempty"`
|
||||
Episode *int `json:"episode,omitempty"`
|
||||
}
|
||||
|
||||
// OfflineIntegrity gives clients stable metadata for local file validation.
|
||||
type OfflineIntegrity struct {
|
||||
ExpectedBytes int64 `json:"expected_bytes"`
|
||||
MediaFileHash string `json:"media_file_hash,omitempty"`
|
||||
MetadataETag string `json:"metadata_etag"`
|
||||
}
|
||||
|
||||
// OfflineManifest is the stable, presigned-URL-free bundle a client stores to
|
||||
// play a managed download fully offline.
|
||||
type OfflineManifest struct {
|
||||
DownloadID string `json:"download_id"`
|
||||
ContentID string `json:"content_id"`
|
||||
EpisodeID string `json:"episode_id,omitempty"`
|
||||
Type string `json:"type"`
|
||||
Revision int `json:"revision"`
|
||||
Quality string `json:"quality"`
|
||||
EffectiveQuality string `json:"effective_quality"`
|
||||
DeliveryFormat string `json:"delivery_format"`
|
||||
TargetBitrateKbps int `json:"target_bitrate_kbps"`
|
||||
MediaFileID int `json:"media_file_id"`
|
||||
FileSize int64 `json:"file_size"`
|
||||
|
||||
Title string `json:"title"`
|
||||
Year int `json:"year,omitempty"`
|
||||
Overview string `json:"overview,omitempty"`
|
||||
Runtime int `json:"runtime,omitempty"`
|
||||
ContentRating string `json:"content_rating,omitempty"`
|
||||
Genres []string `json:"genres,omitempty"`
|
||||
SeriesID string `json:"series_id,omitempty"`
|
||||
SeriesTitle string `json:"series_title,omitempty"`
|
||||
SeasonNumber *int `json:"season_number,omitempty"`
|
||||
EpisodeNumber *int `json:"episode_number,omitempty"`
|
||||
|
||||
// Artwork: stable thumbhashes inline + authenticated proxy URLs (never
|
||||
// presigned S3 URLs). The client downloads the proxy URLs once.
|
||||
PosterThumbhash string `json:"poster_thumbhash,omitempty"`
|
||||
BackdropThumbhash string `json:"backdrop_thumbhash,omitempty"`
|
||||
ArtworkURLs struct {
|
||||
Poster string `json:"poster,omitempty"`
|
||||
Backdrop string `json:"backdrop,omitempty"`
|
||||
Logo string `json:"logo,omitempty"`
|
||||
} `json:"artwork_urls"`
|
||||
|
||||
Container string `json:"container"`
|
||||
CodecVideo string `json:"codec_video"`
|
||||
CodecAudio string `json:"codec_audio"`
|
||||
Resolution string `json:"resolution"`
|
||||
HDR bool `json:"hdr"`
|
||||
Duration int `json:"duration_seconds"`
|
||||
SelectedAudioTrackIndex *int `json:"selected_audio_track_index,omitempty"`
|
||||
AudioTracks []OfflineAudioTrack `json:"audio_tracks,omitempty"`
|
||||
|
||||
Chapters []OfflineChapter `json:"chapters,omitempty"`
|
||||
Intro *Marker `json:"intro,omitempty"`
|
||||
Credits *Marker `json:"credits,omitempty"`
|
||||
Recap *Marker `json:"recap,omitempty"`
|
||||
Preview *Marker `json:"preview,omitempty"`
|
||||
|
||||
Subtitles []OfflineSubtitle `json:"subtitles"`
|
||||
|
||||
StableIdentity OfflineIdentity `json:"stable_identity"`
|
||||
Integrity OfflineIntegrity `json:"integrity"`
|
||||
|
||||
ManifestVersion int `json:"manifest_version"`
|
||||
GeneratedAt string `json:"generated_at"`
|
||||
}
|
||||
|
||||
// ManifestBuilder assembles an OfflineManifest from the catalog detail path and
|
||||
// the download's subtitle assets, stripping every presigned URL.
|
||||
type ManifestBuilder struct {
|
||||
detail ManifestSource
|
||||
subs SubtitleSource
|
||||
fileRepo FileResolver
|
||||
// artifact resolves a download's linked prepared artifact so artifact-backed
|
||||
// manifests can describe the delivered file instead of the catalog source.
|
||||
artifact func(ctx context.Context, id string) (*Artifact, error)
|
||||
}
|
||||
|
||||
// NewManifestBuilder constructs a ManifestBuilder. artifact may be nil when no
|
||||
// prepare-to-file pipeline exists (only original downloads are servable then).
|
||||
func NewManifestBuilder(detail ManifestSource, subs SubtitleSource, fileRepo FileResolver, artifact func(ctx context.Context, id string) (*Artifact, error)) *ManifestBuilder {
|
||||
return &ManifestBuilder{detail: detail, subs: subs, fileRepo: fileRepo, artifact: artifact}
|
||||
}
|
||||
|
||||
// Build assembles the manifest for a managed entry. The filter enforces the
|
||||
// requesting profile's content access (GetItemDetail returns
|
||||
// catalog.ErrItemNotFound when denied).
|
||||
func (b *ManifestBuilder) Build(ctx context.Context, dl *Download, filter catalog.AccessFilter) (*OfflineManifest, error) {
|
||||
return b.build(ctx, dl, filter, nil)
|
||||
}
|
||||
|
||||
// build is Build with an optional per-batch series-detail cache: a season
|
||||
// batch shares one series, so the batch endpoint resolves its detail once
|
||||
// instead of once per episode.
|
||||
func (b *ManifestBuilder) build(ctx context.Context, dl *Download, filter catalog.AccessFilter, seriesCache map[string]*catalog.ItemDetail) (*OfflineManifest, error) {
|
||||
detail, err := b.detail.GetItemDetail(ctx, manifestContentID(dl), filter)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
file := b.lookupFile(ctx, dl.MediaFileID)
|
||||
var seriesDetail *catalog.ItemDetail
|
||||
if dl.EpisodeID != "" && detail.SeriesID != "" {
|
||||
if cached, ok := seriesCache[detail.SeriesID]; ok {
|
||||
seriesDetail = cached
|
||||
} else if sd, err := b.detail.GetItemDetail(ctx, detail.SeriesID, filter); err == nil {
|
||||
seriesDetail = sd
|
||||
if seriesCache != nil {
|
||||
seriesCache[detail.SeriesID] = sd
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
m := &OfflineManifest{
|
||||
DownloadID: dl.ID,
|
||||
ContentID: dl.ContentID,
|
||||
EpisodeID: dl.EpisodeID,
|
||||
Type: detail.Type,
|
||||
Revision: dl.Revision,
|
||||
Quality: dl.Quality,
|
||||
EffectiveQuality: dl.EffectiveQuality,
|
||||
DeliveryFormat: dl.Format,
|
||||
TargetBitrateKbps: dl.TargetBitrateKbps,
|
||||
MediaFileID: dl.MediaFileID,
|
||||
FileSize: dl.FileSize,
|
||||
Title: detail.Title,
|
||||
Year: detail.Year,
|
||||
Overview: detail.Overview,
|
||||
Runtime: detail.Runtime,
|
||||
ContentRating: detail.ContentRating,
|
||||
Genres: detail.Genres,
|
||||
SeriesID: detail.SeriesID,
|
||||
SeriesTitle: detail.SeriesTitle,
|
||||
SeasonNumber: detail.SeasonNumber,
|
||||
EpisodeNumber: detail.EpisodeNumber,
|
||||
PosterThumbhash: detail.PosterThumbhash,
|
||||
BackdropThumbhash: detail.BackdropThumbhash,
|
||||
Intro: toMarker(detail.Intro),
|
||||
Credits: toMarker(detail.Credits),
|
||||
Recap: toMarker(detail.Recap),
|
||||
Preview: toMarker(detail.Preview),
|
||||
StableIdentity: stableIdentity(dl, detail, seriesDetail),
|
||||
Integrity: buildIntegrity(dl, file),
|
||||
ManifestVersion: manifestVersion,
|
||||
GeneratedAt: time.Now().UTC().Format(time.RFC3339),
|
||||
}
|
||||
|
||||
// Artwork: emit a proxy URL only when the source actually has the image.
|
||||
if detail.PosterURL != "" {
|
||||
m.ArtworkURLs.Poster = artworkProxyURL(dl.ID, "poster")
|
||||
}
|
||||
if detail.BackdropURL != "" {
|
||||
m.ArtworkURLs.Backdrop = artworkProxyURL(dl.ID, "backdrop")
|
||||
}
|
||||
if detail.LogoURL != "" {
|
||||
m.ArtworkURLs.Logo = artworkProxyURL(dl.ID, "logo")
|
||||
}
|
||||
|
||||
if v := pickVersion(detail, dl.MediaFileID); v != nil {
|
||||
m.Container = v.Container
|
||||
m.CodecVideo = v.CodecVideo
|
||||
m.CodecAudio = v.CodecAudio
|
||||
m.Resolution = v.Resolution
|
||||
m.HDR = v.HDR
|
||||
m.Duration = v.Duration
|
||||
m.SelectedAudioTrackIndex = v.EffectiveAudioTrackIndex
|
||||
m.AudioTracks = toOfflineAudioTracks(v.AudioTracks)
|
||||
m.Chapters = toOfflineChapters(v.Chapters)
|
||||
}
|
||||
|
||||
// Remux/transcode entries deliver the prepared artifact, not the catalog
|
||||
// source: describe that file so the client picks the right decoder/tracks.
|
||||
if dl.Format != FormatOriginal && dl.ArtifactID != "" && b.artifact != nil {
|
||||
if a, err := b.artifact(ctx, dl.ArtifactID); err == nil && a != nil {
|
||||
applyArtifactParams(m, a)
|
||||
}
|
||||
}
|
||||
|
||||
m.Subtitles = b.buildSubtitles(ctx, dl, file)
|
||||
return m, nil
|
||||
}
|
||||
|
||||
// applyArtifactParams overwrites the source file's media parameters with the
|
||||
// prepared artifact's target parameters. "copy" targets keep the source value.
|
||||
func applyArtifactParams(m *OfflineManifest, a *Artifact) {
|
||||
if a.Container != "" {
|
||||
m.Container = a.Container
|
||||
}
|
||||
if a.CodecVideo != "" && a.CodecVideo != "copy" {
|
||||
m.CodecVideo = a.CodecVideo
|
||||
}
|
||||
if a.CodecAudio != "" && a.CodecAudio != "copy" {
|
||||
m.CodecAudio = a.CodecAudio
|
||||
}
|
||||
if a.Resolution != "" {
|
||||
m.Resolution = a.Resolution
|
||||
}
|
||||
// A prepared file contains exactly one audio stream — the track the encode
|
||||
// selected (playback.PrepareFile maps a single audio track).
|
||||
if len(m.AudioTracks) > 0 {
|
||||
idx := a.AudioTrackIndex
|
||||
if idx < 0 || idx >= len(m.AudioTracks) {
|
||||
idx = 0
|
||||
}
|
||||
track := m.AudioTracks[idx]
|
||||
track.Index = 0
|
||||
track.Default = true
|
||||
if a.CodecAudio != "" && a.CodecAudio != "copy" {
|
||||
track.Codec = a.CodecAudio
|
||||
}
|
||||
m.AudioTracks = []OfflineAudioTrack{track}
|
||||
selected := 0
|
||||
m.SelectedAudioTrackIndex = &selected
|
||||
}
|
||||
}
|
||||
|
||||
// buildSubtitles enumerates external (sidecar) + downloaded (S3) subtitle assets
|
||||
// for the download's media file (already loaded by build — no re-fetch).
|
||||
// Embedded tracks live inside the downloaded video file and need no separate
|
||||
// fetch.
|
||||
func (b *ManifestBuilder) buildSubtitles(ctx context.Context, dl *Download, file *models.MediaFile) []OfflineSubtitle {
|
||||
out := []OfflineSubtitle{}
|
||||
|
||||
if file != nil {
|
||||
for i, ext := range file.ExternalSubtitles {
|
||||
var size int64
|
||||
if info, statErr := os.Stat(ext.Path); statErr == nil {
|
||||
size = info.Size()
|
||||
}
|
||||
out = append(out, OfflineSubtitle{
|
||||
Language: ext.Language,
|
||||
Format: ext.Format,
|
||||
Forced: ext.Forced,
|
||||
HearingImpaired: ext.HearingImpaired,
|
||||
External: true,
|
||||
FetchURL: subtitleProxyURL(dl.ID, fmt.Sprintf("external:%d", i)),
|
||||
FileSize: size,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
if b.subs != nil {
|
||||
if downloaded, err := b.subs.ListDownloadedSubtitles(ctx, dl.MediaFileID); err == nil {
|
||||
for _, sub := range downloaded {
|
||||
out = append(out, OfflineSubtitle{
|
||||
Language: sub.Language,
|
||||
Format: string(sub.Format),
|
||||
HearingImpaired: sub.HearingImpaired,
|
||||
External: false,
|
||||
FetchURL: subtitleProxyURL(dl.ID, fmt.Sprintf("downloaded:%d", sub.ID)),
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
func (b *ManifestBuilder) lookupFile(ctx context.Context, mediaFileID int) *models.MediaFile {
|
||||
if b.fileRepo == nil || mediaFileID <= 0 {
|
||||
return nil
|
||||
}
|
||||
file, err := b.fileRepo.GetByID(ctx, mediaFileID)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return file
|
||||
}
|
||||
|
||||
// manifestContentID resolves the item the manifest describes: the episode's own
|
||||
// content id for episode entries, otherwise the movie's content id.
|
||||
func manifestContentID(dl *Download) string {
|
||||
if dl.EpisodeID != "" {
|
||||
return dl.EpisodeID
|
||||
}
|
||||
return dl.ContentID
|
||||
}
|
||||
|
||||
func artworkProxyURL(downloadID, kind string) string {
|
||||
return apiDownloadsPrefix + downloadID + "/artwork/" + kind
|
||||
}
|
||||
|
||||
func subtitleProxyURL(downloadID, ref string) string {
|
||||
return apiDownloadsPrefix + downloadID + "/subtitles/" + ref
|
||||
}
|
||||
|
||||
func pickVersion(detail *catalog.ItemDetail, mediaFileID int) *catalog.FileVersion {
|
||||
for i := range detail.Versions {
|
||||
if detail.Versions[i].FileID == mediaFileID {
|
||||
return &detail.Versions[i]
|
||||
}
|
||||
}
|
||||
if len(detail.Versions) > 0 {
|
||||
return &detail.Versions[0]
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func toMarker(m *catalog.Marker) *Marker {
|
||||
if m == nil {
|
||||
return nil
|
||||
}
|
||||
return &Marker{Start: m.Start, End: m.End}
|
||||
}
|
||||
|
||||
func toOfflineChapters(chapters []catalog.VersionChapter) []OfflineChapter {
|
||||
if len(chapters) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]OfflineChapter, 0, len(chapters))
|
||||
for _, c := range chapters {
|
||||
out = append(out, OfflineChapter{
|
||||
Index: c.Index,
|
||||
Title: c.Title,
|
||||
StartSeconds: c.StartSeconds,
|
||||
EndSeconds: c.EndSeconds,
|
||||
ThumbnailThumbhash: c.ThumbnailThumbhash,
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func toOfflineAudioTracks(tracks []models.AudioTrack) []OfflineAudioTrack {
|
||||
if len(tracks) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]OfflineAudioTrack, 0, len(tracks))
|
||||
for i, t := range tracks {
|
||||
out = append(out, OfflineAudioTrack{
|
||||
Index: i,
|
||||
Title: firstNonEmpty(t.Title, t.EmbeddedTitle),
|
||||
Language: t.Language,
|
||||
Codec: t.Codec,
|
||||
Layout: t.Layout,
|
||||
Channels: t.Channels,
|
||||
Bitrate: t.Bitrate,
|
||||
SampleRate: t.SampleRate,
|
||||
Default: t.Default,
|
||||
})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func stableIdentity(dl *Download, detail, seriesDetail *catalog.ItemDetail) OfflineIdentity {
|
||||
providerIDs := map[string]string{}
|
||||
addProviderIDs(providerIDs, detail)
|
||||
id := OfflineIdentity{
|
||||
StableType: detail.Type,
|
||||
ProviderIDs: providerIDs,
|
||||
Season: detail.SeasonNumber,
|
||||
Episode: detail.EpisodeNumber,
|
||||
}
|
||||
if dl.EpisodeID != "" {
|
||||
id.StableType = "episode"
|
||||
seriesProviderIDs := map[string]string{}
|
||||
addProviderIDs(seriesProviderIDs, seriesDetail)
|
||||
if len(seriesProviderIDs) > 0 {
|
||||
id.SeriesProviderIDs = seriesProviderIDs
|
||||
}
|
||||
}
|
||||
if len(providerIDs) == 0 {
|
||||
id.ProviderIDs = nil
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
func addProviderIDs(out map[string]string, detail *catalog.ItemDetail) {
|
||||
if detail == nil {
|
||||
return
|
||||
}
|
||||
if detail.ImdbID != "" {
|
||||
out["imdb"] = detail.ImdbID
|
||||
}
|
||||
if detail.TmdbID != "" {
|
||||
out["tmdb"] = detail.TmdbID
|
||||
}
|
||||
if detail.TvdbID != "" {
|
||||
out["tvdb"] = detail.TvdbID
|
||||
}
|
||||
}
|
||||
|
||||
func buildIntegrity(dl *Download, file *models.MediaFile) OfflineIntegrity {
|
||||
hash := ""
|
||||
modified := ""
|
||||
if file != nil {
|
||||
hash = file.FileHash
|
||||
if file.FileModifiedAt != nil {
|
||||
modified = file.FileModifiedAt.UTC().Format(time.RFC3339Nano)
|
||||
}
|
||||
}
|
||||
sum := sha256.Sum256([]byte(fmt.Sprintf("%s|%d|%s|%s|%s|%d|%d|%s",
|
||||
dl.ID, dl.Revision, dl.Format, dl.Quality,
|
||||
dl.EffectiveQuality, dl.TargetBitrateKbps, dl.FileSize, modified)))
|
||||
return OfflineIntegrity{
|
||||
ExpectedBytes: dl.FileSize,
|
||||
MediaFileHash: hash,
|
||||
MetadataETag: hex.EncodeToString(sum[:]),
|
||||
}
|
||||
}
|
||||
|
||||
func firstNonEmpty(values ...string) string {
|
||||
for _, v := range values {
|
||||
if v != "" {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
@@ -0,0 +1,192 @@
|
||||
package downloads
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
"github.com/Silo-Server/silo-server/internal/subtitles"
|
||||
)
|
||||
|
||||
type fakeManifestSource struct {
|
||||
detail *catalog.ItemDetail
|
||||
err error
|
||||
}
|
||||
|
||||
func (f fakeManifestSource) GetItemDetail(context.Context, string, catalog.AccessFilter) (*catalog.ItemDetail, error) {
|
||||
if f.err != nil {
|
||||
return nil, f.err
|
||||
}
|
||||
return f.detail, nil
|
||||
}
|
||||
|
||||
type fakeSubtitleSource struct {
|
||||
downloaded []subtitles.DownloadedSubtitle
|
||||
}
|
||||
|
||||
func (f fakeSubtitleSource) ListDownloadedSubtitles(context.Context, int) ([]subtitles.DownloadedSubtitle, error) {
|
||||
return f.downloaded, nil
|
||||
}
|
||||
|
||||
func (f fakeSubtitleSource) GetSubtitleContent(context.Context, int) (*subtitles.DownloadedSubtitle, []byte, error) {
|
||||
return nil, nil, ErrAssetNotFound
|
||||
}
|
||||
|
||||
type fakeFileResolver struct {
|
||||
file *models.MediaFile
|
||||
}
|
||||
|
||||
func (f fakeFileResolver) GetByID(context.Context, int) (*models.MediaFile, error) {
|
||||
return f.file, nil
|
||||
}
|
||||
func (f fakeFileResolver) GetByContentID(context.Context, string) ([]*models.MediaFile, error) {
|
||||
return nil, nil
|
||||
}
|
||||
func (f fakeFileResolver) GetByEpisodeID(context.Context, string) ([]*models.MediaFile, error) {
|
||||
return nil, nil
|
||||
}
|
||||
func (f fakeFileResolver) ListByEpisodeIDs(context.Context, []string) (map[string][]*models.MediaFile, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// TestManifestBuilderDeniesRestrictedProfile is the Phase 2 acceptance criterion
|
||||
// at the source: when the requesting profile is denied content access,
|
||||
// GetItemDetail returns ErrItemNotFound and Build propagates it.
|
||||
func TestManifestBuilderDeniesRestrictedProfile(t *testing.T) {
|
||||
b := NewManifestBuilder(fakeManifestSource{err: catalog.ErrItemNotFound}, nil, nil, nil)
|
||||
_, err := b.Build(context.Background(), &Download{ID: "dl1", ContentID: "c1"}, catalog.AccessFilter{})
|
||||
if !errors.Is(err, catalog.ErrItemNotFound) {
|
||||
t.Fatalf("Build err = %v, want catalog.ErrItemNotFound", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManifestBuilderAssembles(t *testing.T) {
|
||||
detail := &catalog.ItemDetail{
|
||||
Type: "movie",
|
||||
Title: "The Movie",
|
||||
Year: 2021,
|
||||
Overview: "A film.",
|
||||
Runtime: 120,
|
||||
ContentRating: "PG-13",
|
||||
Genres: []string{"Drama"},
|
||||
PosterURL: "https://s3.example.com/poster.jpg?sig=SECRET",
|
||||
PosterThumbhash: "PHASH",
|
||||
BackdropURL: "https://s3.example.com/backdrop.jpg?sig=SECRET",
|
||||
BackdropThumbhash: "BHASH",
|
||||
ImdbID: "tt123",
|
||||
TmdbID: "456",
|
||||
Intro: &catalog.Marker{Start: 0, End: 60},
|
||||
Versions: []catalog.FileVersion{{
|
||||
FileID: 99,
|
||||
Container: "mkv",
|
||||
CodecVideo: "h264",
|
||||
CodecAudio: "aac",
|
||||
Resolution: "1080p",
|
||||
Duration: 7200,
|
||||
Chapters: []catalog.VersionChapter{{
|
||||
Index: 1, Title: "Opening", StartSeconds: 0, EndSeconds: 600,
|
||||
ThumbnailURL: "https://s3.example.com/chap.jpg?sig=SECRET", ThumbnailThumbhash: "CHASH",
|
||||
}},
|
||||
}},
|
||||
}
|
||||
file := &models.MediaFile{ExternalSubtitles: []models.ExternalSubtitle{
|
||||
{Path: "/media/sub.en.srt", Language: "en", Format: "srt", Forced: true},
|
||||
}}
|
||||
subs := fakeSubtitleSource{downloaded: []subtitles.DownloadedSubtitle{
|
||||
{ID: 7, MediaFileID: 99, Language: "fr", Format: subtitles.SubtitleFormat("vtt")},
|
||||
}}
|
||||
b := NewManifestBuilder(fakeManifestSource{detail: detail}, subs, fakeFileResolver{file: file}, nil)
|
||||
|
||||
dl := &Download{ID: "dl1", ContentID: "c1", MediaFileID: 99, FileSize: 1024, Format: FormatOriginal}
|
||||
m, err := b.Build(context.Background(), dl, catalog.AccessFilter{})
|
||||
if err != nil {
|
||||
t.Fatalf("Build: %v", err)
|
||||
}
|
||||
|
||||
if m.Title != "The Movie" || m.Year != 2021 || m.Runtime != 120 {
|
||||
t.Fatalf("metadata mismatch: %+v", m)
|
||||
}
|
||||
if m.PosterThumbhash != "PHASH" || m.BackdropThumbhash != "BHASH" {
|
||||
t.Fatalf("thumbhashes not inlined: %+v", m)
|
||||
}
|
||||
if m.ArtworkURLs.Poster != "/api/v1/downloads/dl1/artwork/poster" {
|
||||
t.Fatalf("poster url = %q, want proxy path", m.ArtworkURLs.Poster)
|
||||
}
|
||||
if m.ArtworkURLs.Backdrop != "/api/v1/downloads/dl1/artwork/backdrop" {
|
||||
t.Fatalf("backdrop url = %q, want proxy path", m.ArtworkURLs.Backdrop)
|
||||
}
|
||||
if m.ArtworkURLs.Logo != "" {
|
||||
t.Fatalf("logo url = %q, want empty (no LogoURL)", m.ArtworkURLs.Logo)
|
||||
}
|
||||
if m.Container != "mkv" || m.CodecVideo != "h264" || m.Resolution != "1080p" || m.Duration != 7200 {
|
||||
t.Fatalf("playback metadata mismatch: %+v", m)
|
||||
}
|
||||
if m.Intro == nil || m.Intro.End != 60 {
|
||||
t.Fatalf("intro marker = %+v", m.Intro)
|
||||
}
|
||||
if len(m.Chapters) != 1 || m.Chapters[0].ThumbnailThumbhash != "CHASH" {
|
||||
t.Fatalf("chapters = %+v", m.Chapters)
|
||||
}
|
||||
if len(m.Subtitles) != 2 {
|
||||
t.Fatalf("subtitles = %+v, want 2", m.Subtitles)
|
||||
}
|
||||
if m.Subtitles[0].FetchURL != "/api/v1/downloads/dl1/subtitles/external:0" || !m.Subtitles[0].External {
|
||||
t.Fatalf("external subtitle = %+v", m.Subtitles[0])
|
||||
}
|
||||
if m.Subtitles[1].FetchURL != "/api/v1/downloads/dl1/subtitles/downloaded:7" || m.Subtitles[1].External {
|
||||
t.Fatalf("downloaded subtitle = %+v", m.Subtitles[1])
|
||||
}
|
||||
if m.StableIdentity.ProviderIDs["imdb"] != "tt123" || m.StableIdentity.ProviderIDs["tmdb"] != "456" {
|
||||
t.Fatalf("stable identity = %+v", m.StableIdentity)
|
||||
}
|
||||
if m.ManifestVersion != manifestVersion {
|
||||
t.Fatalf("manifest version = %d, want %d", m.ManifestVersion, manifestVersion)
|
||||
}
|
||||
|
||||
// The whole point of the manifest: a stored copy must contain NO presigned
|
||||
// URL. Serialize and assert the upstream signature/host never leaks.
|
||||
encoded, err := json.Marshal(m)
|
||||
if err != nil {
|
||||
t.Fatalf("marshal: %v", err)
|
||||
}
|
||||
for _, leak := range [][]byte{[]byte("sig=SECRET"), []byte("s3.example.com")} {
|
||||
if bytes.Contains(encoded, leak) {
|
||||
t.Fatalf("manifest leaks presigned URL fragment %q: %s", leak, encoded)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseSubtitleRef(t *testing.T) {
|
||||
cases := []struct {
|
||||
ref string
|
||||
wantKind string
|
||||
wantValue int
|
||||
wantErr bool
|
||||
}{
|
||||
{"external:0", "external", 0, false},
|
||||
{"external:12", "external", 12, false},
|
||||
{"downloaded:7", "downloaded", 7, false},
|
||||
{"bogus", "", 0, true},
|
||||
{"external:x", "", 0, true},
|
||||
{"weird:1", "", 0, true},
|
||||
{"", "", 0, true},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.ref, func(t *testing.T) {
|
||||
kind, value, err := parseSubtitleRef(tc.ref)
|
||||
if tc.wantErr {
|
||||
if !errors.Is(err, ErrInvalidSubtitleRef) {
|
||||
t.Fatalf("parseSubtitleRef(%q) err = %v, want ErrInvalidSubtitleRef", tc.ref, err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil || kind != tc.wantKind || value != tc.wantValue {
|
||||
t.Fatalf("parseSubtitleRef(%q) = (%q, %d, %v)", tc.ref, kind, value, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
// Package downloads owns the downloads domain: a single device-aware,
|
||||
// quality-aware registry shared by the web app and mobile clients for offline
|
||||
// playback. It replaces the former internal/download package, absorbing its
|
||||
// bandwidth/quota/serving logic and reshaping the table and /downloads contract. See
|
||||
// docs/superpowers/specs/2026-06-18-offline-sync-mobile-design.md.
|
||||
package downloads
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Format constants are internal delivery formats recorded on a download row.
|
||||
// They are not user-facing quality presets: clients choose a Quality*, then the
|
||||
// service records the concrete delivery format it can fulfill.
|
||||
const (
|
||||
FormatOriginal = "original"
|
||||
FormatRemux = "remux"
|
||||
FormatTranscode = "transcode"
|
||||
)
|
||||
|
||||
// Quality constants are the public download choices clients may request.
|
||||
const (
|
||||
QualityOriginal = "original"
|
||||
Quality20Mbps = "20mbps"
|
||||
Quality10Mbps = "10mbps"
|
||||
Quality5Mbps = "5mbps"
|
||||
Quality2Mbps = "2mbps"
|
||||
Quality1Mbps = "1mbps"
|
||||
)
|
||||
|
||||
// QualityPresets is the ordered UI/API ladder for download quality.
|
||||
var QualityPresets = []string{
|
||||
QualityOriginal,
|
||||
Quality20Mbps,
|
||||
Quality10Mbps,
|
||||
Quality5Mbps,
|
||||
Quality2Mbps,
|
||||
Quality1Mbps,
|
||||
}
|
||||
|
||||
// Download status constants.
|
||||
const (
|
||||
// Ephemeral / web lifecycle (unchanged from downloads v1):
|
||||
// queued -> downloading -> completed | failed | canceled.
|
||||
StatusQueued = "queued"
|
||||
StatusDownloading = "downloading"
|
||||
StatusCompleted = "completed"
|
||||
StatusFailed = "failed"
|
||||
StatusCancelled = "cancelled" //nolint:misspell // persisted DB enum value (migration 042)
|
||||
|
||||
// Managed device-entry lifecycle (Phase 1+):
|
||||
// [preparing ->] ready -> completed, plus revoked (reserved for the
|
||||
// admin revoke flow; see the download limits & restrictions design).
|
||||
StatusPreparing = "preparing"
|
||||
StatusReady = "ready"
|
||||
StatusRevoked = "revoked"
|
||||
)
|
||||
|
||||
// Download kind constants.
|
||||
const (
|
||||
KindQueued = "queued"
|
||||
)
|
||||
|
||||
// Sentinel errors.
|
||||
var (
|
||||
ErrNotFound = errors.New("download not found")
|
||||
ErrDownloadNotAllowed = errors.New("user is not allowed to download")
|
||||
ErrFeatureDisabled = errors.New("downloads are disabled")
|
||||
ErrConcurrentLimitReached = errors.New("concurrent download limit reached")
|
||||
ErrPeriodLimitReached = errors.New("download period limit reached")
|
||||
ErrDownloadNotActive = errors.New("download is not in an active state")
|
||||
ErrStatusConflict = errors.New("download status transition conflict")
|
||||
ErrTranscodeDisabled = errors.New("download transcode is disabled")
|
||||
ErrInvalidQuality = errors.New("invalid download quality")
|
||||
ErrProfileRequired = errors.New("managed download requires a profile")
|
||||
ErrInvalidStatus = errors.New("invalid download status transition")
|
||||
ErrManifestUnavailable = errors.New("offline manifest is not available")
|
||||
ErrInvalidSubtitleRef = errors.New("invalid subtitle reference")
|
||||
ErrAssetNotFound = errors.New("download asset not found")
|
||||
ErrFormatUnavailable = errors.New("requested download format is not available")
|
||||
// ErrQualityUnavailable means the requested quality is valid and permitted in
|
||||
// principle but cannot be fulfilled by the current server wiring.
|
||||
ErrQualityUnavailable = errors.New("requested download quality is not available")
|
||||
// ErrBulkQualityUnavailable keeps season/series batches original-only until
|
||||
// batch artifact UX and storage reporting are explicit.
|
||||
ErrBulkQualityUnavailable = errors.New("bulk quality downloads are not available")
|
||||
// ErrNoDownloadableEpisodes means a series/season download matched a valid
|
||||
// series but none of its episodes have a downloadable file.
|
||||
ErrNoDownloadableEpisodes = errors.New("no downloadable episodes found")
|
||||
|
||||
// Series-monitoring (download subscription) errors.
|
||||
ErrSubscriptionNotFound = errors.New("download subscription not found")
|
||||
ErrSubscriptionsUnavailable = errors.New("download subscriptions are not available")
|
||||
ErrInvalidSubscriptionMode = errors.New("invalid subscription mode")
|
||||
ErrSeasonsRequired = errors.New("season_numbers is required for specific_seasons mode")
|
||||
ErrInvalidSeasonNumbers = errors.New("season_numbers contains an out-of-range season number")
|
||||
ErrNotSeries = errors.New("content_id is not a series")
|
||||
)
|
||||
|
||||
// Download represents a row in the downloads table. It carries both the
|
||||
// ephemeral/web lifecycle (DeviceID == "") and the managed device-entry
|
||||
// lifecycle (DeviceID set); ProfileID/DeviceID/ArtifactID map to nullable
|
||||
// columns and are empty strings when unset.
|
||||
type Download struct {
|
||||
ID string
|
||||
UserID int
|
||||
ProfileID string // "" for ephemeral/web rows
|
||||
DeviceID string // "" = ephemeral; set = managed device entry
|
||||
MediaFileID int
|
||||
ContentID string
|
||||
EpisodeID string
|
||||
BatchID string
|
||||
Kind string // direct or queued
|
||||
Status string // see Status* constants
|
||||
Format string // internal delivery format; see Format* constants
|
||||
Quality string // user-requested quality; see Quality* constants
|
||||
EffectiveQuality string // actual quality after original compatibility fallback
|
||||
TargetBitrateKbps int // 0 for original/remux; bitrate cap for transcode
|
||||
Revision int // increments when a managed row is replaced in place
|
||||
ArtifactID string // "" until a remux/transcode artifact is linked (Phase 3)
|
||||
FileSize int64
|
||||
BytesSent int64
|
||||
ErrorMessage string
|
||||
CreatedAt time.Time
|
||||
UpdatedAt time.Time
|
||||
CompletedAt *time.Time
|
||||
}
|
||||
|
||||
// SkippedDownload explains why a bulk series/season request did not create a
|
||||
// row for an episode.
|
||||
type SkippedDownload struct {
|
||||
EpisodeID string `json:"episode_id"`
|
||||
Reason string `json:"reason"`
|
||||
}
|
||||
|
||||
// IsManaged reports whether this is a managed device-library entry (DeviceID set)
|
||||
// rather than an ephemeral/account-level row.
|
||||
func (d *Download) IsManaged() bool {
|
||||
return d.DeviceID != ""
|
||||
}
|
||||
@@ -0,0 +1,246 @@
|
||||
package downloads
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/playback"
|
||||
"github.com/Silo-Server/silo-server/internal/subtitles"
|
||||
)
|
||||
|
||||
// authorizeManagedAsset applies invariant-2 row authorization (user, profile,
|
||||
// device) and the revoked guard to a managed entry before any asset is served.
|
||||
// The per-profile content-access re-check is performed by each caller (via
|
||||
// GetItemDetail or EnsureAccessible) before bytes leave the server.
|
||||
func (s *Service) authorizeManagedAsset(ctx context.Context, userID int, profileID, deviceID, downloadID string) (*Download, error) {
|
||||
if _, err := s.enabledConfig(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if profileID == "" || deviceID == "" {
|
||||
return nil, ErrProfileRequired
|
||||
}
|
||||
dl, err := s.repo.GetManagedByID(ctx, downloadID, userID, profileID, deviceID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if dl.Status == StatusRevoked {
|
||||
return nil, fmt.Errorf("download is revoked: %w", ErrDownloadNotActive)
|
||||
}
|
||||
return dl, nil
|
||||
}
|
||||
|
||||
// BuildManifest returns the offline manifest for a managed entry, authorized on
|
||||
// (user, profile, device) with a per-profile content-access re-check inside the
|
||||
// builder's GetItemDetail call.
|
||||
func (s *Service) BuildManifest(ctx context.Context, userID int, profileID, deviceID, downloadID string, filter catalog.AccessFilter) (*OfflineManifest, error) {
|
||||
dl, err := s.authorizeManagedAsset(ctx, userID, profileID, deviceID, downloadID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if s.manifest == nil {
|
||||
return nil, ErrManifestUnavailable
|
||||
}
|
||||
return s.manifest.Build(ctx, dl, filter)
|
||||
}
|
||||
|
||||
// SkippedManifest reports a batch entry whose manifest could not be built —
|
||||
// one bad episode (revoked, deleted from the catalog, access-filtered) must
|
||||
// not make the rest of a season's manifests unfetchable.
|
||||
type SkippedManifest struct {
|
||||
DownloadID string `json:"download_id"`
|
||||
Reason string `json:"reason"` // revoked | not_found | error
|
||||
}
|
||||
|
||||
// BuildBatchManifests returns the manifests for every managed entry in a batch
|
||||
// owned by the calling profile/device, plus the entries it had to skip.
|
||||
// Entries in one batch share a series, so the series detail is resolved once.
|
||||
func (s *Service) BuildBatchManifests(ctx context.Context, userID int, profileID, deviceID, batchID string, filter catalog.AccessFilter) ([]*OfflineManifest, []SkippedManifest, error) {
|
||||
if _, err := s.enabledConfig(ctx); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
if profileID == "" || deviceID == "" {
|
||||
return nil, nil, ErrProfileRequired
|
||||
}
|
||||
if s.manifest == nil {
|
||||
return nil, nil, ErrManifestUnavailable
|
||||
}
|
||||
rows, err := s.repo.ListManagedByBatch(ctx, userID, profileID, deviceID, batchID)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
if len(rows) == 0 {
|
||||
return nil, nil, ErrNotFound
|
||||
}
|
||||
out := make([]*OfflineManifest, 0, len(rows))
|
||||
skipped := make([]SkippedManifest, 0)
|
||||
seriesCache := make(map[string]*catalog.ItemDetail, 1)
|
||||
for _, dl := range rows {
|
||||
if dl.Status == StatusRevoked {
|
||||
skipped = append(skipped, SkippedManifest{DownloadID: dl.ID, Reason: "revoked"})
|
||||
continue
|
||||
}
|
||||
m, err := s.manifest.build(ctx, dl, filter, seriesCache)
|
||||
if err != nil {
|
||||
reason := "error"
|
||||
if errors.Is(err, catalog.ErrItemNotFound) {
|
||||
reason = "not_found"
|
||||
} else {
|
||||
slog.Warn("batch manifest build failed", "download_id", dl.ID, "batch_id", batchID, "error", err)
|
||||
}
|
||||
skipped = append(skipped, SkippedManifest{DownloadID: dl.ID, Reason: reason})
|
||||
continue
|
||||
}
|
||||
out = append(out, m)
|
||||
}
|
||||
return out, skipped, nil
|
||||
}
|
||||
|
||||
// ServeArtwork streams poster/backdrop/logo bytes for a managed entry through
|
||||
// the image resolver (never a presigned redirect), re-checking per-profile
|
||||
// access via GetItemDetail before serving.
|
||||
func (s *Service) ServeArtwork(ctx context.Context, w http.ResponseWriter, r *http.Request, userID int, profileID, deviceID, downloadID, kind string, filter catalog.AccessFilter) error {
|
||||
dl, err := s.authorizeManagedAsset(ctx, userID, profileID, deviceID, downloadID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if s.artworkSource == nil {
|
||||
return ErrManifestUnavailable
|
||||
}
|
||||
detail, err := s.artworkSource.GetItemDetail(ctx, manifestContentID(dl), filter)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var url string
|
||||
switch kind {
|
||||
case "poster":
|
||||
url = detail.PosterURL
|
||||
case "backdrop":
|
||||
url = detail.BackdropURL
|
||||
case "logo":
|
||||
url = detail.LogoURL
|
||||
default:
|
||||
return ErrAssetNotFound
|
||||
}
|
||||
if url == "" {
|
||||
return ErrAssetNotFound
|
||||
}
|
||||
return s.streamArtwork(ctx, w, r, url)
|
||||
}
|
||||
|
||||
func (s *Service) streamArtwork(ctx context.Context, w http.ResponseWriter, _ *http.Request, url string) error {
|
||||
client := s.httpClient
|
||||
if client == nil {
|
||||
client = http.DefaultClient
|
||||
}
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("building artwork request: %w", err)
|
||||
}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("fetching artwork: %w", err)
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("artwork upstream status %d: %w", resp.StatusCode, ErrAssetNotFound)
|
||||
}
|
||||
if ct := resp.Header.Get("Content-Type"); ct != "" {
|
||||
w.Header().Set("Content-Type", ct)
|
||||
}
|
||||
if cl := resp.Header.Get("Content-Length"); cl != "" {
|
||||
w.Header().Set("Content-Length", cl)
|
||||
}
|
||||
// Artwork is immutable for a stored manifest; let the client cache it once.
|
||||
w.Header().Set("Cache-Control", "private, max-age=31536000, immutable")
|
||||
if _, err := io.Copy(w, resp.Body); err != nil {
|
||||
return fmt.Errorf("streaming artwork: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ServeSubtitle streams a subtitle asset (external sidecar or downloaded S3 file)
|
||||
// for a managed entry, authorized on (user, profile, device) with a per-profile
|
||||
// content-access re-check. ref encodes "external:{index}" or "downloaded:{id}".
|
||||
func (s *Service) ServeSubtitle(ctx context.Context, w http.ResponseWriter, _ *http.Request, userID int, profileID, deviceID, downloadID, ref string, filter catalog.AccessFilter) error {
|
||||
dl, err := s.authorizeManagedAsset(ctx, userID, profileID, deviceID, downloadID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := s.itemAccess.EnsureAccessible(ctx, dl.ContentID, filter); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
kind, value, err := parseSubtitleRef(ref)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch kind {
|
||||
case "external":
|
||||
idx := value
|
||||
file, err := s.fileRepo.GetByID(ctx, dl.MediaFileID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("loading media file: %w", err)
|
||||
}
|
||||
if file == nil || idx < 0 || idx >= len(file.ExternalSubtitles) {
|
||||
return ErrAssetNotFound
|
||||
}
|
||||
ext := file.ExternalSubtitles[idx]
|
||||
data, err := playback.LoadExternalSubtitleRaw(ext.Path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("reading external subtitle: %w", ErrAssetNotFound)
|
||||
}
|
||||
writeSubtitle(w, ext.Format, data)
|
||||
return nil
|
||||
case "downloaded":
|
||||
if s.subtitleSource == nil {
|
||||
return ErrManifestUnavailable
|
||||
}
|
||||
sub, data, err := s.subtitleSource.GetSubtitleContent(ctx, value)
|
||||
if err != nil {
|
||||
return fmt.Errorf("loading downloaded subtitle: %w", ErrAssetNotFound)
|
||||
}
|
||||
// The subtitle must belong to this download's media file; a download id
|
||||
// never grants access to an arbitrary subtitle id.
|
||||
if sub == nil || sub.MediaFileID != dl.MediaFileID {
|
||||
return ErrAssetNotFound
|
||||
}
|
||||
writeSubtitle(w, string(sub.Format), data)
|
||||
return nil
|
||||
default:
|
||||
return ErrInvalidSubtitleRef
|
||||
}
|
||||
}
|
||||
|
||||
// parseSubtitleRef parses a subtitle reference of the form "external:{index}"
|
||||
// or "downloaded:{id}" into its kind and integer value.
|
||||
func parseSubtitleRef(ref string) (kind string, value int, err error) {
|
||||
k, v, ok := strings.Cut(ref, ":")
|
||||
if !ok {
|
||||
return "", 0, ErrInvalidSubtitleRef
|
||||
}
|
||||
switch k {
|
||||
case "external", "downloaded":
|
||||
n, perr := strconv.Atoi(v)
|
||||
if perr != nil {
|
||||
return "", 0, ErrInvalidSubtitleRef
|
||||
}
|
||||
return k, n, nil
|
||||
default:
|
||||
return "", 0, ErrInvalidSubtitleRef
|
||||
}
|
||||
}
|
||||
|
||||
// writeSubtitle writes subtitle bytes with a format-appropriate content type
|
||||
// (the shared subtitles mapping — no local copy to drift).
|
||||
func writeSubtitle(w http.ResponseWriter, format string, data []byte) {
|
||||
w.Header().Set("Content-Type", subtitles.SubtitleContentType(subtitles.SubtitleFormat(strings.ToLower(format))))
|
||||
w.Header().Set("Cache-Control", "private, max-age=31536000, immutable")
|
||||
_, _ = w.Write(data)
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user